Data table-based data processing method and device, storage medium and electronic equipment

Abstract

The invention discloses a data table-based data processing method and device, a storage medium and electronic equipment. The method comprises the steps that a data processing request is obtained, and the data processing request is used for requesting adjustment processing of data in a target data table; according to the data processing request, determining a target reference data table structure matched with the target data table from a plurality of reference data table structures stored in a custom table structure database, each reference data table structure is a standard table data structure obtained by carrying out user-defined combination on all data table metadata according to different definition conditions; and processing the data in the target data table according to the target reference data table structure. According to the method and the device, the technical problem of low data processing efficiency caused by repeated calibration of data due to low accuracy of manual data registration is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data processing method and device based on a data table, a storage medium, and an electronic device. Background Art

[0002] When maintaining the data table structure during the R&D process, basic table information, data sources, fields, indexes, constraints, partitions, and lifecycles are typically registered based on the actual situation. Field information such as field type, field length, association criteria, and dictionary information all require manual registration.

[0003] However, as business needs evolve, the frequency and cost of subsequent data governance and maintenance are increasing year by year. However, due to existing issues such as data standards and data dictionary implementation, data lifecycle issues, and data dictionary discrepancies, manual rectification will make it difficult to ensure data quality. For example, during data standards implementation, obvious errors such as standardizing account data to date were discovered. Such data standardization errors can seriously affect upstream and downstream information exchange and create audit risks.

[0004] That is, in the methods provided in the prior art, due to the inaccuracy of manual registration data, it is difficult to ensure consistency between upstream and downstream data, and the data has to be calibrated repeatedly, resulting in low data processing efficiency.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0006] The embodiments of the present application provide a data processing method and device, a storage medium and an electronic device based on a data table, so as to at least solve the technical problem that the data needs to be repeatedly calibrated due to the low accuracy of manually registered data, thereby resulting in low data processing efficiency.

[0007] According to one aspect of an embodiment of the present application, a data table-based data processing method is provided, including: obtaining a data processing request, wherein the above-mentioned data processing request is used to request adjustment processing of data in a target data table; according to the above-mentioned data processing request, determining a target reference data table structure that matches the above-mentioned target data table from multiple reference data table structures stored in a custom table structure database, wherein each of the above-mentioned reference data table structures is a standard table data structure obtained by customizing the metadata of a full data table according to different definition conditions; and processing the data in the above-mentioned target data table according to the above-mentioned target reference data table structure.

[0008] As an optional implementation, the above-mentioned determination of a target reference data table structure matching the target data table from a plurality of reference data table structures stored in a custom table structure database according to the above-mentioned data processing request includes: when the above-mentioned data processing request is used to request establishment of the above-mentioned target data table as a newly added data table, obtaining table attribute field information of the above-mentioned target data table currently entered; searching for at least one first candidate reference data table structure matching the table attribute field information of the above-mentioned target data table in the above-mentioned custom table structure database, wherein each of the above-mentioned first candidate reference data table structures includes the above-mentioned table attribute field information and table verification field information for verifying the data in the above-mentioned target data table; constructing a first data table list to be recommended based on the structure identifier of the above-mentioned at least one first candidate reference data table structure; and in response to a first target structure identifier selected from the above-mentioned first data table list, determining the above-mentioned first candidate reference data table structure matching the above-mentioned first target structure identifier as the above-mentioned target reference data table structure.

[0009] As an optional implementation, the above-mentioned processing of the data in the above-mentioned target data table according to the above-mentioned target reference data table structure includes: according to the above-mentioned target reference data table structure, associating the above-mentioned table attribute field information of the above-mentioned target data table and the target table verification field information contained in the above-mentioned target reference data table structure to obtain the above-mentioned target data table.

[0010] As an optional implementation, the above-mentioned determining, based on the above-mentioned data processing request, a target reference data table structure matching the above-mentioned target data table from a plurality of reference data table structures stored in a custom table structure database includes: in a case where the above-mentioned data processing request is used to request adjustment of the above-mentioned target data table in an abnormal state, obtaining abnormal table attribute field information associated with the table check field information indicating the above-mentioned abnormal state; searching for at least one second candidate reference data table structure matching the above-mentioned abnormal table attribute field information in the above-mentioned custom table structure database, wherein each of the above-mentioned second candidate reference data table structures includes the above-mentioned abnormal table attribute field information and the table check field information in a normal state matching the above-mentioned abnormal table attribute field information; constructing a second data table list to be recommended based on the structure identifier of the above-mentioned at least one second candidate reference data table structure; and in response to a second target structure identifier selected from the above-mentioned second data table list, determining the above-mentioned second candidate reference data table structure matching the above-mentioned second target structure identifier as the above-mentioned target reference data table structure.

[0011] As an optional implementation, the processing of the data in the above-mentioned target data table according to the above-mentioned target reference data table structure includes: adjusting the table check field information in the above-mentioned target data table according to the table check field information under the above-mentioned normal state indicated by the above-mentioned target reference data table structure to obtain the above-mentioned target data table.

[0012] As an optional implementation, before obtaining the above-mentioned data processing request, it also includes: obtaining the metadata of the full data table; classifying the above-mentioned metadata of the full data table to obtain a metadata set of the full data table; customizing the above-mentioned metadata set of the full data table to obtain the above-mentioned multiple reference data table structures, wherein each of the above-mentioned reference data table structures respectively includes table attribute field information and table verification field information with standard association relationships.

[0013] According to another aspect of an embodiment of the present application, a data table-based data processing device is also provided, including: a first acquisition unit, used to obtain a data processing request, wherein the above-mentioned data processing request is used to request adjustment processing of the data in the target data table; a determination unit, used to determine, according to the above-mentioned data processing request, a target reference data table structure that matches the above-mentioned target data table from multiple reference data table structures stored in a custom table structure database, wherein each of the above-mentioned reference data table structures is a standard table data structure obtained by customizing the metadata of the full data table according to different definition conditions; a processing unit, used to process the data in the above-mentioned target data table according to the above-mentioned target reference data table structure.

[0014] As an optional embodiment, the above-mentioned determination unit also includes: a first acquisition module, which is used to obtain the table attribute field information of the currently entered target data table when the above-mentioned data processing request is used to request the establishment of the above-mentioned target data table as a new data table; a first search module, which is used to search for at least one first candidate reference data table structure matching the table attribute field information of the above-mentioned target data table in the above-mentioned custom table structure database, wherein each of the above-mentioned first candidate reference data table structures includes the above-mentioned table attribute field information and table verification field information for verifying the data in the above-mentioned target data table; a first construction module, which is used to construct a list of first data tables to be recommended based on the structure identifier of the above-mentioned at least one first candidate reference data table structure; a first determination module, which is used to determine the above-mentioned first candidate reference data table structure matching the above-mentioned first target structure identifier as the above-mentioned target reference data table structure in response to the first target structure identifier selected from the above-mentioned first data table list.

[0015] As an optional implementation, the above-mentioned processing unit also includes: an association module, which is used to associate the above-mentioned table attribute field information of the above-mentioned target data table and the target table verification field information contained in the above-mentioned target reference data table structure according to the above-mentioned target reference data table structure to obtain the above-mentioned target data table.

[0016] As an optional embodiment, the above-mentioned determination unit also includes: a second acquisition module, which is used to obtain the abnormal table attribute field information associated with the table check field information indicating the above-mentioned abnormal state when the above-mentioned data processing request is used to request adjustment of the above-mentioned target data table in the abnormal state; a second search module, which is used to find at least one second candidate reference data table structure matching the above-mentioned abnormal table attribute field information in the above-mentioned custom table structure database, wherein each of the above-mentioned second candidate reference data table structures includes the above-mentioned abnormal table attribute field information and the table check field information in the normal state matching the above-mentioned abnormal table attribute field information; a second construction module, which is used to construct a second data table list to be recommended based on the structure identifier of the above-mentioned at least one second candidate reference data table structure; a second determination module, which is used to determine the above-mentioned second candidate reference data table structure matching the above-mentioned second target structure identifier as the above-mentioned target reference data table structure in response to the second target structure identifier selected from the above-mentioned second data table list.

[0017] As an optional implementation, the above-mentioned processing unit also includes: an adjustment module, which is used to adjust the table check field information in the above-mentioned target data table according to the table check field information under the above-mentioned normal state indicated by the above-mentioned target reference data table structure to obtain the above-mentioned target data table.

[0018] As an optional implementation, the above-mentioned device also includes: a second acquisition unit, which acquires the full data table metadata; a classification unit, which is used to classify the above-mentioned full data table metadata to obtain a full data table metadata set; a combination unit, which is used to customize the above-mentioned full data table metadata set to obtain the above-mentioned multiple reference data table structures, wherein each of the above-mentioned reference data table structures respectively includes table attribute field information and table verification field information with standard association relationships.

[0019] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is configured to execute the above-mentioned data processing method based on the data table when running.

[0020] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described data table-based data processing method.

[0021] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the data table-based data processing method through the computer program.

[0022] In an embodiment of the present application, after obtaining a data processing request, according to the obtained data processing request, a target reference data table structure that matches the target data table is determined from a plurality of reference data table structures stored in a custom table structure database, wherein the data processing request is used to request adjustment processing of the data in the target data table, and each reference data table structure is a standard table data structure obtained by customizing the metadata of the full data table according to different definition conditions; the data in the target data table is processed according to the target reference data table structure. After obtaining a data processing request, according to the data processing request, a target reference data table structure that matches the target data table is determined from a plurality of reference data table structures stored in a custom table structure database, and the data in the target data table is processed according to the target reference data table structure, thereby realizing automatic and intelligent processing of the data in the target data table using the preset custom table structure database, achieving the purpose of automatic and intelligent data processing, thereby improving data processing efficiency, and solving the technical problem that the data needs to be repeatedly calibrated due to the low accuracy of manually registered data, thereby leading to low data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 is a flowchart of an optional data processing method based on a data table according to an embodiment of the present application;

[0025] Figure 2 is a flowchart of another optional data processing method based on a data table according to an embodiment of the present application;

[0026] Figure 3is a flowchart of another optional data processing method based on a data table according to an embodiment of the present application;

[0027] Figure 4 is a flowchart of another optional data processing method based on a data table according to an embodiment of the present application;

[0028] Figure 5 is a schematic structural diagram of an optional data table-based data processing device according to an embodiment of the present application;

[0029] Figure 6 It is a schematic structural diagram of an optional computer system according to an embodiment of the present application.

[0030] Figure 7 It is a schematic structural diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] According to one aspect of the embodiment of the present application, a data processing method based on a data table is provided, such as Figure 1 As shown, the method includes:

[0034] S102, obtaining a data processing request, wherein the data processing request is used to request adjustment processing of data in a target data table;

[0035] Optionally, in this embodiment, the above-mentioned data processing request may be, but is not limited to, a request to establish a target data table as a newly added data table, or a request to adjust a target data table that is in an abnormal state. The above-mentioned target data table may be, but is not limited to, a data table that includes some metadata contained in a full data table. The above-mentioned metadata may include, but is not limited to, basic table information and table field information. The above-mentioned table basic information may include, but is not limited to, business field, subject domain, functional classification characteristics, database type, system English abbreviation, business system abbreviation, table English name, table Chinese name, etc. The above-mentioned table field information may include, but is not limited to, table field English name, table field Chinese name, field type, length, decimal places, whether to associate with standards, data standards, data dictionary, etc. The above-mentioned full data table may be, but is not limited to, a wide table, as shown in Table 1:

[0036] Table 1

[0037]

[0038] S104: Determine, based on the data processing request, a target reference data table structure that matches the target data table from a plurality of reference data table structures stored in the custom table structure database, wherein each reference data table structure is a standard table data structure obtained by customizing and combining metadata of all data tables according to different definition conditions.

[0039] Optionally, in this embodiment, the above-mentioned custom table structure database may be, but is not limited to, a custom rule base storing multiple custom rules and rule names, and the above-mentioned reference data table structure may be, but is not limited to, the above-mentioned custom rules. The above-mentioned target reference data table structure may include, but is not limited to, metadata information matched by the metadata in the above-mentioned target data table under a certain custom rule. The above-mentioned custom rules may include, but are not limited to, multiple rule details, used to indicate the association relationship between table basic information and table field information, table basic information and table basic information, and table field information and table field information. The above-mentioned association relationship may be, but is not limited to, the mapping relationship between the above-mentioned metadata obtained after the data governance model classifies and analyzes the above-mentioned metadata. The above-mentioned rule details may be, but are not limited to, the table basic information and table field information in Table 1 above. The above-mentioned custom rule base may be, but is not limited to, storing the above-mentioned custom rules in the form of a table, as shown in Table 2:

[0040] Table 2

[0041]

[0042]

[0043] S106: Process the data in the target data table according to the target reference data table structure.

[0044] Through the embodiment of the present application, after obtaining a data processing request, according to the obtained data processing request, a target reference data table structure that matches the target data table is determined from multiple reference data table structures stored in the custom table structure database, wherein the data processing request is used to request adjustment processing of the data in the target data table, and each reference data table structure is a standard table data structure obtained by customizing the metadata of the full data table according to different definition conditions; the data in the target data table is processed according to the target reference data table structure. After obtaining a data processing request, according to the data processing request, a target reference data table structure that matches the target data table is determined from multiple reference data table structures stored in the custom table structure database, and the data in the target data table is processed according to the target reference data table structure, thereby realizing automatic and intelligent processing of the data in the target data table using the preset custom table structure database, achieving the purpose of automatic and intelligent data processing, thereby improving data processing efficiency, and solving the technical problem that the data needs to be repeatedly calibrated due to the low accuracy of manually registered data, thereby leading to low data processing efficiency.

[0045] As an optional solution, the above step S104, based on the data processing request, determines a target reference data table structure that matches the target data table from multiple reference data table structures stored in the custom table structure database, including:

[0046] S1, when the data processing request is for requesting to establish a target data table as a newly added data table, obtaining table attribute field information of the currently entered target data table;

[0047] Optionally, in this embodiment, the above-mentioned table attribute field information can be but is not limited to the table field information or table basic information in the above-mentioned Table 1. For example, the above-mentioned table attribute field information can be the English name of the table field in the table field information contained in the above-mentioned Table 1, the data standard, or the business field, subject domain, etc. in the table basic information contained in the above-mentioned Table 1.

[0048] S2, searching in the custom table structure database for at least one first candidate reference data table structure that matches table attribute field information of the target data table, wherein each first candidate reference data table structure includes table attribute field information and table verification field information for verifying data in the target data table;

[0049] Optionally, in this embodiment, the first candidate reference data table structure may be, but is not limited to, a custom rule in the custom rule library. The table validation field information may be, but is not limited to, metadata information matched by the table attribute field information in the custom table structure database. For example, the table attribute field information may be the credit field in the business domain or the account field in the subject domain. The table validation field information may be, but is not limited to, data standard information or data dictionary information matched by the credit field and the account field in the custom rule library.

[0050] S3, constructing a list of first data tables to be recommended based on a structure identifier of at least one first candidate reference data table structure;

[0051] Optionally, in this embodiment, the above-mentioned structure identifier may be but is not limited to the rule name in the above-mentioned Table 2, and the above-mentioned first data table list may be but is not limited to a data set storing the above-mentioned at least one first candidate reference data table structure. For example, when the above-mentioned table attribute field information is a business field or a subject domain, and the above-mentioned table verification field information is the data standard 1 and data dictionary 1 matched by the above-mentioned business field and the above-mentioned subject domain in the above-mentioned custom rule library, the above-mentioned first data table list may be a database containing the above-mentioned data standard information and the above-mentioned data dictionary information.

[0052] S4 , in response to a first target structure identifier selected from the first data table list, determining a first candidate reference data table structure that matches the first target structure identifier as a target reference data table structure.

[0053] For example, assuming that the table attribute field information in the target data table is business domain, subject domain, functional classification characteristics, database type, table field English name, etc., the custom table structure database can be a custom rule library including multiple custom rules, the table check field information is data standard information, data dictionary information, the first candidate reference data table structure is the mapping relationship between the table attribute field information and the table check field information, and the first data table list is a data set containing the metadata standard and dictionary information recommended by the mapping relationship. Figure 2As shown, in the case where the data processing request is used to request the establishment of a target data table as a newly added data table, step S202 is executed to enter elements such as business fields, subject domains, functional classification characteristics, database types, and English names of table fields. After the entry is completed, step S204 is executed to match the entered information with the relevant rules in the custom rule library in the custom rule library. After the matching is completed, step S206 is executed to use the matching results to drive the data governance model to perform data classification analysis. The data governance model here can be a neural network model with data classification analysis and data mapping relationship extraction. For example, the data governance model can be a long short-term memory network (Long Short-Term Memory, LSTM) model. Then step S208 is executed to extract the mapping relationship between each field under each application and business field dimension and elements such as data standards and data dictionaries. After the mapping relationship is extracted, the data standards, data dictionaries and other elements included in the extracted mapping relationship are collected and step S210 is executed to form a recommended data standard and data dictionary information data set. Under the rule indicated by the rule name, the information associated with elements such as business fields is the data standard and data dictionary in the data set.

[0054] Through the embodiments of the present application, when a data processing request is used to request the establishment of a target data table as a newly added data table, at least one first candidate reference data table structure that matches the table attribute field information of the target data table is found in the custom table structure database based on the table attribute field information of the currently entered target data table, and based on the structure identifier of at least one first candidate reference data table structure, a list of first data tables to be recommended is constructed, so that in response to the first target structure identifier selected from the first data table list, the first candidate reference data table structure that matches the first target structure identifier is determined as the target reference data table structure, thereby achieving the purpose of intelligently and dynamically adding data based on the custom table structure database, improving data governance efficiency, and enhancing data governance flexibility.

[0055] As an optional solution, the above S106, processing the data in the target data table according to the target reference data table structure, includes:

[0056] S1. According to the target reference data table structure, the table attribute field information of the target data table is associated with the target table verification field information contained in the target reference data table structure to obtain the target data table.

[0057] Through the embodiments of the present application, according to the target reference data table structure, the table attribute field information of the target data table and the target table verification field information contained in the target reference data table structure are associated to obtain the target data table, thereby achieving the purpose of dynamically improving the associated information of the data, enhancing the flexibility of data governance, and facilitating data governance in richer scenarios.

[0058] As an optional solution, the above S104, based on the data processing request, determining a target reference data table structure that matches the target data table from a plurality of reference data table structures stored in the custom table structure database, further includes:

[0059] S1, when the data processing request is for requesting adjustment of a target data table in an abnormal state, obtaining abnormal table attribute field information associated with table check field information indicating the abnormal state;

[0060] S2, searching for at least one second candidate reference data table structure matching the abnormal table attribute field information in the custom table structure database, wherein each second candidate reference data table structure includes the abnormal table attribute field information and table check field information in a normal state that matches the abnormal table attribute field information;

[0061] S3, constructing a list of second data tables to be recommended based on the structure identifier of at least one second candidate reference data table structure;

[0062] S4 , in response to a second target structure identifier selected from the second data table list, determining a second candidate reference data table structure that matches the second target structure identifier as a target reference data table structure.

[0063] Optionally, in this embodiment, the abnormal state may refer to, but is not limited to, a situation where the association between the table field information and the basic table information in the target data table does not conform to a custom rule in the custom rule library. The second candidate reference data table structure may refer to, but is not limited to, a custom rule in the custom rule library. The structure identifier may refer to, but is not limited to, the name of the rule corresponding to the second candidate reference data table structure. The second data table list may refer to, but is not limited to, a data set storing at least one custom rule.

[0064] For example, assuming that the table attribute field information in the target data table is the table field information associated with the anomaly and its associated data standards and dictionary information, the custom table structure database may be a custom rule library including multiple custom rules, the table check field information is the data standard information and dictionary information, the first candidate reference data table structure is the mapping relationship between the table field information and the data standard and dictionary information, and the first data table list is a data set containing the metadata standards and dictionary information recommended by the mapping relationship. Figure 3 As shown, when the data processing request is used to request the establishment of a target data table as a newly added data table, step S302 is executed to enter the table field information of the associated exception and its associated data standards and dictionary information. After the entry is completed, step S304 is executed to match the entered information with the relevant rules in the custom rule library in the custom rule library. After the matching is completed, step S306 is executed to use the matching results to drive the data governance model to perform data classification analysis. The data governance model here can be a neural network model with data classification analysis and data mapping relationship extraction. For example, the data governance model can be a long short-term memory network (Long Short-Term Memory, LSTM) model. Then step S308 is executed to extract the mapping relationship between the table field information and the data standard and dictionary information. After the mapping relationship extraction is completed, the extracted mapping relationship is included in the data standard, data dictionary and other elements, and step S310 is executed to form a recommended data standard and data dictionary information data set.

[0065] Through the embodiments of the present application, when a data processing request is used to request an adjustment to a target data table in an abnormal state, abnormal table attribute field information associated with the table check field information indicating the abnormal state is obtained, and based on the obtained abnormal table attribute field information, at least one second candidate reference data table structure matching the abnormal table attribute field information is found in the custom table structure database, and a second data table list to be recommended is constructed based on the structure identifier of at least one second candidate reference data table structure, so that in response to the second target structure identifier selected from the second data table list, the second candidate reference data table structure matching the second target structure identifier can be determined as the target reference data table structure, thereby achieving the purpose of correcting the associated abnormal data based on the custom table structure database, improving the accuracy of data governance, and enhancing the flexibility of data governance.

[0066] As an optional solution, the above S106, processing the data in the target data table according to the target reference data table structure, includes:

[0067] S1, adjusting the table check field information in the target data table according to the table check field information in the normal state indicated by the target reference data table structure to obtain the target data table.

[0068] Optionally, in this embodiment, the above adjustment may be, but is not limited to, correcting the above table check field information in the above target data table according to the table check field information under the normal state indicated by the above target reference data table structure, so that the association relationship between the data in the corrected above target data table complies with the custom rules corresponding to the above target reference data table structure.

[0069] Through the embodiments of the present application, the table check field information in the target data table is adjusted according to the table check field information under the normal state indicated by the target reference data table structure, so that the associated abnormal data can be corrected and adjusted in a timely manner, thereby improving the accuracy of data governance, avoiding unnecessary multiple data corrections, and improving data governance efficiency.

[0070] As an optional solution, before the above step S102 of obtaining the data processing request, the following steps may be further included:

[0071] S1, obtain metadata of the full data table;

[0072] S2, classify the metadata of the full data table to obtain a metadata set of the full data table;

[0073] S3, performing custom combination on the metadata set of the full data table to obtain multiple reference data table structures, wherein each reference data table structure includes table attribute field information and table check field information with standard association relationships.

[0074] Optionally, in this embodiment, the full data table metadata may be, but is not limited to, the table field information and table basic information shown in Table 1. The classification processing may be, but is not limited to, classifying and arranging the acquired table field information and table basic information into a wide table format as shown in Table 1, and the full data table metadata set may be, but is not limited to, the wide table obtained after the classification and arrangement.

[0075] Through the embodiments of the present application, after obtaining the metadata of the full data table, the metadata of the full data table is classified and processed to obtain a full data table metadata set, and the full data table metadata set is customized and combined to obtain multiple reference data table structures, so that data can be intelligently managed based on the preset reference data table structure, thereby improving data management efficiency.

[0076] Specific combination Figure 4 The process shown is used to illustrate the process of applying the embodiment of the present application to data governance:

[0077] S402, rule presetting. Collect and organize metadata from all data tables. The metadata includes basic table information and table field information. Basic table information includes business domain, subject domain, functional classification characteristics, database type, system English abbreviation, business system abbreviation, table English name, and table Chinese name. Table field information includes table field English name, table field Chinese name, field type, length, decimal places, whether it is associated with a standard, data standard, and data dictionary. Freely combine the table field information with the basic table information to obtain multiple custom rules. The obtained multiple custom rules are aggregated to form a custom rule library. Based on the specified one or more custom rules, use the big model to classify and analyze the metadata and extract the mapping relationship between the metadata.

[0078] S404, Data Acquisition. A data processing request is obtained. Here, the data processing request is a request to establish a target data table as a new data table, or a request to adjust a target data table that is in an abnormal state. When the data processing request is a request to establish a target data table as a new data table, the target data table is an incremental data table containing newly added metadata. When the data processing request is a request to adjust a target data table that is in an abnormal state, the target data table is a full data table with metadata associated with the abnormality.

[0079] S406, request type determination: Determine whether the data processing request type is a request to create a target data table as a newly added data table or a request to adjust a target data table in an abnormal state.

[0080] S408, Data Governance. This includes:

[0081] S4082, data association is added; in the case where the data processing request is a request to establish a target data table as a newly added data table, the target data table is an incremental data table containing newly added metadata. According to the preset custom rule library, the metadata information matched by the newly added metadata in the incremental data table under one or more rules is obtained, and the newly added metadata and the metadata information matched by it are entered into the full data table.

[0082] S4084, Data Association Adjustment: If the data processing request is to adjust a target data table in an abnormal state, the target data table is the full data table containing metadata associated with the anomaly. Based on a preset custom rule base, metadata information that correctly matches the metadata associated with the anomaly in the full data table under one or more rules is obtained, and the metadata associated with the anomaly and its correctly matched metadata are entered into the full data table.

[0083] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0084] According to another aspect of the embodiment of the present application, a data processing device based on a data table for implementing a data processing method based on a data table is also provided. Figure 5 As shown, the device includes:

[0085] A first acquiring unit 502 is configured to acquire a data processing request, wherein the data processing request is used to request adjustment processing of data in a target data table;

[0086] A determining unit 504 is configured to determine, based on the data processing request, a target reference data table structure that matches the target data table from a plurality of reference data table structures stored in the custom table structure database, wherein each reference data table structure is a standard table data structure obtained by customizing and combining metadata of a full data table according to different definition conditions;

[0087] The processing unit 506 is configured to process the data in the target data table according to the target reference data table structure.

[0088] Optionally, the embodiments in this solution may include but are not limited to the above-mentioned reference method embodiments, which will not be described in detail here.

[0089] As an optional solution, the determination unit includes:

[0090] A first acquisition module is configured to acquire table attribute field information of a currently entered target data table when the data processing request is for requesting to establish a target data table as a newly added data table;

[0091] a first search module configured to search within the user-defined table structure database for at least one first candidate reference data table structure that matches table attribute field information of a target data table, wherein each first candidate reference data table structure includes table attribute field information and table verification field information for verifying data in the target data table;

[0092] A first building module is configured to build a list of first data tables to be recommended based on a structure identifier of at least one first candidate reference data table structure;

[0093] The first determining module is configured to, in response to a first target structure identifier selected from the first data table list, determine a first candidate reference data table structure that matches the first target structure identifier as a target reference data table structure.

[0094] Optionally, the embodiments in this solution may refer to, but are not limited to, the above-mentioned method embodiments, which will not be described in detail here.

[0095] As an optional solution, the processing unit further includes: an association module for associating the table attribute field information of the target data table with the target table verification field information contained in the target reference data table structure according to the target reference data table structure to obtain the target data table.

[0096] Optionally, the embodiments in this solution may refer to, but are not limited to, the above-mentioned method embodiments, which will not be described in detail here.

[0097] As an optional solution, the determining unit further includes:

[0098] A second acquisition module is configured to acquire abnormal table attribute field information associated with table check field information indicating an abnormal state when the data processing request is for requesting adjustment of a target data table in an abnormal state;

[0099] A second search module is configured to search the user-defined table structure database for at least one second candidate reference data table structure that matches the abnormal table attribute field information, wherein each second candidate reference data table structure includes the abnormal table attribute field information and table check field information in a normal state that matches the abnormal table attribute field information;

[0100] A second building module is configured to build a list of second data tables to be recommended based on a structure identifier of at least one second candidate reference data table structure;

[0101] The second determining module is configured to determine, in response to a second target structure identifier selected from the second data table list, a second candidate reference data table structure matching the second target structure identifier as a target reference data table structure.

[0102] Optionally, the embodiments in this solution may refer to, but are not limited to, the above-mentioned method embodiments, which will not be described in detail here.

[0103] As an optional solution, the processing unit further includes:

[0104] The adjustment module is used to adjust the table check field information in the target data table according to the table check field information in the normal state indicated by the target reference data table structure to obtain the target data table.

[0105] Optionally, the embodiments in this solution may refer to, but are not limited to, the above-mentioned method embodiments, which will not be described in detail here.

[0106] As an optional solution, the above device also includes:

[0107] The second acquisition unit acquires metadata of the entire data table;

[0108] A classification unit, configured to classify the above-mentioned full data table metadata to obtain a full data table metadata set;

[0109] The combination unit is used to perform custom combination on the above-mentioned full data table metadata set to obtain the above-mentioned multiple reference data table structures, wherein each of the above-mentioned reference data table structures includes table attribute field information and table verification field information with standard association relationships.

[0110] Optionally, the embodiments in this solution may refer to, but are not limited to, the above-mentioned method embodiments, which will not be described in detail here.

[0111] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional implementations of the above-mentioned data table-based data processing method.

[0112] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0113] S1, obtaining a data processing request, wherein the data processing request is used to request adjustment processing of data in a target data table;

[0114] S2, based on the data processing request, determining a target reference data table structure that matches the target data table from multiple reference data table structures stored in the custom table structure database, wherein each reference data table structure is a standard table data structure obtained by customizing and combining metadata of all data tables according to different definition conditions;

[0115] S3, processing the data in the target data table according to the target reference data table structure.

[0116] According to one aspect of the present application, a computer program product is provided. The computer program product includes a computer program / instruction. The computer program / instruction includes a program code for executing the method shown in the above flowchart.

[0117] Figure 6 The block diagram schematically shows a computer system structure of an electronic device used to implement an embodiment of the present application.

[0118] It should be noted that Figure 6 The computer system of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0119] like Figure 6 As shown, the computer system includes a central processing unit 601 (CPU), which can perform various appropriate actions and processes according to the program stored in the read-only memory 602 (ROM) or the program loaded from the storage part 608 into the random access memory 603 (RAM). Various programs and data required for system operation are also stored in the random access memory 603. The central processing unit 601, the read-only memory 602 and the random access memory 603 are connected to each other via a bus 604. An input / output interface 605 (i.e., an I / O interface) is also connected to the bus 604.

[0120] The following components are connected to the input / output interface 605: an input section 606 including a keyboard, a mouse, and the like; an output section 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 608 including a hard disk; and a communication section 609 including a network interface card such as a local area network card or a modem. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output interface 605 as needed. Removable media 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, are installed in the drive 610 as needed, so that computer programs read therefrom can be installed into the storage section 608 as needed.

[0121] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609 and / or installed from a removable medium 611. When the computer program is executed by the central processing unit 601, the various functions defined in the system of the present application are performed.

[0122] According to another aspect of the embodiment of the present application, an electronic device for implementing the above-mentioned data processing method based on the data table is also provided. Figure 7 As shown, the electronic device includes a memory 702 and a processor 704. The memory 702 stores a computer program, and the processor 704 is configured to execute the steps in any of the above method embodiments through the computer program.

[0123] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0124] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0125] S1, obtaining a data processing request, wherein the data processing request is used to request adjustment processing of data in a target data table;

[0126] S2, based on the data processing request, determining a target reference data table structure that matches the target data table from multiple reference data table structures stored in the custom table structure database, wherein each reference data table structure is a standard table data structure obtained by customizing and combining metadata of all data tables according to different definition conditions;

[0127] S3, processing the data in the target data table according to the target reference data table structure.

[0128] Alternatively, those skilled in the art will appreciate that Figure 7 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 7 The structure of the electronic device is not limited. For example, the electronic device may also include Figure 7 More or fewer components (such as network interfaces, etc.) as shown in, or with Figure 7 Different configurations shown.

[0129] Among them, the memory 702 can be used to store software programs and modules, such as the program instructions / modules corresponding to the data table-based data processing method and device in the embodiment of the present application. The processor 704 executes various functional applications and data processing by running the software programs and modules stored in the memory 702, that is, realizing the above-mentioned data table-based data processing method. The memory 702 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 702 may further include a memory remotely located relative to the processor 704, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned networks include but are not limited to the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. Among them, the memory 702 can be used specifically, but not limited to, to store information such as data processing requests and custom table structure databases. As an example, if Figure 7 As shown, the memory 702 may include, but is not limited to, the first acquisition unit 502, the determination unit 504, and the processing unit 506 in the data processing device based on the data table. In addition, it may also include, but is not limited to, other module units in the data processing device based on the data table, which will not be repeated in this example.

[0130] Optionally, the transmission device 706 is configured to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 706 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 706 is a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0131] In addition, the electronic device further includes: a display 708 for displaying the custom table structure database; and a connection bus 710 for connecting various module components in the electronic device.

[0132] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes via network communication. The nodes may form a point-to-point network, and any computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the point-to-point network.

[0133] Alternatively, in the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0134] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0135] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application.

[0136] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0138] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0139] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0140] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method based on a data table, characterized in that: include: Obtaining a data processing request, wherein the data processing request is used to request adjustment processing of data in a target data table; According to the data processing request, a target reference data table structure matching the target data table is determined from a plurality of reference data table structures stored in a custom table structure database, wherein each of the reference data table structures is a standard table data structure obtained by customizing and combining metadata of a full data table according to different definition conditions; The data in the target data table is processed according to the target reference data table structure.

2. The method according to claim 1, characterized in that Determining, according to the data processing request, a target reference data table structure matching the target data table from a plurality of reference data table structures stored in a custom table structure database comprises: In a case where the data processing request is used to request to establish the target data table as a newly added data table, obtaining table attribute field information of the target data table currently entered; Searching for at least one first candidate reference data table structure matching the table attribute field information of the target data table in the custom table structure database, wherein each of the first candidate reference data table structures includes the table attribute field information and table verification field information for verifying the data in the target data table; Based on the structure identifier of the at least one first candidate reference data table structure, construct a list of first data tables to be recommended; In response to a first target structure identifier selected from the first data table list, the first candidate reference data table structure matching the first target structure identifier is determined as the target reference data table structure.

3. The method according to claim 2, characterized in that The processing of the data in the target data table according to the target reference data table structure includes: According to the target reference data table structure, the table attribute field information of the target data table and the target table verification field information contained in the target reference data table structure are associated to obtain the target data table.

4. The method according to claim 1, characterized in that: Determining, according to the data processing request, a target reference data table structure matching the target data table from a plurality of reference data table structures stored in a custom table structure database comprises: In a case where the data processing request is used to request adjustment of the target data table in an abnormal state, acquiring abnormal table attribute field information associated with table check field information indicating the abnormal state; Searching for at least one second candidate reference data table structure matching the abnormal table attribute field information in the custom table structure database, wherein each of the second candidate reference data table structures includes the abnormal table attribute field information and table check field information in a normal state matching the abnormal table attribute field information; Building a second data table list to be recommended based on the structure identifier of the at least one second candidate reference data table structure; In response to a second target structure identifier selected from the second data table list, the second candidate reference data table structure matching the second target structure identifier is determined as the target reference data table structure.

5. The method according to claim 4, characterized in that The processing of the data in the target data table according to the target reference data table structure includes: According to the table check field information in the normal state indicated by the target reference data table structure, the table check field information in the target data table is adjusted to obtain the target data table.

6. The method according to any one of claims 1 to 5, characterized in that Before obtaining the data processing request, the method further includes: Get metadata of all data tables; Classifying the metadata of the full data table to obtain a full data table metadata set; The full data table metadata set is customized and combined to obtain the multiple reference data table structures, wherein each of the reference data table structures includes table attribute field information and table check field information having a standard association relationship.

7. A data processing device based on a data table, characterized in that: include: A first acquisition unit, used to acquire a data processing request, wherein the data processing request is used to request adjustment processing of data in a target data table; a determination unit, configured to determine, according to the data processing request, a target reference data table structure matching the target data table from a plurality of reference data table structures stored in a custom table structure database, wherein each of the reference data table structures is a standard table data structure obtained by customizing and combining metadata of a full data table according to different definition conditions; A processing unit is used to process the data in the target data table according to the target reference data table structure.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program is executed by a processor to perform the method described in any one of claims 1 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

Images