Data table merging method, device, equipment and storage medium

By identifying and counting the number of merged fields in the pending data table, detecting accuracy, collecting business data, and writing to the output table in parallel, the problem of low data table merging is solved, and efficient and accurate data table merging is achieved.

CN114510487BActive Publication Date: 2024-11-01PING AN TECH (SHENZHEN) CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210167652.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-23
Publication Date
2024-11-01
Estimated Expiration
2042-02-23

AI Technical Summary

Technical Problem

During the real-time indicator development process of existing technology, data table merging is inefficient, resulting in the inability to effectively mine information.

Method used

By identifying the pending data table, counting the number of merged fields, detecting the generation accuracy, collecting business data, and writing in parallel from the configuration table positioning output table based on the merged fields to generate the target data table.

Benefits of technology

It improves the efficiency and accuracy of data table merging, reduces the splicing processing of data tables to be processed, and improves the efficiency of business data merging.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114510487B_ABST
    Figure CN114510487B_ABST
Patent Text Reader

Abstract

The present invention relates to data processing, and provides a data table merging method, device, equipment and storage medium. The method can receive a data table merging request; identify multiple data tables to be processed according to the data table merging request, each of the data tables to be processed includes a primary key; count the number of fields of each of the merged fields in the multiple data tables to be processed; detect the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields; collect business data from the multiple data tables to be processed according to the generation accuracy and the merged fields; locate the output table from the configuration table based on the merged fields; write the business data to the output table in parallel based on the primary key and the field types of the multiple merged fields to obtain a target data table, which can improve the efficiency of data table merging. In addition, the present invention also relates to blockchain technology, and the target data table can be stored in a blockchain.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data table merging method, device, equipment and storage medium. Background Art

[0002] In the process of real-time indicator development, in order to improve development efficiency, different indicators are usually distributed to different development teams for development. Therefore, in the process of mining indicator data, there is usually a need to associate and merge data tables. At present, the merging of data tables is usually achieved by splicing dimension tables. However, this method will increase the complexity of merging, resulting in a decrease in the efficiency of merging data tables, which is not conducive to mining effective information in real time. Summary of the invention

[0003] In view of the above, it is necessary to provide a data table merging method, device, equipment and storage medium, which can improve the efficiency of data table merging.

[0004] In one aspect, the present invention provides a data table merging method, the data table merging method comprising:

[0005] Receive data table merge requests;

[0006] Identifying a plurality of data tables to be processed according to the data table merge request, each of the data tables to be processed comprising a primary key;

[0007] Counting the number of fields of each of the merged fields in the multiple data tables to be processed;

[0008] Detecting the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields;

[0009] Collecting business data from the plurality of data tables to be processed according to the generation accuracy and the merged fields;

[0010] Locating an output table from the configuration table based on the merge fields;

[0011] The business data is written into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

[0012] According to a preferred embodiment of the present invention, identifying a plurality of to-be-processed data tables according to the data table merge request comprises:

[0013] Parsing the message of the data table merge request to obtain the data information carried in the message;

[0014] Extracting a database identifier and a key value dimension from the data information;

[0015] Determine the database corresponding to the database identifier as the target database;

[0016] Data tables that match both the key value dimension and the preset status are selected from the target database as the multiple data tables to be processed.

[0017] According to a preferred embodiment of the present invention, counting the number of fields of each of the merged fields in the plurality of data tables to be processed comprises:

[0018] Extracting a plurality of merged fields from each of the to-be-processed data tables;

[0019] Writing each of the merged fields into a preset query instruction to obtain a field query statement for each of the merged fields;

[0020] Query the multiple data tables to be processed based on the field query statement to obtain query results;

[0021] The number of data tables in the query result is counted to obtain the number of fields.

[0022] According to a preferred embodiment of the present invention, detecting the generation accuracy of the merged fields by the multiple to-be-processed data tables according to the number of fields includes:

[0023] Determine the merged fields whose number of fields is greater than or equal to the preset number as repeated fields;

[0024] Filtering out a plurality of data tables to be tested from the plurality of data tables to be processed based on the repeated fields;

[0025] Obtaining a historical merge record of each of the data tables to be tested, and obtaining a merge accuracy of each of the historical merge records;

[0026] Calculating the generation accuracy of the repeated fields for each of the to-be-processed data tables according to the merging accuracy includes:

[0027]

[0028] Where y refers to the generation accuracy, x1, x2, ..., x n They respectively refer to the combined accuracy, and k refers to the preset adjustment ratio.

[0029] According to a preferred embodiment of the present invention, collecting business data from the plurality of to-be-processed data tables according to the generation accuracy and the merged fields comprises:

[0030] Determine the data table to be tested with the highest generation accuracy as the characteristic data table;

[0031] Collecting data corresponding to the repeated field from the feature data table as the business data;

[0032] Determine the merged fields that are smaller than the preset number as a single field;

[0033] The data corresponding to the single field is collected from the to-be-processed data table containing the single field as the business data.

[0034] According to a preferred embodiment of the present invention, locating the output table from the configuration table based on the merge field comprises:

[0035] Acquire an index table of each merged field from the configuration table;

[0036] Filter out the index table corresponding to all merge fields as the initial table;

[0037] If there are multiple initial tables, then obtain the running status of each initial table;

[0038] The initial table whose operating state is a configuration state is determined as the output table.

[0039] According to a preferred embodiment of the present invention, the business data is written in parallel to the output table based on the primary key and the field type of the plurality of merged fields to obtain a target data table including:

[0040] Matching the business data corresponding to each of the merged fields with the setting data corresponding to the preset type, and determining the preset type corresponding to the setting data that successfully matches the business data as the field type;

[0041] Obtaining an execution thread corresponding to the field type from a preset thread pool;

[0042] Based on the primary key, the business data corresponding to the field type is written into the output table in parallel by the execution thread to obtain the target data table.

[0043] On the other hand, the present invention further provides a data table merging device, the data table merging device comprising:

[0044] A receiving unit, used for receiving a data table merging request;

[0045] an identification unit, configured to identify a plurality of to-be-processed data tables according to the data table merging request, each of the to-be-processed data tables comprising a primary key;

[0046] A statistical unit, used for counting the number of fields of each of the merged fields in the plurality of data tables to be processed;

[0047] A detection unit, configured to detect the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields;

[0048] A collection unit, configured to collect business data from the plurality of data tables to be processed according to the generation accuracy and the merged fields;

[0049] a positioning unit, used for locating an output table from the configuration table based on the merged field;

[0050] A writing unit is used to write the business data into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

[0051] On the other hand, the present invention further provides an electronic device, comprising:

[0052] a memory storing computer readable instructions; and

[0053] A processor executes the computer-readable instructions stored in the memory to implement the data table merging method.

[0054] On the other hand, the present invention further proposes a computer-readable storage medium, in which computer-readable instructions are stored. The computer-readable instructions are executed by a processor in an electronic device to implement the data table merging method.

[0055] It can be seen from the above technical solutions that the present invention can accurately screen out the merged fields that need to be detected and analyzed by counting the number of fields of each merged field in the multiple data tables to be processed. By detecting the generation accuracy of the merged field by the multiple data tables to be processed, it is beneficial to accurately collect the business data corresponding to the merged field from the multiple data tables to be processed, thereby improving the generation accuracy of the target data table. In addition, based on the merged field, the output table can be located from the configuration table, and then the business data can be written to the output table in parallel according to the field type. Since the output table can be directly located without splicing the multiple data tables to be processed, the merging efficiency of the business data can be improved. At the same time, since the present invention writes the business data in parallel, the data table merging efficiency can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a flow chart of a preferred embodiment of the data table merging method of the present invention.

[0057] Figure 2 It is a functional module diagram of a preferred embodiment of the data table merging device of the present invention.

[0058] Figure 3 It is a structural schematic diagram of an electronic device of a preferred embodiment of the data table merging method of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0060] like Figure 1 FIG. 1 is a flow chart of a preferred embodiment of the data table merging method of the present invention. According to different requirements, the order of the steps in the flow chart can be changed, and some steps can be omitted.

[0061] The data table merging method can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0063] The data table merging method is applied to one or more electronic devices, which are devices that can automatically perform numerical calculations and / or information processing according to pre-set or stored computer-readable instructions, and whose hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0064] The electronic device may be any electronic product that can perform human-computer interaction with a user, such as a personal computer, a tablet computer, a smart phone, a personal digital assistant (PDA), a game console, an interactive network television (IPTV), a smart wearable device, etc.

[0065] The electronic device may include a network device and / or a user device. The network device includes, but is not limited to, a single network electronic device, an electronic device group consisting of multiple network electronic devices, or a cloud consisting of a large number of hosts or network electronic devices based on cloud computing.

[0066] The network where the electronic device is located includes, but is not limited to: the Internet, a wide area network, a metropolitan area network, a local area network, a virtual private network (VPN), etc.

[0067] S10, receiving a data table merge request.

[0068] In at least one embodiment of the present invention, the data table merge request can be triggered and generated by any business personnel. The data table merge request carries a tag for indicating the database identifier, the database identifier, a tag for indicating the key value dimension, and the key value dimension.

[0069] In at least one embodiment of the present invention, the electronic device performs authority authentication on the triggering user of the data table merging request, and when the triggering user passes the authority authentication, the electronic device receives the data table merging request.

[0070] S11, identifying a plurality of data tables to be processed according to the data table merge request, each of the data tables to be processed including a primary key.

[0071] In at least one embodiment of the present invention, the multiple data tables to be processed refer to data tables that need to be merged.

[0072] In at least one embodiment of the present invention, the electronic device identifying a plurality of to-be-processed data tables according to the data table merging request comprises:

[0073] Parsing the message of the data table merge request to obtain the data information carried in the message;

[0074] Extracting a database identifier and a key value dimension from the data information;

[0075] Determine the database corresponding to the database identifier as the target database;

[0076] Data tables that match both the key value dimension and the preset status are selected from the target database as the multiple data tables to be processed.

[0077] The data information includes, but is not limited to: a tag for indicating the database identifier, the database identifier, a tag for indicating the key value dimension, and the key value dimension, etc.

[0078] The key value dimension refers to the dimension corresponding to the primary key. For example, the key value dimension may refer to the number of characters of the key value corresponding to the primary key. For example, if the key value corresponding to the primary key is abcdef, then the key value dimension is 6.

[0079] The preset state refers to a state where data merging is not performed.

[0080] By parsing the message, since there is no need to parse the entire request, the parsing efficiency of the data information can be improved, the target database can be accurately located through the database identifier, and then the multiple data tables to be processed that need to be merged can be screened out from the target database through the key value dimension and the preset status.

[0081] S12, counting the number of fields of each of the merged fields in the multiple data tables to be processed.

[0082] In at least one embodiment of the present invention, the merged field refers to a field included in the plurality of data tables to be processed. For example, the merged field may be a date, business sales volume, and the like.

[0083] The field quantity refers to the number of the merged fields in the multiple data tables to be processed. For example, merged field A appears in data table 001 to be processed and data table 002 to be processed, and the field quantity of merged field A is 2.

[0084] In at least one embodiment of the present invention, the electronic device counting the number of fields of each of the merged fields in the plurality of data tables to be processed comprises:

[0085] Extracting a plurality of merged fields from each of the to-be-processed data tables;

[0086] Writing each of the merged fields into a preset query instruction to obtain a field query statement for each of the merged fields;

[0087] Query the multiple data tables to be processed based on the field query statement to obtain query results;

[0088] The number of data tables in the query result is counted to obtain the number of fields.

[0089] The preset query instruction refers to an SQL statement that does not contain a query object.

[0090] Each of the field query statements contains only a unique merge field.

[0091] The query result is a table of data to be processed that contains corresponding query fields.

[0092] The preset query instruction can quickly generate the field query statement, and then query the multiple data tables to be processed based on the field query statement, thereby improving the efficiency of generating the query result. Since the query result is a data table to be processed containing corresponding query fields, the number of fields can be accurately determined by counting the number of query results.

[0093] S13: Detecting the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields.

[0094] In at least one embodiment of the present invention, the generation accuracy refers to the generation accuracy of the multiple data tables to be processed for repeated fields, wherein the repeated fields refer to merge fields whose number of fields is greater than or equal to a preset number. The preset number is usually set to 2. Correspondingly, the repeated fields refer to merge fields whose number of fields is greater than 2, for example, if the number of fields of merge field A is 2, then merge field A is the repeated field.

[0095] In at least one embodiment of the present invention, the electronic device detecting, according to the number of fields, the generation accuracy of the merged fields by the multiple to-be-processed data tables comprises:

[0096] Determine the merged fields whose number of fields is greater than or equal to the preset number as repeated fields;

[0097] Filtering out a plurality of data tables to be tested from the plurality of data tables to be processed based on the repeated fields;

[0098] Obtaining a historical merge record of each of the data tables to be tested, and obtaining a merge accuracy of each of the historical merge records;

[0099] Calculating the generation accuracy of the repeated fields for each of the to-be-processed data tables according to the merging accuracy includes:

[0100]

[0101] Where y refers to the generation accuracy, x1, x2, ..., x n They respectively refer to the combined accuracy, and k refers to the preset adjustment ratio.

[0102] The data table to be tested refers to the data table to be processed that contains the repeated fields.

[0103] The historical merge records refer to the records of the data table to be tested participating in the merge.

[0104] The merging accuracy may be a numerical value identified by an auditing user for the historical merging record.

[0105] The preset adjustment ratio can be set according to actual needs, and the present invention will not elaborate on this.

[0106] By comparing the number of fields with the preset number, repeated fields can be accurately screened out from the merged fields, and then the multiple data tables to be tested can be screened out according to the repeated fields, avoiding the analysis of the data tables to be processed that do not contain the repeated fields, and improving the generation effectiveness of the generation accuracy. Furthermore, according to the historical merge records of each data table to be tested and the corresponding merge accuracy, the generation accuracy of the repeated fields for each data table to be processed can be accurately generated. At the same time, since there is no need to analyze all the merged fields, the detection efficiency of the generation accuracy can be improved.

[0107] S14: collecting business data from the multiple data tables to be processed according to the generation accuracy and the merged fields.

[0108] In at least one embodiment of the present invention, the business data includes data information of the repeated fields in the characteristic data table and data information of the single fields in the corresponding data table to be processed. The characteristic data table refers to the data table to be tested with the highest generation accuracy, and the single field refers to a merged field less than the preset number.

[0109] In at least one embodiment of the present invention, the electronic device collecting business data from the plurality of to-be-processed data tables according to the generation accuracy and the merged fields comprises:

[0110] Determine the data table to be tested with the highest generation accuracy as the characteristic data table;

[0111] Collecting data corresponding to the repeated field from the feature data table as the business data;

[0112] Determine the merged fields that are smaller than the preset number as a single field;

[0113] The data corresponding to the single field is collected from the to-be-processed data table containing the single field as the business data.

[0114] Through the above implementation, the business data can be collected by adopting a corresponding collection method based on the repeated field and the single field, thereby improving the comprehensiveness of the collection of the business data.

[0115] S15: Locate an output table from the configuration table based on the merged fields.

[0116] In at least one embodiment of the present invention, the configuration table stores mapping relationships between multiple fields and index tables of the fields.

[0117] The output table refers to the index table corresponding to all merged fields in the configuration table.

[0118] In at least one embodiment of the present invention, the electronic device locating the output table from the configuration table based on the merge field comprises:

[0119] Acquire an index table of each merged field from the configuration table;

[0120] Filter out the index table corresponding to all merge fields as the initial table;

[0121] If there are multiple initial tables, then obtain the running status of each initial table;

[0122] The initial table whose operating state is a configuration state is determined as the output table.

[0123] Among them, the initial table refers to the index table corresponding to all merged fields in the configuration table, for example, merged field B, merged field C, merged field D. If in the configuration table, the merged field B, the merged field C, and the merged field D all correspond to index table 1 and index table 2, then the initial table is: the index table 1 and the index table 2.

[0124] The configuration state refers to an initial table in an idle state, that is, the output table refers to an initial table to which no data is currently written.

[0125] The configuration table can be used to quickly filter out the initial table from the index table, and when there are multiple initial tables, the output table can be selected based on the operating status of the multiple initial tables. Since the output table is in the configuration state, there is no need to wait for other data to be written when writing the business data, thereby improving the data writing efficiency of the output table.

[0126] S16, writing the business data into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

[0127] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target data table, the above-mentioned target data table can also be stored in a node of a blockchain.

[0128] In at least one embodiment of the present invention, the preset type includes: numeric type, character type, etc.

[0129] The target data table refers to a data table generated after merging the data tables to be processed.

[0130] In at least one embodiment of the present invention, the electronic device writes the business data into the output table in parallel based on the primary key and the field type of the plurality of merged fields, and the target data table obtained includes:

[0131] Matching the business data corresponding to each of the merged fields with the setting data corresponding to the preset type, and determining the preset type corresponding to the setting data that successfully matches the business data as the field type;

[0132] Obtaining an execution thread corresponding to the field type from a preset thread pool;

[0133] Based on the primary key, the business data corresponding to the field type is written into the output table in parallel by the execution thread to obtain the target data table.

[0134] The preset thread pool stores a plurality of idle threads, and each idle thread has a corresponding processing type.

[0135] By matching the business data corresponding to each merged field with the setting data corresponding to the preset type, the field type can be accurately determined, and then based on the field type, the corresponding execution thread can be obtained from the preset thread pool, thereby improving the efficiency of the execution thread in writing the business data. In addition, by writing the business data in parallel through multiple execution threads, the efficiency of writing the business data can be further improved.

[0136] It can be seen from the above technical solutions that the present invention can accurately screen out the merged fields that need to be detected and analyzed by counting the number of fields of each merged field in the multiple data tables to be processed. By detecting the generation accuracy of the merged field by the multiple data tables to be processed, it is beneficial to accurately collect the business data corresponding to the merged field from the multiple data tables to be processed, thereby improving the generation accuracy of the target data table. In addition, based on the merged field, the output table can be located from the configuration table, and then the business data can be written to the output table in parallel according to the field type. Since the output table can be directly located without splicing the multiple data tables to be processed, the merging efficiency of the business data can be improved. At the same time, since the present invention writes the business data in parallel, the data table merging efficiency can be further improved.

[0137] like Figure 2, which is a functional module diagram of a preferred embodiment of the data table merging device of the present invention. The data table merging device 11 includes a receiving unit 110, an identification unit 111, a statistical unit 112, a detection unit 113, a collection unit 114, a positioning unit 115 and a writing unit 116. The module / unit referred to in the present invention refers to a series of computer-readable instruction segments that can be acquired by the processor 13 and can complete a fixed function, which is stored in the memory 12. In this embodiment, the functions of each module / unit will be described in detail in subsequent embodiments.

[0138] The receiving unit 110 receives a data table merging request.

[0139] In at least one embodiment of the present invention, the data table merge request can be triggered and generated by any business personnel. The data table merge request carries a tag for indicating the database identifier, the database identifier, a tag for indicating the key value dimension, and the key value dimension.

[0140] In at least one embodiment of the present invention, the receiving unit 110 performs authority authentication on the triggering user of the data table merge request. When the triggering user passes the authority authentication, the receiving unit 110 receives the data table merge request.

[0141] The identification unit 111 identifies a plurality of data tables to be processed according to the data table merging request, each of the data tables to be processed includes a primary key.

[0142] In at least one embodiment of the present invention, the multiple data tables to be processed refer to data tables that need to be merged.

[0143] In at least one embodiment of the present invention, the identifying unit 111 identifies multiple data tables to be processed according to the data table merge request, including:

[0144] Parsing the message of the data table merge request to obtain the data information carried in the message;

[0145] Extracting a database identifier and a key value dimension from the data information;

[0146] Determine the database corresponding to the database identifier as the target database;

[0147] Data tables that match both the key value dimension and the preset status are selected from the target database as the multiple data tables to be processed.

[0148] The data information includes, but is not limited to: a tag for indicating the database identifier, the database identifier, a tag for indicating the key value dimension, and the key value dimension, etc.

[0149] The key value dimension refers to the dimension corresponding to the primary key. For example, the key value dimension may refer to the number of characters of the key value corresponding to the primary key. For example, if the key value corresponding to the primary key is abcdef, then the key value dimension is 6.

[0150] The preset state refers to a state where data merging is not performed.

[0151] By parsing the message, since there is no need to parse the entire request, the parsing efficiency of the data information can be improved, the target database can be accurately located through the database identifier, and then the multiple data tables to be processed that need to be merged can be screened out from the target database through the key value dimension and the preset status.

[0152] The counting unit 112 counts the number of fields of each of the merged fields in the multiple data tables to be processed.

[0153] In at least one embodiment of the present invention, the merged field refers to a field included in the plurality of data tables to be processed. For example, the merged field may be a date, business sales volume, and the like.

[0154] The field quantity refers to the number of the merged fields in the multiple data tables to be processed. For example, merged field A appears in data table 001 to be processed and data table 002 to be processed, and the field quantity of merged field A is 2.

[0155] In at least one embodiment of the present invention, the counting unit 112 counting the number of fields of each of the merged fields in the plurality of data tables to be processed includes:

[0156] Extracting a plurality of merged fields from each of the to-be-processed data tables;

[0157] Writing each of the merged fields into a preset query instruction to obtain a field query statement for each of the merged fields;

[0158] Query the multiple data tables to be processed based on the field query statement to obtain query results;

[0159] The number of data tables in the query result is counted to obtain the number of fields.

[0160] The preset query instruction refers to an SQL statement that does not contain a query object.

[0161] Each of the field query statements contains only a unique merge field.

[0162] The query result is a table of data to be processed that contains corresponding query fields.

[0163] The preset query instruction can quickly generate the field query statement, and then query the multiple data tables to be processed based on the field query statement, thereby improving the efficiency of generating the query result. Since the query result is a data table to be processed containing corresponding query fields, the number of fields can be accurately determined by counting the number of query results.

[0164] The detection unit 113 detects the generation accuracy of the merged fields by the multiple to-be-processed data tables according to the number of fields.

[0165] In at least one embodiment of the present invention, the generation accuracy refers to the generation accuracy of the multiple data tables to be processed for repeated fields, wherein the repeated fields refer to merge fields whose number of fields is greater than or equal to a preset number. The preset number is usually set to 2. Correspondingly, the repeated fields refer to merge fields whose number of fields is greater than 2, for example, if the number of fields of merge field A is 2, then merge field A is the repeated field.

[0166] In at least one embodiment of the present invention, the detecting unit 113 detects the generation accuracy of the merged fields by the multiple to-be-processed data tables according to the number of fields, including:

[0167] Determine the merged fields whose number of fields is greater than or equal to the preset number as repeated fields;

[0168] Filtering out a plurality of data tables to be tested from the plurality of data tables to be processed based on the repeated fields;

[0169] Obtaining a historical merge record of each of the data tables to be tested, and obtaining a merge accuracy of each of the historical merge records;

[0170] Calculating the generation accuracy of the repeated fields for each of the to-be-processed data tables according to the merging accuracy includes:

[0171]

[0172] Where y refers to the generation accuracy, x1, x2, ..., x n They respectively refer to the combined accuracy, and k refers to the preset adjustment ratio.

[0173] The data table to be tested refers to the data table to be processed that contains the repeated fields.

[0174] The historical merge records refer to the records of the data table to be tested participating in the merge.

[0175] The merging accuracy may be a numerical value identified by an auditing user for the historical merging record.

[0176] The preset adjustment ratio can be set according to actual needs, and the present invention will not elaborate on this.

[0177] By comparing the number of fields with the preset number, repeated fields can be accurately screened out from the merged fields, and then the multiple data tables to be tested can be screened out according to the repeated fields, avoiding the analysis of the data tables to be processed that do not contain the repeated fields, and improving the generation effectiveness of the generation accuracy. Furthermore, according to the historical merge records of each data table to be tested and the corresponding merge accuracy, the generation accuracy of the repeated fields for each data table to be processed can be accurately generated. At the same time, since there is no need to analyze all the merged fields, the detection efficiency of the generation accuracy can be improved.

[0178] The collecting unit 114 collects business data from the multiple data tables to be processed according to the generation accuracy and the merge fields.

[0179] In at least one embodiment of the present invention, the business data includes data information of the repeated fields in the characteristic data table and data information of the single fields in the corresponding data table to be processed. The characteristic data table refers to the data table to be tested with the highest generation accuracy, and the single field refers to a merged field less than the preset number.

[0180] In at least one embodiment of the present invention, the collecting unit 114 collects the business data from the plurality of to-be-processed data tables according to the generation accuracy and the merged fields, including:

[0181] Determine the data table to be tested with the highest generation accuracy as the characteristic data table;

[0182] Collecting data corresponding to the repeated field from the feature data table as the business data;

[0183] Determine the merged fields that are smaller than the preset number as a single field;

[0184] The data corresponding to the single field is collected from the to-be-processed data table containing the single field as the business data.

[0185] Through the above implementation, the business data can be collected by adopting a corresponding collection method based on the repeated field and the single field, thereby improving the comprehensiveness of the collection of the business data.

[0186] The positioning unit 115 locates the output table from the configuration table based on the merge field.

[0187] In at least one embodiment of the present invention, the configuration table stores mapping relationships between multiple fields and index tables of the fields.

[0188] The output table refers to the index table corresponding to all merged fields in the configuration table.

[0189] In at least one embodiment of the present invention, the locating unit 115 locates the output table from the configuration table based on the merge field, including:

[0190] Acquire an index table of each merged field from the configuration table;

[0191] Filter out the index table corresponding to all merge fields as the initial table;

[0192] If there are multiple initial tables, then obtain the running status of each initial table;

[0193] The initial table whose operating state is a configuration state is determined as the output table.

[0194] Among them, the initial table refers to the index table corresponding to all merged fields in the configuration table, for example, merged field B, merged field C, merged field D. If in the configuration table, the merged field B, the merged field C, and the merged field D all correspond to index table 1 and index table 2, then the initial table is: the index table 1 and the index table 2.

[0195] The configuration state refers to an initial table in an idle state, that is, the output table refers to an initial table to which no data is currently written.

[0196] The configuration table can be used to quickly filter out the initial table from the index table, and when there are multiple initial tables, the output table can be selected based on the operating status of the multiple initial tables. Since the output table is in the configuration state, there is no need to wait for other data to be written when writing the business data, thereby improving the data writing efficiency of the output table.

[0197] The writing unit 116 writes the business data into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

[0198] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned target data table, the above-mentioned target data table can also be stored in a node of a blockchain.

[0199] In at least one embodiment of the present invention, the preset type includes: numeric type, character type, etc.

[0200] The target data table refers to a data table generated after merging the data tables to be processed.

[0201] In at least one embodiment of the present invention, the writing unit 116 writes the business data to the output table in parallel based on the primary key and the field type of the plurality of merge fields, and the target data table obtained includes:

[0202] Matching the business data corresponding to each of the merged fields with the setting data corresponding to the preset type, and determining the preset type corresponding to the setting data that successfully matches the business data as the field type;

[0203] Obtaining an execution thread corresponding to the field type from a preset thread pool;

[0204] Based on the primary key, the business data corresponding to the field type is written into the output table in parallel by the execution thread to obtain the target data table.

[0205] The preset thread pool stores a plurality of idle threads, and each of the idle threads has a corresponding processing type.

[0206] By matching the business data corresponding to each merged field with the setting data corresponding to the preset type, the field type can be accurately determined, and then based on the field type, the corresponding execution thread can be obtained from the preset thread pool, thereby improving the efficiency of the execution thread in writing the business data. In addition, by writing the business data in parallel through multiple execution threads, the efficiency of writing the business data can be further improved.

[0207] It can be seen from the above technical solutions that the present invention can accurately screen out the merged fields that need to be detected and analyzed by counting the number of fields of each merged field in the multiple data tables to be processed. By detecting the generation accuracy of the merged field by the multiple data tables to be processed, it is beneficial to accurately collect the business data corresponding to the merged field from the multiple data tables to be processed, thereby improving the generation accuracy of the target data table. In addition, based on the merged field, the output table can be located from the configuration table, and then the business data can be written to the output table in parallel according to the field type. Since the output table can be directly located without splicing the multiple data tables to be processed, the merging efficiency of the business data can be improved. At the same time, since the present invention writes the business data in parallel, the data table merging efficiency can be further improved.

[0208] like Figure 3 FIG. 1 is a schematic diagram of the structure of an electronic device of a preferred embodiment of the data table merging method of the present invention.

[0209] In one embodiment of the present invention, the electronic device 1 includes, but is not limited to, a memory 12, a processor 13, and computer-readable instructions stored in the memory 12 and executable on the processor 13, such as a data table merging program.

[0210] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 1 and does not constitute a limitation on the electronic device 1. The electronic device 1 may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the electronic device 1 may also include input and output devices, network access devices, buses, etc.

[0211] The processor 13 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 13 is the computing core and control center of the electronic device 1, and uses various interfaces and lines to connect various parts of the entire electronic device 1, and executes the operating system of the electronic device 1 and various installed applications, program codes, etc.

[0212] Exemplarily, the computer-readable instructions may be divided into one or more modules / units, which are stored in the memory 12 and executed by the processor 13 to complete the present invention. The one or more modules / units may be a series of computer-readable instruction segments capable of completing specific functions, which are used to describe the execution process of the computer-readable instructions in the electronic device 1. For example, the computer-readable instructions may be divided into a receiving unit 110, an identification unit 111, a statistics unit 112, a detection unit 113, a collection unit 114, a positioning unit 115, and a writing unit 116.

[0213] The memory 12 can be used to store the computer-readable instructions and / or modules, and the processor 13 implements various functions of the electronic device 1 by running or executing the computer-readable instructions and / or modules stored in the memory 12, and calling the data stored in the memory 12. The memory 12 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. The memory 12 can include non-volatile and volatile memories, such as: a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other storage devices.

[0214] The memory 12 may be an external memory and / or an internal memory of the electronic device 1. Furthermore, the memory 12 may be a memory in a physical form, such as a memory stick, a TF card (Trans-flash Card), and the like.

[0215] If the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also instruct the relevant hardware to complete it through computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium. When the computer-readable instructions are executed by the processor, the steps of the above-mentioned various method embodiments can be implemented.

[0216] The computer-readable instructions include computer-readable instruction codes, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer-readable instruction code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM).

[0217] The blockchain referred to in the present invention is a new application mode of computer technologies such as distributed data table merging, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0218] Combination Figure 1 The memory 12 in the electronic device 1 stores computer-readable instructions to implement a data table merging method, and the processor 13 can execute the computer-readable instructions to implement:

[0219] Receive data table merge requests;

[0220] Identifying a plurality of data tables to be processed according to the data table merge request, each of the data tables to be processed comprising a primary key;

[0221] Counting the number of fields of each of the merged fields in the multiple data tables to be processed;

[0222] Detecting the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields;

[0223] Collecting business data from the plurality of data tables to be processed according to the generation accuracy and the merged fields;

[0224] Locating an output table from the configuration table based on the merge fields;

[0225] The business data is written into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

[0226] Specifically, the specific implementation method of the processor 13 for the above-mentioned computer readable instructions can refer to Figure 1 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0227] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0228] The computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions are used to implement the following steps when executed by the processor 13:

[0229] Receive data table merge requests;

[0230] Identifying a plurality of data tables to be processed according to the data table merge request, each of the data tables to be processed comprising a primary key;

[0231] Counting the number of fields of each of the merged fields in the multiple data tables to be processed;

[0232] Detecting the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields;

[0233] Collecting business data from the plurality of data tables to be processed according to the generation accuracy and the merged fields;

[0234] Locating an output table from the configuration table based on the merge fields;

[0235] The business data is written into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

[0236] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0238] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0239] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units or devices mentioned can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for merging data tables. It is characterized in that The data table merging method comprises: Receive data table merge requests; Identifying a plurality of data tables to be processed according to the data table merge request, each of the data tables to be processed comprising a primary key; Counting the number of fields of each merged field in the plurality of data tables to be processed, comprising: extracting a plurality of the merged fields from each of the data tables to be processed; writing each of the merged fields into a preset query instruction to obtain a field query statement for each of the merged fields; querying the plurality of data tables to be processed based on the field query statement to obtain a query result; counting the number of data tables in the query result to obtain the number of fields; Detecting the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields, including: determining the merged fields whose number of fields is greater than or equal to a preset number as repeated fields; screening out multiple data tables to be tested from the multiple data tables to be processed based on the repeated fields; obtaining the historical merge records of each of the data tables to be tested, and obtaining the merge accuracy of each of the historical merge records; calculating the generation accuracy of the repeated fields by each of the data tables to be processed according to the merge accuracy, including: ;in, refers to the generation accuracy, are the combined accuracy, Refers to the preset adjustment ratio; Collecting business data from the plurality of data tables to be processed according to the generation accuracy and the merged fields; Locating the output table from the configuration table based on the merged field includes: obtaining the index table of each merged field from the configuration table; screening out the index tables corresponding to all the merged fields as the initial table; if there are multiple initial tables, obtaining the running status of each initial table; determining the initial table whose running status is the configuration status as the output table; The business data is written into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

2. The data table merging method according to claim 1, It is characterized in that The identifying a plurality of data tables to be processed according to the data table merging request comprises: Parsing the message of the data table merge request to obtain the data information carried in the message; Extracting a database identifier and a key value dimension from the data information; Determine the database corresponding to the database identifier as the target database; Data tables that match both the key value dimension and the preset status are selected from the target database as the multiple data tables to be processed.

3. The data table merging method according to claim 1, It is characterized in that The collecting of business data from the plurality of to-be-processed data tables according to the generation accuracy and the merged fields comprises: Determine the data table to be tested with the highest generation accuracy as the characteristic data table; Collecting data corresponding to the repeated field from the feature data table as the business data; Determine the merged fields that are smaller than the preset number as a single field; The data corresponding to the single field is collected from the to-be-processed data table containing the single field as the business data.

4. The data table merging method according to claim 1, It is characterized in that The business data is written into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table including: Matching the business data corresponding to each of the merged fields with the setting data corresponding to the preset type, and determining the preset type corresponding to the setting data that successfully matches the business data as the field type; Obtaining an execution thread corresponding to the field type from a preset thread pool; Based on the primary key, the business data corresponding to the field type is written into the output table in parallel by the execution thread to obtain the target data table.

5. A data table merging device, It is characterized in that The data table merging device comprises: A receiving unit, used for receiving a data table merging request; an identification unit, configured to identify a plurality of to-be-processed data tables according to the data table merging request, each of the to-be-processed data tables comprising a primary key; A statistical unit is used to count the number of fields of each merged field in the multiple data tables to be processed, including: extracting multiple merged fields from each of the data tables to be processed; writing each of the merged fields into a preset query instruction to obtain a field query statement for each of the merged fields; querying the multiple data tables to be processed based on the field query statement to obtain a query result; counting the number of data tables in the query result to obtain the number of fields; A detection unit is used to detect the generation accuracy of the merged fields by the multiple data tables to be processed according to the number of fields, including: determining the merged fields whose number of fields is greater than or equal to a preset number as repeated fields; screening out multiple data tables to be tested from the multiple data tables to be processed based on the repeated fields; obtaining the historical merge records of each of the data tables to be tested, and obtaining the merge accuracy of each of the historical merge records; calculating the generation accuracy of each of the data tables to be processed for the repeated fields according to the merge accuracy, including: ;in, refers to the generation accuracy, are the combined accuracy, Refers to the preset adjustment ratio; A collection unit, configured to collect business data from the plurality of data tables to be processed according to the generation accuracy and the merged fields; A positioning unit, used to locate the output table from the configuration table based on the merged field, including: obtaining the index table of each merged field from the configuration table; screening out the index tables corresponding to all the merged fields as the initial table; if there are multiple initial tables, obtaining the running status of each initial table; determining the initial table whose running status is the configuration status as the output table; A writing unit is used to write the business data into the output table in parallel based on the primary key and the field type of the plurality of merged fields to obtain a target data table.

6. An electronic device, It is characterized in that The electronic device comprises: a memory storing computer-readable instructions; and A processor executes computer-readable instructions stored in the memory to implement the data table merging method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, Features: The computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions are executed by a processor in an electronic device to implement the data table merging method as described in any one of claims 1 to 4.

Citation Information

Patent Citations

  • Data table merging processing method and device

    CN110347683A

  • Data query method and system and computer readable storage medium

    CN112506964A

  • Method and device for merging various business index data in real time and computer equipment

    CN113220707A

  • Database part creation, merge and reuse

    US20090132576A1