Data updating method, device, equipment and storage medium of distributed database

Abstract

The present invention discloses a data updating method for a distributed database, wherein a field extraction operation is performed on each table to be processed to obtain each initial field to be converted; it is determined whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new and old field comparison table; if so, the comparison field corresponding to the initial field to be converted is determined according to the target comparison relationship, and the comparison field is determined as the target field; if not, the initial field to be converted is determined as the target field; taking the table to be processed as a unit, a summary calculation is performed on each initial field to be converted and each target field, and the target field to be converted is determined according to the summary calculation result; a deletion script is called to perform a deletion operation on each target field to be converted, and a write script is called to perform a write operation on the target field corresponding to each target field to be converted. The present invention improves the data updating efficiency and simplifies the data updating process. The present invention also discloses a device, an equipment and a storage medium, which have corresponding technical effects.

Description

Technical Field

[0001] The present invention relates to the field of distributed storage technology, and in particular to a data updating method, device, equipment and computer-readable storage medium for a distributed database. Background Art

[0002] During the operation of the tax information system, due to policy adjustments, agency adjustments, administrative division adjustments, etc., some fields of the generated business data need to be adjusted, such as the tax authority code, administrative division code, etc.

[0003] Under current circumstances, update scripts are written based on the fields that need to be changed, and the scripts are executed through operation and maintenance work to update the data. Since the distributed relational data is operated around the sub-library and sub-table keys, whether the sub-library and sub-table keys are specified during operation has a great impact on performance. If the field to be updated is not a sub-library and sub-table key field, then the field is directly used as the update condition to perform the data update operation. During the data update process, a full table scan will be generated, which will not only reduce the efficiency of the operation and maintenance process, but also have a serious impact on the performance of the production database, which may drag down the database and affect the production business processing. In the case of sub-library and sub-table keys, since the sub-library and sub-table keys are involved, it is necessary to manually generate deletion and writing scripts, first delete the existing data, update the values ​​of the sub-library and sub-table fields, and then write the data to redistribute the data to the new sub-library. Since the amount of data involved is huge, and the operation and maintenance process involves the deletion of the original data, the new data is organized based on the original data. This process is very cumbersome and error-prone, and the workload of manual participation is large, and the processing efficiency is low.

[0004] To sum up, how to effectively solve the problems of full table scanning affecting production business processing, low efficiency of operation and maintenance process, cumbersome and error-prone process of manually generating deletion and writing scripts, and low processing efficiency are problems that technical personnel in this field urgently need to solve. Summary of the invention

[0005] The object of the present invention is to provide a data updating method for a distributed database, which avoids full table scanning when updating key fields of non-sub-library and sub-tables, improves data updating efficiency, simplifies the data updating process, and improves the accuracy of data updating; another object of the present invention is to provide a data updating device, equipment and computer-readable storage medium for a distributed database.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] A data updating method for a distributed database, comprising:

[0008] Perform field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted;

[0009] For each initial field to be converted, determining whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table;

[0010] If yes, determining the reference field corresponding to the initial field to be converted according to the target reference relationship, and determining the reference field as the target field;

[0011] If not, determining the initial field to be converted as the target field;

[0012] Taking the table to be processed as a unit, performing summary calculation on each of the initial fields to be converted, performing summary calculation on each of the target fields, and determining the target field to be converted according to the summary calculation result;

[0013] The pre-generated deletion script is called to perform a deletion operation on each of the target fields to be converted, and the pre-generated writing script is called to perform a writing operation on the target fields respectively corresponding to each of the target fields to be converted.

[0014] In a specific implementation of the present invention, taking the table to be processed as a unit, performing summary calculation on each of the initial fields to be converted, performing summary calculation on each of the target fields, and determining the target field to be converted according to the summary calculation result, including:

[0015] Merge the initial to-be-converted fields in each to-be-processed table in a preset order, and perform summary calculation on the merged result to obtain a first summary field;

[0016] Merge the target fields corresponding to the initial to-be-converted fields in each to-be-processed table according to the preset order, and perform summary calculation on the merged result to obtain a second summary field;

[0017] For each table to be processed, determining whether there is a corresponding relationship between the first summary field and the second summary field in a pre-established conversion intermediate table;

[0018] If so, the to-be-processed table in the conversion intermediate table in which the correspondence between the first summary field and the second summary field exists is determined as the target to-be-processed table, and each of the initial to-be-converted fields in the target to-be-processed table is determined as the target to-be-converted field.

[0019] In a specific implementation of the present invention, a field extraction operation is performed on the data to be processed in each table to be processed, including:

[0020] Parse the received field extraction request to obtain the fields to be extracted;

[0021] Determine whether the field to be extracted is a sub-library or sub-table field;

[0022] If so, each of the to-be-processed tables is determined according to the fields of the sub-library and sub-table, and a field extraction operation is performed on the to-be-processed data in each of the to-be-processed tables.

[0023] In a specific implementation of the present invention, after calling a pre-generated deletion script to perform a deletion operation on each of the target fields to be converted, and calling a pre-generated write script to perform a write operation on the target fields respectively corresponding to each of the target fields to be converted, the method further includes:

[0024] Put the deletion script and the writing script into a script table to be executed in a production library supporting business handling;

[0025] Exporting the deletion script and the writing script from the to-be-executed script table;

[0026] The deletion script is executed by using a preset operation and maintenance tool to delete each of the target fields to be converted in the production library, and the writing script is executed to write the target fields corresponding to each of the target fields to be converted into the production library.

[0027] In a specific implementation of the present invention, when it is determined that the field to be extracted is not a sub-library or sub-table field, the method further includes:

[0028] Determine whether the field to be extracted exists in the preset range extraction table;

[0029] If so, searching the preset range extraction table for the target field extraction range corresponding to the field to be extracted;

[0030] Each of the to-be-processed tables is determined according to the target field extraction range, and a field extraction operation is performed on the to-be-processed data in each of the to-be-processed tables.

[0031] In a specific implementation of the present invention, after calling a pre-generated deletion script to perform a deletion operation on each of the target fields to be converted, and calling a pre-generated write script to perform a write operation on the target fields respectively corresponding to each of the target fields to be converted, the method further includes:

[0032] Obtain the target sub-library sub-table keys corresponding to the to-be-processed tables to which the target to-be-converted fields belong and the target primary keys corresponding to the target to-be-converted fields;

[0033] Searching for each target field to be converted from a production database supporting business handling according to each target sub-library sub-table key and each target primary key;

[0034] A deletion operation is performed on each of the target fields to be converted in the production library, and a target field corresponding to each of the target fields to be converted is written into the production library.

[0035] In a specific implementation of the present invention, after performing a field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted, the method further includes:

[0036] Each of the initial fields to be converted is backed up to a preset backup table, and a backup batch number corresponding to the current backup is generated.

[0037] A data updating device for a distributed database, comprising:

[0038] A field extraction module is used to perform field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted;

[0039] A first judgment module is used to judge, for each initial field to be converted, whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table;

[0040] A first target field determination module is used to determine the reference field corresponding to the initial field to be converted according to the target reference relationship when it is determined that there is a target reference relationship corresponding to the initial field to be converted in the pre-established new-old field reference table, and determine the reference field as the target field;

[0041] A second target field determination module, configured to determine the initial field to be converted as the target field when it is determined that there is no target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table;

[0042] A target field to be converted determination module is used to perform summary calculation on each of the initial fields to be converted and on each of the target fields, taking the table to be processed as a unit, and determine the target field to be converted according to the summary calculation result;

[0043] The field deletion and writing module is used to call the pre-generated deletion script to perform a deletion operation on each of the target fields to be converted, and call the pre-generated writing script to perform a writing operation on the target fields respectively corresponding to each of the target fields to be converted.

[0044] A data updating device for a distributed database, comprising:

[0045] Memory for storing computer programs;

[0046] A processor is used to implement the steps of the distributed database data updating method as described above when executing the computer program.

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the distributed database data update method as described above.

[0048] The data updating method of the distributed database provided by the present invention performs field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted; for each initial field to be converted, it is judged whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table; if so, the comparison field corresponding to the initial field to be converted is determined according to the target comparison relationship, and the comparison field is determined as the target field; if not, the initial field to be converted is determined as the target field; taking the table to be processed as a unit, a summary calculation is performed on each initial field to be converted, and a summary calculation is performed on each target field, and a target field to be converted is determined according to the summary calculation result; a pre-generated deletion script is called to perform a deletion operation on each target field to be converted, and a pre-generated writing script is called to perform a writing operation on the target fields corresponding to each target field to be converted.

[0049] It can be seen from the above technical solution that by presetting the old and new field comparison table storing the comparison relationship between the initial fields to be converted and the comparison fields, after extracting each initial field to be converted, the target field corresponding to each initial field to be converted is determined according to the old and new field comparison table. By taking the table to be processed as a unit, a summary calculation is performed on each initial field to be converted, and a summary calculation is performed on each target field, and the final target field to be converted is determined according to the summary calculation result. And the pre-generated deletion script is automatically called to delete each target field to be converted, and the pre-generated write script is automatically called to write the corresponding target field. The present invention avoids full table scanning when updating non-sub-library and sub-table key fields, improves data update efficiency, and greatly reduces the impact on production business processing. By automatically generating deletion scripts and write scripts to delete and write fields, the data update process is greatly simplified and the accuracy of data update is improved.

[0050] Correspondingly, the present invention also provides a distributed database data update device, equipment and computer-readable storage medium corresponding to the above-mentioned distributed database data update method, which has the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a method for updating data in a distributed database according to an embodiment of the present invention;

[0053] Figure 2 Another implementation flow chart of the data updating method of the distributed database in the embodiment of the present invention;

[0054] Figure 3 It is a structural block diagram of a data updating device for a distributed database in an embodiment of the present invention;

[0055] Figure 4 A structural block diagram of a data updating device for a distributed database in an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of the specific structure of a data updating device for a distributed database provided in this embodiment. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0058] See also Figure 1 , Figure 1 This is a flow chart of an implementation of a method for updating data in a distributed database in an embodiment of the present invention. The method may include the following steps:

[0059] S101: Performing a field extraction operation on the data to be processed in each table to be processed to obtain each initial field to be converted.

[0060] Determine each table to be processed according to the data update request. When data in the data table in the distributed database needs to be updated, perform field extraction on the data to be processed in each table to be processed to obtain each initial field to be converted. For example, when the field to be updated is a sub-library or sub-table field, accurate field extraction can be performed directly based on the sub-library or sub-table field. If the field to be updated is not a sub-library or sub-table field, field extraction can be performed according to a predetermined range.

[0061] S102: for each initial field to be converted, determine whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table. If yes, execute step S103; if not, execute step S104.

[0062] A new and old field comparison table storing the comparison relationship between the initial to-be-converted field and the comparison field is pre-set. After the field extraction operation is performed on the to-be-processed data in each to-be-processed table to obtain each initial to-be-converted field, it is determined for each initial to-be-converted field whether there is a target comparison relationship corresponding to the initial to-be-converted field in the pre-established new and old field comparison table. If yes, step S103 is executed, and if not, step S104 is executed.

[0063] S103: Determine a reference field corresponding to the initial field to be converted according to the target reference relationship, and determine the reference field as the target field.

[0064] When it is determined that a target comparison relationship corresponding to a certain initial field to be converted exists in the pre-established new-old field comparison table, a comparison field corresponding to the initial field to be converted is determined according to the target comparison relationship, and the comparison field is determined as the target field.

[0065] S104: Determine the initial field to be converted as the target field.

[0066] When it is determined that there is no target comparison relationship corresponding to a certain initial field to be converted in the pre-established new-old field comparison table, the initial field to be converted is determined as the target field.

[0067] S105: Taking the table to be processed as a unit, perform summary calculation on each initial field to be converted, perform summary calculation on each target field, and determine the target field to be converted according to the summary calculation result.

[0068] After determining the target fields corresponding to the initial fields to be converted, the summary calculation is performed on the initial fields to be converted and the target fields to obtain the summary calculation results, taking the table to be processed as a unit. Thus, the comprehensive summary calculation results of the initial fields to be converted and the comprehensive summary calculation results of the target fields in each table to be processed are obtained, and the target fields to be converted that really need to be updated are obtained by comparing the comprehensive summary calculation results of the initial fields to be converted and the comprehensive summary calculation results of the target fields.

[0069] S106: Calling a pre-generated deletion script to perform a deletion operation on each target field to be converted, and calling a pre-generated write script to perform a write operation on the target fields respectively corresponding to each target field to be converted.

[0070] Generate a delete script and write script in advance. After determining the target fields to be converted based on the summary calculation results, call the pre-generated delete script to delete each target field to be converted, and call the pre-generated write script to write the target fields corresponding to each target field to be converted. This achieves automatic and accurate update of database data, reduces the workload of manual participation, improves data update efficiency, and reduces the impact on database performance.

[0071] It can be seen from the above technical solution that by presetting the old and new field comparison table storing the comparison relationship between the initial fields to be converted and the comparison fields, after extracting each initial field to be converted, the target field corresponding to each initial field to be converted is determined according to the old and new field comparison table. By taking the table to be processed as a unit, a summary calculation is performed on each initial field to be converted, and a summary calculation is performed on each target field, and the final target field to be converted is determined according to the summary calculation result. And the pre-generated deletion script is automatically called to delete each target field to be converted, and the pre-generated write script is automatically called to write the corresponding target field. The present invention avoids full table scanning when updating non-sub-library and sub-table key fields, improves data update efficiency, and greatly reduces the impact on production business processing. By automatically generating deletion scripts and write scripts to delete and write fields, the data update process is greatly simplified and the accuracy of data update is improved.

[0072] It should be noted that, based on the above embodiment, the embodiment of the present invention also provides corresponding improved solutions. In the subsequent embodiments, the same steps or corresponding steps as those in the above embodiment can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be repeated one by one in the following improved embodiments.

[0073] See also Figure 2 , Figure 2 Another implementation flow chart of the method for updating data in a distributed database in an embodiment of the present invention is shown below. The method may include the following steps:

[0074] S201: Parse the received field extraction request to obtain the fields to be extracted.

[0075] When it is necessary to update data in a data table in a distributed database, the request end generates a field extraction request and sends the field extraction request to the data update management center. The data update management center receives the field extraction request and parses the received field extraction request to obtain the fields to be extracted.

[0076] S202: Determine whether the field to be extracted is a sub-library or sub-table field. If so, execute step S203; if not, execute step S204.

[0077] After parsing to obtain the fields to be extracted, determine whether the fields to be extracted are sub-library and sub-table fields. If so, it means that the fields can be accurately extracted according to the sub-library and sub-table fields, and execute step S203. If not, it means that the fields cannot be accurately extracted according to the sub-library and sub-table fields, and execute step S204.

[0078] S203: Determine each table to be processed according to the fields of the sub-library and sub-table, and perform field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted.

[0079] When it is determined that the field to be extracted is a sub-library or sub-table field, it means that the field can be accurately extracted according to the sub-library or sub-table field, each table to be processed is determined according to the sub-library or sub-table field, and the field extraction operation is performed on the data to be processed in each table to be processed to obtain the initial fields to be converted, thereby achieving accurate extraction of the fields and improving the efficiency of field extraction.

[0080] S204: Determine whether there are fields to be extracted in the preset range extraction table. If yes, execute step S205. If not, perform full table extraction to obtain the initial fields to be converted.

[0081] An extraction table storing the correspondence between each field and the extraction range of each field is pre-established. When it is determined that the field to be extracted is not a sub-library or sub-table field, it means that the field cannot be accurately extracted according to the sub-library or sub-table field. It is determined whether the field to be extracted exists in the preset range extraction table. If so, it means that the field can be extracted according to the range, and step S205 is executed. If not, it means that the field cannot be extracted according to the range, and the entire table is extracted to obtain the initial fields to be converted.

[0082] S205: Searching for a target field extraction range corresponding to the field to be extracted from a preset range extraction table.

[0083] When it is determined that there are fields to be extracted in the preset range extraction table, it means that you can extract fields by range, and find the target field extraction range corresponding to the field to be extracted from the preset range extraction table. For example, when you need to update the tax authority code, you can extract the tax authority code according to the province range.

[0084] S206: Determine each table to be processed according to the target field extraction range, perform field extraction operations on the data to be processed in each table to be processed, and obtain each initial field to be converted.

[0085] After finding the target field extraction range corresponding to the field to be extracted from the preset range extraction table, determine each table to be processed according to the target field extraction range, perform field extraction operations on the data to be processed in each table to be processed, and obtain each initial field to be converted. By extracting fields by range, it avoids the need to extract the entire table, thus improving data extraction efficiency.

[0086] S207: Back up each initial field to be converted to a preset backup table, and generate a backup batch number corresponding to the current backup.

[0087] The backup table is pre-set. After extracting each initial field to be converted, each initial field to be converted is backed up to the preset backup table, and a backup batch number corresponding to the current backup is generated. By backing up each initial field to be converted, subsequent data tracking is convenient, and by setting the backup batch number, subsequent quick search of the backup field is convenient.

[0088] S208: for each initial field to be converted, determine whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table. If yes, execute step S209; if not, execute step S210.

[0089] S209: Determine the reference field corresponding to the initial field to be converted according to the target reference relationship, and determine the reference field as the target field.

[0090] S210: Determine the initial field to be converted as the target field.

[0091] S211: Merge the initial to-be-converted fields in each to-be-processed table in a preset order, and perform summary calculation on the merged result to obtain a first summary field.

[0092] The merging order of the initial fields to be converted in each table to be processed is preset, and the initial fields to be converted in each table to be processed are merged according to the preset order, and the digest of the merged result is calculated using the message digest algorithm (Message Digest Algorithm 5, MD5) to obtain the first digest field.

[0093] S212: merging the target fields corresponding to the initial to-be-converted fields in each to-be-processed table in a preset order, and performing digest calculation on the merging result using a message digest algorithm to obtain a second digest field.

[0094] The target fields corresponding to the initial to-be-converted fields in each to-be-processed table are merged in a preset order, and a summary calculation is performed on the merged result to obtain a second summary field.

[0095] It should be noted that the first and second in the first summary field and the second summary field are only used to distinguish the result of the summary calculation after merging the initial fields to be converted from the result of the summary calculation after merging the target field, and there is no order or size distinction.

[0096] S213: for each table to be processed, determine whether there is a corresponding relationship between the first summary field and the second summary field in the pre-established conversion intermediate table. If yes, execute step S214; if not, output a prompt message that there is no field to be converted that meets the preset requirements.

[0097] After merging the initial fields to be converted and performing summary calculation on the merged result to obtain the first summary field, and merging the target fields and performing summary calculation on the merged result to obtain the second summary field, it is determined for each table to be processed whether there is a corresponding relationship between the first summary field and the second summary field in the pre-established conversion intermediate table. If so, it means that it is necessary to perform a field conversion operation on the initial fields to be converted in the processing table, and step S214 is executed. If not, it means that it is not necessary to perform a field conversion operation on the initial fields to be converted in the processing table, and a prompt message is output that there is no field to be converted that meets the preset requirements.

[0098] In addition, each time a data update task is received, the conversion intermediate table will be initialized according to the current update task, thereby clearing the relevant data content of the previous data update to avoid interference with the current data update.

[0099] S214: Determine the to-be-processed table in the conversion intermediate table where the correspondence between the first summary field and the second summary field exists as the target to-be-processed table, and determine each initial to-be-converted field in the target to-be-processed table as the target to-be-converted field.

[0100] When it is determined that there is a correspondence between the first summary field and the second summary field in the conversion intermediate table pre-established for a certain table to be processed, it means that it is necessary to perform a field conversion operation on the initial field to be converted in the processing table, and the table to be processed in which there is a correspondence between the first summary field and the second summary field in the conversion intermediate table is determined as the target table to be processed, and each initial field to be converted in the target table to be processed is determined as the target field to be converted.

[0101] S215: Calling a pre-generated deletion script to perform a deletion operation on each target field to be converted, and calling a pre-generated write script to perform a write operation on the target fields respectively corresponding to each target field to be converted.

[0102] S216: When the field to be extracted is a sub-library or sub-table field, the deletion script and the writing script are put into the script table to be executed in the production library supporting business processing.

[0103] A table of scripts to be executed is pre-established in the production database supporting business processing. When the field to be extracted is a sub-database or sub-table field, the deletion script and the writing script are put into the table of scripts to be executed in the production database supporting business processing.

[0104] It should be noted that if the target field to be converted is not a sharded database or table key, when updating data, the value of the target field can be directly used to replace the value of the target field to be converted to achieve rapid data update; if the target field to be converted is a sharded database or table key, you need to first call the pre-generated deletion script to delete the target field to be converted, and then call the pre-generated write script to write to the target field corresponding to the target field to be converted.

[0105] S217: Export the delete script and write script from the script table to be executed.

[0106] After the deletion script and the writing script are put into the script table to be executed in the production library supporting business processing, the deletion script and the writing script are exported from the script table to be executed.

[0107] S218: Use the preset operation and maintenance tool to execute a deletion script to delete each target field to be converted in the production library, and execute a writing script to write the target fields corresponding to each target field to be converted into the production library.

[0108] An operation and maintenance tool for script execution is pre-set in the production library. After the deletion script and the writing script are exported from the script table to be executed, the preset operation and maintenance tool is used to execute the deletion script to delete each target field to be converted in the production library, and the writing script is executed to write the target fields corresponding to each target field to be converted into the production library.

[0109] S219: When there are fields to be extracted in the preset range extraction table, the target sub-library and sub-table keys corresponding to the to-be-processed tables to which the target fields to be converted belong and the target primary keys corresponding to the target fields to be converted are obtained.

[0110] Each field in the table to be processed has a unique primary key. When there are fields to be extracted in the preset range extraction table, the target sub-library and sub-table keys corresponding to the table to be processed to which each target field to be converted belongs and the target primary keys corresponding to each target field to be converted are obtained.

[0111] S220: Searching for each target field to be converted from a production database supporting business processing according to each target sub-library sub-table key and each target primary key.

[0112] After obtaining the target sub-library sub-table keys corresponding to the to-be-processed tables to which the target fields to be converted belong and the target primary keys corresponding to the target fields to be converted, search for the target fields to be converted from the production database that supports business processing based on the target sub-library sub-table keys and the target primary keys. By using the target sub-library sub-table keys and the target primary keys to search for the corresponding fields in the production database, the efficiency of field search is greatly improved, and the update speed can still be guaranteed even when the data volume is large.

[0113] S221: Delete each target field to be converted in the production database, and write the target field corresponding to each target field to be converted into the production database.

[0114] After finding each target field to be converted from the production database supporting business processing according to each target sub-database sub-table key and each target primary key, delete each target field to be converted in the production database, and write the target field corresponding to each target field to be converted into the production database. This realizes the rapid deletion of the target field to be converted in the production database and the rapid writing of the target field.

[0115] Corresponding to the above method embodiment, the present invention further provides a data updating device for a distributed database. The data updating device for a distributed database described below and the data updating method for a distributed database described above can refer to each other.

[0116] See also Figure 3 , Figure 3 This is a structural block diagram of a data updating device for a distributed database in an embodiment of the present invention. The device may include:

[0117] A field extraction module 31 is used to perform field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted;

[0118] The first judgment module 32 is used to judge, for each initial field to be converted, whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table;

[0119] A first target field determination module 33 is used to determine the reference field corresponding to the initial field to be converted according to the target reference relationship when it is determined that there is a target reference relationship corresponding to the initial field to be converted in the pre-established new-old field reference table, and determine the reference field as the target field;

[0120] A second target field determination module 34 is used to determine the initial field to be converted as the target field when it is determined that there is no target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table;

[0121] The target to-be-converted field determination module 35 is used to perform summary calculation on each initial to-be-converted field and perform summary calculation on each target field in the to-be-processed table, and determine the target to-be-converted field according to the summary calculation result;

[0122] The field deletion and writing module 36 is used to call the pre-generated deletion script to perform a deletion operation on each target field to be converted, and call the pre-generated writing script to perform a writing operation on the target field corresponding to each target field to be converted.

[0123] It can be seen from the above technical solution that by presetting the old and new field comparison table storing the comparison relationship between the initial fields to be converted and the comparison fields, after extracting each initial field to be converted, the target field corresponding to each initial field to be converted is determined according to the old and new field comparison table. By taking the table to be processed as a unit, a summary calculation is performed on each initial field to be converted, and a summary calculation is performed on each target field, and the final target field to be converted is determined according to the summary calculation result. And the pre-generated deletion script is automatically called to delete each target field to be converted, and the pre-generated write script is automatically called to write the corresponding target field. The present invention avoids full table scanning when updating non-sub-library and sub-table key fields, improves data update efficiency, and greatly reduces the impact on production business processing. By automatically generating deletion scripts and write scripts to delete and write fields, the data update process is greatly simplified and the accuracy of data update is improved.

[0124] In a specific implementation of the present invention, the target field to be converted determining module 35 includes:

[0125] A first summary field obtaining submodule is used to merge the initial to-be-converted fields in each to-be-processed table in a preset order, and perform summary calculation on the merged result to obtain a first summary field;

[0126] A second summary field obtaining submodule is used to merge the target fields corresponding to the initial to-be-converted fields in each to-be-processed table in a preset order, and perform summary calculation on the merged result to obtain a second summary field;

[0127] A first judgment submodule is used to judge, for each table to be processed, whether there is a corresponding relationship between the first summary field and the second summary field in the pre-established conversion intermediate table;

[0128] The target to-be-converted field determination submodule is used to, when it is determined that there is a correspondence between the first summary field and the second summary field in the pre-established conversion intermediate table, determine the to-be-processed table in which there is a correspondence between the first summary field and the second summary field in the conversion intermediate table as the target to-be-processed table, and determine each initial to-be-converted field in the target to-be-processed table as the target to-be-converted field.

[0129] In a specific embodiment of the present invention, the field extraction module 31 includes:

[0130] The request parsing submodule is used to parse the received field extraction request to obtain the fields to be extracted;

[0131] The second judgment submodule is used to judge whether the field to be extracted is a sub-library or sub-table field;

[0132] The field extraction submodule is used to determine each table to be processed according to the sub-library and sub-table fields when it is determined that the field to be extracted is a sub-library and sub-table field, and perform field extraction operations on the data to be processed in each table to be processed.

[0133] In a specific embodiment of the present invention, the device may further include:

[0134] A script delivery module is used to call a pre-generated deletion script to perform a deletion operation on each target field to be converted, call a pre-generated write script to perform a write operation on the target field corresponding to each target field to be converted, and then deliver the deletion script and the write script to the script table to be executed in the production library supporting business processing;

[0135] The script export module is used to export the delete script and write script from the script table to be executed;

[0136] The first production library field deletion and writing module is used to use the preset operation and maintenance tool to execute the deletion script to delete each target field to be converted in the production library, and execute the writing script to write the target field corresponding to each target field to be converted into the production library.

[0137] In a specific embodiment of the present invention, the device may further include:

[0138] The second judgment module is used to judge whether there is a field to be extracted in the preset range extraction table when it is determined that the field to be extracted is not a sub-library or sub-table field;

[0139] An extraction range search module is used to search the target field extraction range corresponding to the field to be extracted from the preset range extraction table when it is determined that there is a field to be extracted in the preset range extraction table;

[0140] The field extraction module 31 is specifically a module that determines each table to be processed according to the target field extraction range and performs field extraction operations on the data to be processed in each table to be processed.

[0141] In a specific embodiment of the present invention, the device may further include:

[0142] The primary key acquisition module is used to obtain the target sub-library and sub-table keys corresponding to the to-be-processed tables to which the target fields to be converted belong and the target primary keys corresponding to the target fields to be converted, after calling the pre-generated deletion script to perform a deletion operation on each target field to be converted and calling the pre-generated writing script to perform a writing operation on the target fields corresponding to the target fields to be converted;

[0143] The module for searching for fields to be converted is used to search for each target field to be converted from the production database supporting business processing according to each target sub-database sub-table key and each target primary key;

[0144] The second production library field deletion and writing module is used to delete each target field to be converted in the production library, and write the target field corresponding to each target field to be converted into the production library.

[0145] In a specific embodiment of the present invention, the device may further include:

[0146] The field backup and batch number generation module is used to perform field extraction operations on the data to be processed in each table to be processed, obtain each initial field to be converted, back up each initial field to be converted to a preset backup table, and generate a backup batch number corresponding to the current backup.

[0147] Corresponding to the above method embodiment, see Figure 4 , Figure 4 This is a schematic diagram of a data updating device for a distributed database provided by the present invention, and the device may include:

[0148] A memory 332, for storing computer programs;

[0149] The processor 322 is used to implement the steps of the distributed database data updating method of the above method embodiment when executing the computer program.

[0150] For details, please refer to Figure 5 , Figure 5 A specific structural diagram of a data update device for a distributed database provided in this embodiment, which may have relatively large differences due to different configurations or performances, may include a processor (central processing units, CPU) 322 (for example, one or more processors) and a memory 332, wherein the memory 332 stores one or more computer applications 342 or data 344. Among them, the memory 332 may be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), each of which may include a series of instruction operations in the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332, and execute a series of instruction operations in the memory 332 on the data update device 301 for the distributed database.

[0151] The data updating device 301 of the distributed database may further include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0152] The steps in the distributed database data updating method described above can be implemented by the structure of the distributed database data updating device.

[0153] Corresponding to the above method embodiment, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the following steps can be implemented:

[0154] Perform field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted; for each initial field to be converted, determine whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new and old field comparison table; if so, determine the comparison field corresponding to the initial field to be converted according to the target comparison relationship, and determine the comparison field as the target field; if not, determine the initial field to be converted as the target field; take the table to be processed as a unit, perform summary calculations on each initial field to be converted, perform summary calculations on each target field, and determine the target field to be converted according to the summary calculation results; call the pre-generated deletion script to perform a deletion operation on each target field to be converted, and call the pre-generated write script to perform a write operation on the target fields corresponding to each target field to be converted.

[0155] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0156] For an introduction to the computer-readable storage medium provided by the present invention, please refer to the above method embodiment, and the present invention will not be elaborated here.

[0157] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the devices, equipment and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0158] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the technical solution and core ideas of the present invention. It should be pointed out that for ordinary technicians in this technical field, the present invention can also be improved and modified without departing from the principles of the present invention, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A data updating method for a distributed database, characterized in that: include: Perform field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted; For each initial field to be converted, determining whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table; If yes, determining the reference field corresponding to the initial field to be converted according to the target reference relationship, and determining the reference field as the target field; If not, determining the initial field to be converted as the target field; Taking the table to be processed as a unit, performing summary calculation on each of the initial fields to be converted, performing summary calculation on each of the target fields, and determining the target field to be converted according to the summary calculation result; Retrieving a pre-generated delete script to perform a delete operation on each of the target fields to be converted, and retrieving a pre-generated write script to perform a write operation on the target fields respectively corresponding to each of the target fields to be converted; Perform field extraction operations on the data to be processed in each table to be processed, including: Parse the received field extraction request to obtain the fields to be extracted; Determine whether the field to be extracted is a sub-library or sub-table field; If so, then determine each of the to-be-processed tables according to the sub-library and sub-table fields, and perform field extraction operations on the to-be-processed data in each of the to-be-processed tables; When it is determined that the field to be extracted is not a sub-library or sub-table field, the method further includes: Determine whether the field to be extracted exists in the preset range extraction table; If so, searching the preset range extraction table for the target field extraction range corresponding to the field to be extracted; Determine each of the to-be-processed tables according to the target field extraction range, and perform field extraction operations on the to-be-processed data in each of the to-be-processed tables; Performing summary calculation on each of the initial fields to be converted, and performing summary calculation on each of the target fields, including: A message digest algorithm is used to perform a digest calculation on each of the initial fields to be converted, and a message digest algorithm is used to perform a digest calculation on each of the target fields.

2. The data updating method of a distributed database according to claim 1, characterized in that: Taking the table to be processed as a unit, performing summary calculation on each of the initial fields to be converted, performing summary calculation on each of the target fields, and determining the target field to be converted according to the summary calculation result, including: Merge the initial to-be-converted fields in each to-be-processed table in a preset order, and perform summary calculation on the merged result to obtain a first summary field; Merge the target fields corresponding to the initial to-be-converted fields in each to-be-processed table according to the preset order, and perform summary calculation on the merged result to obtain a second summary field; For each table to be processed, determining whether there is a corresponding relationship between the first summary field and the second summary field in a pre-established conversion intermediate table; If so, the to-be-processed table in the conversion intermediate table in which the correspondence between the first summary field and the second summary field exists is determined as the target to-be-processed table, and each of the initial to-be-converted fields in the target to-be-processed table is determined as the target to-be-converted field.

3. The data updating method of a distributed database according to claim 1, characterized in that: After calling the pre-generated deletion script to perform a deletion operation on each of the target fields to be converted, and calling the pre-generated writing script to perform a writing operation on the target fields respectively corresponding to each of the target fields to be converted, the method further includes: Put the deletion script and the writing script into a script table to be executed in a production library supporting business handling; Exporting the deletion script and the writing script from the to-be-executed script table; The deletion script is executed by using a preset operation and maintenance tool to delete each of the target fields to be converted in the production library, and the writing script is executed to write the target fields corresponding to each of the target fields to be converted into the production library.

4. The data updating method of a distributed database according to claim 1, characterized in that: After calling the pre-generated deletion script to perform a deletion operation on each of the target fields to be converted, and calling the pre-generated writing script to perform a writing operation on the target fields respectively corresponding to each of the target fields to be converted, the method further includes: Obtain the target sub-library sub-table keys corresponding to the to-be-processed tables to which the target to-be-converted fields belong and the target primary keys corresponding to the target to-be-converted fields; Searching for each target field to be converted from a production database supporting business handling according to each target sub-library sub-table key and each target primary key; A deletion operation is performed on each of the target fields to be converted in the production library, and a target field corresponding to each of the target fields to be converted is written into the production library.

5. The data updating method of a distributed database according to claim 1, characterized in that: After performing field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted, the following steps are also included: Each of the initial fields to be converted is backed up to a preset backup table, and a backup batch number corresponding to the current backup is generated.

6. A data updating device for a distributed database, characterized in that: include: A field extraction module is used to perform field extraction operations on the data to be processed in each table to be processed to obtain each initial field to be converted; A first judgment module is used to judge, for each initial field to be converted, whether there is a target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table; A first target field determination module is used to determine the reference field corresponding to the initial field to be converted according to the target reference relationship when it is determined that there is a target reference relationship corresponding to the initial field to be converted in the pre-established new-old field reference table, and determine the reference field as the target field; A second target field determination module, configured to determine the initial field to be converted as the target field when it is determined that there is no target comparison relationship corresponding to the initial field to be converted in the pre-established new-old field comparison table; A target field to be converted determination module is used to perform summary calculation on each of the initial fields to be converted and on each of the target fields, taking the table to be processed as a unit, and determine the target field to be converted according to the summary calculation result; A field deletion and writing module is used to call a pre-generated deletion script to perform a deletion operation on each of the target fields to be converted, and call a pre-generated writing script to perform a writing operation on the target fields respectively corresponding to each of the target fields to be converted; The field extraction module includes: The request parsing submodule is used to parse the received field extraction request to obtain the fields to be extracted; The second judgment submodule is used to judge whether the field to be extracted is a sub-library or sub-table field; The field extraction submodule is used to determine each table to be processed according to the sub-library and sub-table fields when it is determined that the field to be extracted is a sub-library and sub-table field, and perform field extraction operations on the data to be processed in each table to be processed; Also includes: The second judgment module is used to judge whether there is a field to be extracted in the preset range extraction table when it is determined that the field to be extracted is not a sub-library or sub-table field; An extraction range search module is used to search the target field extraction range corresponding to the field to be extracted from the preset range extraction table when it is determined that there is a field to be extracted in the preset range extraction table; The field extraction module is specifically a module that determines each table to be processed according to the target field extraction range and performs field extraction operations on the data to be processed in each table to be processed; The target to-be-converted field determination module is specifically a module that uses a message digest algorithm to perform digest calculations on each of the initial to-be-converted fields, and uses a message digest algorithm to perform digest calculations on each of the target fields.

7. A data updating device for a distributed database, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the distributed database data updating method as claimed in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data updating method of the distributed database as claimed in any one of claims 1 to 5 are implemented.

Images