Data synchronization method and device, computer equipment and readable storage medium

Abstract

The invention relates to the field of big data, and particularly discloses a data synchronization method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining to-be-synchronized data of a host; screening mixed code field data mixed with different code systems and application partition identifiers under different application partitions from the to-be-synchronized data; the code system of the application partition identifier and the mixed code field data is converted into a target code system, a target application partition identifier and target mixed code field data are obtained to update the to-be-synchronized data, and the target code system is matched with the code system of the distributed system; and synchronizing the updated to-be-synchronized data to the distributed system. By adopting the method, the data synchronization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data synchronization method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] With the development of science and technology, many existing old systems will be updated and iterated. In order to preserve the data in the old system, you can choose to synchronize the data in the old system to the database of the new system.

[0003] Taking the synchronization of host data to distributed systems as an example, in traditional technologies, commonly used data synchronization methods include but are not limited to manual SQL (Structured Query Language) script synchronization, ETL (Extract-Transform-Load) tool migration, etc. Specifically, manual SQL script synchronization refers to manually writing data scripts to synchronize data in the host to the distributed system through data scripts; ETL tool migration refers to regularly extracting data from the host and loading it into the distributed system through professional ETL tools. However, the above data synchronization methods still have the problem of inefficient data synchronization. Summary of the invention

[0004] Based on this, it is necessary to provide an efficient data synchronization method, apparatus, computer equipment, computer-readable storage medium and computer program product to address the above technical issues.

[0005] In a first aspect, the present application provides a data synchronization method, comprising:

[0006] Get the host's data to be synchronized;

[0007] Filtering mixed field data with different coding systems and application partition identifiers under different application partitions from the data to be synchronized;

[0008] Convert the code system of the application partition identifier and the mixed code field data into the target code system, obtain the target application partition identifier and the target mixed code field data, so as to update the data to be synchronized, and the target code system matches the code system of the distributed system;

[0009] Synchronize the updated data to be synchronized to the distributed system.

[0010] In one embodiment, converting the code system of the application partition identifier into the target code system to obtain the target application partition identifier includes:

[0011] Convert each character in the field data associated with the application partition identifier into an ASSI code value;

[0012] The ASSIC code values ​​obtained by the cumulative conversion are obtained to obtain the initial application partition identification code value;

[0013] The code system of the initial application partition identification code value is converted into the target code system to obtain the target application partition identification.

[0014] In one embodiment, converting the code system of the mixed code field data into a target code system to obtain the target mixed code field data further includes:

[0015] In the case where the mixed code field data includes multiple byte data, the code system of each byte data is converted into the target code system according to the occupied length of each byte data;

[0016] Concatenate multiple converted byte data to obtain the target mixed code field data.

[0017] In one embodiment, the method further comprises:

[0018] In the case where the data to be synchronized is incremental data to be synchronized, determining a business-related field of the incremental data to be synchronized;

[0019] Determine the business type of the incremental data to be synchronized based on the business association field;

[0020] Based on the business type, determine the target field data to be converted in the business-related field;

[0021] Convert the target field data's code system to the target code system.

[0022] In one embodiment, after synchronizing the updated data to be synchronized to the distributed system, the method further includes:

[0023] Check whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system;

[0024] If it is detected that the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, a synchronization success message is pushed;

[0025] If it is detected that the data to be synchronized in the host is inconsistent with the updated data to be synchronized in the distributed system, a synchronization failure message is pushed.

[0026] In one embodiment, detecting whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system includes:

[0027] In the case where the data to be synchronized is the stock data to be synchronized, a preset data consistency detection script is called to batch detect whether the stock data to be synchronized in the host is consistent with the updated stock data to be synchronized in the distributed system;

[0028] In the case where the data to be synchronized is incremental data to be synchronized, obtaining first incremental data to be synchronized in the distributed system and second incremental data to be synchronized in the host, wherein the second incremental data to be synchronized has the same primary key as the first incremental data to be synchronized;

[0029] Check whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

[0030] In one embodiment, obtaining the data to be synchronized of the host includes:

[0031] Filter the stock data to be synchronized from the host data in the downloaded historical time period;

[0032] The incremental data to be synchronized is obtained from the host data update message queue, where the incremental data to be synchronized includes the incremental data to be synchronized and the existing data to be synchronized.

[0033] In a second aspect, the present application further provides a data synchronization device, including:

[0034] An acquisition module is used to acquire the data to be synchronized from the host;

[0035] A screening module, used to screen mixed code field data with different code systems and application partition identifiers under different application partitions from the data to be synchronized;

[0036] A processing module, used to convert the code system of the application partition identifier and the mixed code field data into a target code system, obtain a target application partition identifier and a target mixed code field data, so as to update the data to be synchronized, and the target code system matches the code system of the distributed system;

[0037] The synchronization module is used to synchronize the updated data to be synchronized to the distributed system.

[0038] In one of the embodiments, the processing module is also used to convert each character in the field data associated with the application partition identifier into an ASSI code value, accumulate the converted ASSI code values ​​to obtain an initial application partition identifier code value, convert the code system of the initial application partition identifier code value into a target code system to obtain a target application partition identifier.

[0039] In one of the embodiments, the processing module is also used to convert the coding system of each byte data into the target coding system according to the placeholder length of each byte data when the mixed code field data includes multiple byte data, and splice the multiple converted byte data to obtain the target mixed code field data.

[0040] In one of the embodiments, the data synchronization device also includes a business association module, which is used to determine the business association field of the incremental data to be synchronized when the data to be synchronized is incremental data to be synchronized, determine the business type of the incremental data to be synchronized according to the business association field, determine the target field data to be converted in the business association field based on the business type, and convert the code system of the target field data into the target code system.

[0041] In one of the embodiments, the data synchronization device also includes a consistency detection module, which is used to detect whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system. If it is detected that the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, a synchronization success message is pushed; if it is detected that the data to be synchronized in the host is inconsistent with the updated data to be synchronized in the distributed system, a synchronization failure message is pushed.

[0042] In one of the embodiments, the consistency detection module is also used to call a preset data consistency detection script when the data to be synchronized is existing data to be synchronized, batch detect whether the existing data to be synchronized in the host is consistent with the updated existing data to be synchronized in the distributed system, and when the data to be synchronized is incremental data to be synchronized, obtain the first incremental data to be synchronized in the distributed system and the second incremental data to be synchronized in the host, the second incremental data to be synchronized has the same primary key as the first incremental data to be synchronized, and detect whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

[0043] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0044] Get the host's data to be synchronized;

[0045] Filtering mixed field data with different coding systems and application partition identifiers under different application partitions from the data to be synchronized;

[0046] Convert the code system of the application partition identifier and the mixed code field data into the target code system, obtain the target application partition identifier and the target mixed code field data, so as to update the data to be synchronized, and the target code system matches the code system of the distributed system;

[0047] Synchronize the updated data to be synchronized to the distributed system.

[0048] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0049] Get the host's data to be synchronized;

[0050] Filtering mixed field data with different coding systems and application partition identifiers under different application partitions from the data to be synchronized;

[0051] Convert the code system of the application partition identifier and the mixed code field data into the target code system, obtain the target application partition identifier and the target mixed code field data, so as to update the data to be synchronized, and the target code system matches the code system of the distributed system;

[0052] Synchronize the updated data to be synchronized to the distributed system.

[0053] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0054] Get the host's data to be synchronized;

[0055] Filtering mixed field data with different coding systems and application partition identifiers under different application partitions from the data to be synchronized;

[0056] Convert the code system of the application partition identifier and the mixed code field data into the target code system, obtain the target application partition identifier and the target mixed code field data, so as to update the data to be synchronized, and the target code system matches the code system of the distributed system;

[0057] Synchronize the updated data to be synchronized to the distributed system.

[0058] The above-mentioned data synchronization method, device, computer equipment, computer-readable storage medium and computer program product are different from the traditional method of directly synchronizing host data to a distributed system, and propose a code value conversion mechanism with strong applicability and flexibility. Specifically, first, from the data to be synchronized, mixed-code field data mixed with different code systems and application partition identifiers under different application partitions are screened, which can reduce unnecessary data code system conversion and improve the efficiency of data synchronization. Then, by uniformly converting the code system of the application partition identifier and the mixed-code field data into a code system that matches the distributed system, the data to be synchronized is updated, which can simplify the data synchronization processing between different systems, improve the consistency and compatibility of data in the distributed system, reduce the garbled code phenomenon caused by inconsistent code systems, and further improve the data synchronization efficiency. Finally, the updated data to be synchronized is synchronized to the distributed system, so that the synchronized host data can be adapted to the needs of the distributed system. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments of the present application or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 An application environment diagram of a data synchronization method in an embodiment;

[0061] Figure 2 A schematic diagram of a flow chart of a data synchronization method in an embodiment;

[0062] Figure 3 is a flowchart of a data synchronization method in another embodiment;

[0063] Figure 4 is a flowchart of a data synchronization method in yet another embodiment;

[0064] Figure 5 A flowchart of a data synchronization method in another embodiment;

[0065] Figure 6 A detailed flowchart of a data synchronization method in an embodiment;

[0066] Figure 7 is a structural block diagram of a data synchronization device in an embodiment;

[0067] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, groups, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0069] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to host data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0070] The data synchronization method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown in the figure, the host 102 communicates with the server 104 through the network, and the distributed system 106 communicates with the server 104. A data synchronization system 108 is deployed on the server 104, and the server 104 can obtain the data to be synchronized sent by the host 102 through the data synchronization system 108 to perform the code conversion of the data to be synchronized, obtain the updated data to be synchronized, and then synchronize the updated data to be synchronized to the distributed system 106. The data storage system can be integrated on the server 104.

[0071] When the host 102 responds to the data synchronization request initiated by the user, or when the host 102 meets the preset data synchronization requirement conditions, the host 102 sends the data to be synchronized to the server 104, wherein the preset data synchronization requirement conditions include but are not limited to preset time conditions, preset location conditions, etc.

[0072] Server 104 obtains the data to be synchronized from the host, and selects mixed-code field data with different coding systems and application partition identifiers under different application partitions from the data to be synchronized, and converts the coding systems of the application partition identifier and the mixed-code field data into target coding systems suitable for the requirements of the distributed system 106, and obtains the target application partition identifier and the target mixed-code field data to update the data to be synchronized. Server 104 synchronizes the updated data to be synchronized to the distributed system 106.

[0073] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0074] In an exemplary embodiment, Figure 2 As shown, a data synchronization method is provided, which is applied to Figure 1 The server 104 in FIG. 1 is used as an example for explanation. Among them:

[0075] S200, obtaining the host's data to be synchronized.

[0076] The data to be synchronized are several pieces of data in the host. Based on business needs, this part of data needs to be retained in the distributed system.

[0077] Specifically, when the host responds to a data synchronization request initiated by a user, or when the host meets a preset data synchronization request condition, the host sends the data to be synchronized to the server, wherein the preset data synchronization request condition includes but is not limited to a preset time condition, a preset location condition, etc.

[0078] S400 , screening mixed code field data containing mixed code systems and application partition identifiers under different application partitions from the data to be synchronized.

[0079] In order to facilitate memory and search, certain rules must be followed when compiling codes. These rules are called code systems. For example, code systems include but are not limited to binary, hexadecimal, and octal. Mixed code field data contains data of different code systems. For example, a field data may contain binary data and hexadecimal data. This field data can be called mixed code field data.

[0080] An application partition refers to a partition area for application data. The application partition identifier in this application generally consists of five letters and is used to distinguish the type of application partition, such as country A, country B, and other country types.

[0081] Specifically, there are more than one data to be synchronized, and each data to be synchronized contains data of several fields. In order to meet the data format requirements of distributed storage, it is necessary to determine whether the data of each field is mixed field data mixed with different coding systems. If so, it is determined that the field data is mixed field data, and the mixed field data is subsequently converted to a coding system. If not, it is determined that the field data is not mixed field data, and there is no need to perform subsequent coding system conversion on the field data.

[0082] The application partition identifier is the identification data related to the business in the data to be synchronized. Therefore, the application partition identifier also needs to be converted to meet the data format requirements of the distributed system application and storage. Furthermore, since the specific code position of the application partition identifier in the data to be synchronized has been pre-set, for example, the code position of the application partition identifier belongs to 0-4 of the code positions of the data to be synchronized. Based on the pre-set code position of the application partition identifier, the application partition identifier under the application partition where the data to be synchronized is located is filtered from the data to be synchronized.

[0083] S600, converting the coding system of the application partition identifier and the mixed code field data into the target coding system, obtaining the target application partition identifier and the target mixed code field data, so as to update the data to be synchronized.

[0084] The target encoding matches the encoding of the distributed system application partition identifier. Generally, the target encoding in the distributed system is UTF-8 (8-bit, Universal Character Set / Unicode TransformationFormat) code data. UTF-8 is a variable-length character encoding for Unicode, used to represent any character in the Unicode standard, and the first byte in its encoding is still compatible with ASCII, so that the original software that processes ASCII characters can continue to be used without or with only a few modifications.

[0085] Specifically, the data to be synchronized of the host has a data storage format and storage method unique to the host. In order to store the data smoothly in the distributed system, the mixed-code field data with different coding systems and the application partition identifier related to the business can be processed by performing job conversion through JCL (Job Control Language), that is, the coding system of the application partition identifier and the mixed-code field data is converted to a coding system that matches the distributed system application partition identifier, and the target application partition identifier and the target mixed-code field data are obtained to update the data to be synchronized. The coding system of the target application partition identifier and the target mixed-code field data is the target coding system.

[0086] S800, synchronizing the updated data to be synchronized to the distributed system.

[0087] Specifically, the data synchronization system deployed on the server synchronizes the updated data to be synchronized to the distributed system through the XCOM (Xtensiable Communication Of Middleware) interface function.

[0088] In one embodiment, before synchronizing the updated data to be synchronized to the distributed system, since the total data length of the host and the distributed system is different, the updated data to be synchronized needs to be packaged with the data length so that the data length of the updated data to be synchronized is converted to a target length that meets the data length requirement of the distributed system, so that the distributed system can receive the data and perform subsequent data storage, processing and other operations on the converted data to be synchronized.

[0089] In another embodiment, after the code conversion is completed, a separator symbol "|" is automatically added to the front and end of each converted field data. After the distributed system receives the updated data to be synchronized, the updated data to be synchronized will fall into the database of the distributed system according to the separator. That is, the corresponding column of the field data between each two separators in the database of the distributed system is found, and the data is truncated and stored according to the column.

[0090] In the above data synchronization method, different from the traditional method of directly synchronizing host data to a distributed system, a code value conversion mechanism with strong applicability and flexibility is proposed. Specifically, first, from the data to be synchronized, mixed field data with different coding systems and application partition identifiers under different application partitions are screened, which can reduce unnecessary data synchronization and improve the efficiency of data synchronization. Then, by uniformly converting the coding system of the application partition identifier and the mixed field data into a coding system that matches the distributed system, the data to be synchronized is updated, which can simplify the data synchronization processing between different systems, improve the consistency and compatibility of data in the distributed system, reduce the garbled code phenomenon caused by inconsistent coding systems, and further improve the data synchronization efficiency. Finally, the updated data to be synchronized is synchronized to the distributed system, so that the synchronized host data can be adapted to the needs of the distributed system.

[0091] In an exemplary embodiment, Figure 3 As shown, S600, including:

[0092] S620, convert each character in the field data associated with the application partition identifier into an ASIC code value.

[0093] S640, accumulating the converted ASSIC code values ​​to obtain an initial application partition identification code value.

[0094] S660, convert the code system of the initial application partition identification code value into a target code system to obtain a target application partition identification.

[0095] S680, converting the code system of the mixed code field data into the target code system to obtain the target mixed code field data to update the data to be synchronized.

[0096] The ASSI code refers to the use of a specified 7-bit or 8-bit binary number combination to represent 128 or 256 possible characters.

[0097] Specifically, the application partition identifier can be composed of several letters. Due to business needs, the application partition identifier of the host needs to be converted into an application partition identifier that meets the needs of the distributed system. Specifically, the code conversion of the application partition identifier is not only a simple conversion of the code system of the application partition identifier into the target code system, but also requires the participation of the field data associated with the application partition identifier, such as the reconciliation classification number, the reconciliation classification date, etc., that is, each character of the associated field data is converted into an ASSI code, and then each ASSI code value is superimposed, and the superimposed ASSI code value is used as the initial application partition identifier code value. The code system of the initial application partition identifier code value is converted from the ASSI code to the target code system, and the result obtained after the conversion is determined as the target application partition identifier. In other words, converting the code system of the application partition identifier into the target code system is actually a conversion based on the field data associated with the application partition identifier to obtain an updated target application partition identifier.

[0098] Furthermore, in the process of converting each character of the preset field data of other preset fields into an ASSI code, the data synchronization system first converts the code system of each character of the preset field data into hexadecimal, and then converts it from hexadecimal into an ASSI code.

[0099] In the above embodiment, by converting the code system of each character in the field data associated with the application partition identifier into an ASSI code, accumulating each ASSI code value to obtain an initial application partition identifier code value, and converting the initial application partition identifier code value from the ASSI code into a target code system to obtain a target application partition identifier, an accurate application partition identifier can be accurately obtained based on the target field data, and since the code system of the field data associated with the application partition identifier is converted, an accurate application partition identifier for synchronization to a distributed system can be obtained.

[0100] In an exemplary embodiment, converting the code system of the mixed code field data into the target code system to obtain the target mixed code field data further includes:

[0101] In the case where the mixed code field data includes multiple byte data, the coding system of each byte data is converted into the target coding system according to the placeholder length of each byte data, and the multiple converted byte data are spliced ​​to obtain the target mixed code field data.

[0102] The mixed code field may be a large field, and the large field refers to a field containing multiple bytes.

[0103] Specifically, in order to improve the efficiency and accuracy of the mixed code field data code conversion, the mixed code field data can be divided into a plurality of byte data according to the field byte code of the data, and then the code of each byte data is converted into the target code according to the occupied length of each byte data, and then the target mixed code field data is obtained by splicing a plurality of converted byte data.

[0104] For example, ABC corresponds to nine-byte mixed-code field data, where A corresponds to 0-2-bit byte data, B corresponds to 3-5-bit byte data, and C corresponds to 6-8-bit byte data. During code conversion, the 0-2-bit byte data, 3-5-bit byte data, and 6-8-bit byte data are converted into three letters respectively, and then the three letters are spliced ​​into the complete target mixed-code field data.

[0105] In one embodiment, converting the code system of each byte data into the target code system includes: screening out the byte data whose code system is not the target code system, and converting the code system of this part of the byte data into the target code system.

[0106] For example, let the target encoding be UTF-8, and there is mixed encoding data of UTF-8 and EBCDIC in the mixed field data. In the process of synchronizing data to the distributed system, first, according to the occupied length of each byte data in the mixed field data, the mixed field data is divided into multiple byte data. Among them, the encoding of some byte data is UTF-8, and the encoding of other byte data is EBCDIC. The encoding of the byte data of the EBCDIC code is converted to UTF-8.

[0107] In the above embodiment, when the mixed code field data includes multiple byte data, according to the occupied length of each byte data, the code system of each byte data is converted into the target code system, and multiple converted byte data are spliced, and the code systems of multiple byte data are converted into the target code system, thereby reducing the conversion time and improving the efficiency of data synchronization. Furthermore, compared with the code system conversion of longer byte data, the code system conversion of shorter byte data can obtain the target mixed code field data with higher accuracy.

[0108] In an exemplary embodiment, Figure 4 As shown, the method also includes:

[0109] S120: When the data to be synchronized is incremental data to be synchronized, determine a service-related field of the incremental data to be synchronized.

[0110] S140: Determine the business type of the incremental data to be synchronized according to the business association field.

[0111] S160: Determine target field data to be converted in the business-related field based on the business type.

[0112] S180, converting the code system of the target field data into the target code system.

[0113] The business-related fields refer to fields related to the business. Different businesses will obtain different data and store them in the data table of the host.

[0114] Specifically, the incremental data to be synchronized includes several fields, some of which are fields associated with services. Different services correspond to different field truncation positions, and the business-related fields of the incremental data to be synchronized are determined according to the field truncation positions.

[0115] Since different business types correspond to different business association fields, it is possible to determine which business the incremental data to be synchronized is associated with based on the business association field, that is, determine the business type of the incremental data to be synchronized.

[0116] Furthermore, among the multiple business-related fields corresponding to different business types, some are target fields that need to be converted. Therefore, the target field to be converted in the business-related field can be determined by the data conversion rule corresponding to the business type, and then the target field data of the target field can be converted into data with the target code system. Specifically, determining the target field to be converted in the business-related field means determining the field whose code system of the data in the business-related field is not the target code system.

[0117] For example, through the field truncation position, it is determined that the business-related field to be synchronized is a field related to card opening. According to the business-related field, it is determined that the business type of the incremental data to be synchronized is the card opening type. For the card opening type, the fields in the business-related field that need to be converted to code system are determined, for example, the data corresponding to the card opening time and the card opening account number need to be converted to code system, and the field data that needs to be converted to code system is converted to data with the target code system.

[0118] In the above embodiment, when the data to be synchronized is incremental data to be synchronized, by determining the service type of the incremental data to be synchronized, the target field data to be converted can be accurately coded based on the service type, thereby improving the accuracy of data synchronization.

[0119] In an exemplary embodiment, Figure 5 As shown, after S800, it also includes:

[0120] S920, detecting whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system.

[0121] S940: If it is detected that the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, a synchronization success message is pushed.

[0122] S960: If it is detected that the data to be synchronized in the host is inconsistent with the updated data to be synchronized in the distributed system, a synchronization failure message is pushed.

[0123] Specifically, after the data synchronization is completed, it is necessary to detect whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, so as to verify the data synchronization process.

[0124] Furthermore, it is detected whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, that is, whether the existing data to be synchronized in the host is consistent with the updated existing data to be synchronized in the distributed system, and whether the incremental data to be synchronized in the host is consistent with the updated incremental data to be synchronized in the distributed system.

[0125] If the two are detected to be consistent, a synchronization success message is generated and pushed. If the existing data to be synchronized in the host is detected to be inconsistent with the updated existing data to be synchronized in the distributed system, or the incremental existing data to be synchronized in the host is detected to be inconsistent with the updated incremental existing data to be synchronized in the distributed system, a synchronization failure message is generated and pushed.

[0126] In the above embodiment, by detecting whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, the data synchronization process can be accurately verified to further increase the reliability of the data synchronization process.

[0127] In an exemplary embodiment, Figure 6 As shown, S920 includes:

[0128] S922: When the data to be synchronized is stock data to be synchronized, a preset data consistency detection script is called to batch detect whether the stock data to be synchronized in the host is consistent with the updated stock data to be synchronized in the distributed system.

[0129] S924, when the data to be synchronized is incremental data to be synchronized, obtain the first incremental data to be synchronized in the distributed system and the second incremental data to be synchronized in the host, the second incremental data to be synchronized has the same primary key as the first incremental data to be synchronized.

[0130] S926, detecting whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

[0131] Specifically, the present application has different data consistency detection processes for existing data to be synchronized and incremental data to be synchronized.

[0132] For the stock data to be synchronized, call the preset data consistency detection script to batch detect whether the stock data to be synchronized in the host is consistent with the updated stock data to be synchronized in the distributed system.

[0133] For the incremental data to be synchronized, the consistency of the data is ensured by comparing the single SQL statement. Specifically, during the data synchronization process, the host contains several pieces of second incremental data to be synchronized, and after the data synchronization is completed, the distributed system contains several pieces of first incremental data to be synchronized.

[0134] Record the primary key of each first incremental data to be synchronized in the distributed system and each second incremental data to be synchronized in the host, query the first incremental data to be synchronized and the second incremental data to be synchronized corresponding to each identical primary key, and check whether the first incremental data to be synchronized and the second incremental data to be synchronized with the identical primary key are consistent.

[0135] For example, let the encoding system of the first incremental data to be synchronized be UTF-8, and the encoding system of the second incremental data to be synchronized be EBCDIC. The specific detection process of detecting whether the first incremental data to be synchronized and the second incremental data to be synchronized with the same primary key are consistent includes the following three situations:

[0136] The first method is to convert the first incremental data to be synchronized into data whose coding system is EBCDIC code, and detect whether the converted first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

[0137] The second method is to convert the second incremental data to be synchronized into data with a coding system of UTF-8, and detect whether the converted second incremental data to be synchronized is consistent with the first incremental data to be synchronized.

[0138] The first method is to convert the first incremental data to be synchronized and the second incremental data to be synchronized into data with an intermediate code system, and detect whether the converted first incremental data to be synchronized is consistent with the converted second incremental data to be synchronized.

[0139] In one of the embodiments, in the process of detecting whether the first incremental data to be synchronized and the second incremental data to be synchronized with the same primary key are consistent, not only the code conversion is required, but also some special business field data therein need to be converted into field data with the same field format.

[0140] In one embodiment, when detecting whether the first incremental data to be synchronized and the second incremental data to be synchronized with the same primary key are consistent, the fields of the incremental data to be synchronized can be compared one by one to ensure that the data converted in each field is correct. In practical applications, some specific fields can be set to not be compared.

[0141] In the above embodiment, when the data to be synchronized is the stock data to be synchronized, the stock data to be synchronized in the host is batch-checked to see if it is consistent with the updated stock data to be synchronized in the distributed system, thereby improving the efficiency of detecting the data synchronization process of the stock data to be synchronized. When the data to be synchronized is the incremental data to be synchronized, consistency detection is performed one by one on each incremental data to be synchronized with the same primary key in the host and the distributed system, thereby improving the accuracy of data synchronization verification.

[0142] In an exemplary embodiment, obtaining the data to be synchronized of the host includes:

[0143] The stock data to be synchronized is filtered out from the host data in the downloaded historical time period, and the incremental data to be synchronized is obtained from the host data update message queue. The data to be synchronized includes the incremental data to be synchronized and the stock data to be synchronized.

[0144] The incremental data to be synchronized includes the updated data of the host in the current time period. The data to be synchronized includes the incremental data to be synchronized and the stock data to be synchronized.

[0145] Specifically, in this embodiment, data synchronization needs to be performed on both the incremental data to be synchronized and the existing data to be synchronized.

[0146] For the stock data to be synchronized in the historical time period, the host data in the historical time period is downloaded through the download program and stored in the file, and the server selects the stock data to be synchronized from the file. Commonly used download programs include but are not limited to GBAFGUNL program.

[0147] For the incremental data to be synchronized in the current time period, in order to ensure the timeliness of the data, the incremental data to be synchronized needs to be synchronized through the message queue. That is to say, when the host data is added, deleted, or modified, the host will record the SQL statement of each addition, deletion, or modification operation and send it to the server through the message queue.

[0148] In this embodiment, by acquiring the incremental data to be synchronized and the stock data to be synchronized of the host, the data to be synchronized of the host can be fully synchronized to the distributed system, thereby reducing the possibility of some important data not being synchronized and improving the accuracy of data synchronization.

[0149] In a specific application embodiment, the data to be synchronized of the host is synchronized to the distributed system, and the data synchronization method adopted includes:

[0150] 1. Data Synchronization

[0151] For the existing data to be synchronized:

[0152] 1) Use the download program to download the synchronized stock data.

[0153] 2) The server obtains the downloaded stock data to be synchronized and transforms the stock data to be synchronized, including:

[0154] A. For application partition identifier: convert the code system of each character in the field data associated with the application partition identifier, such as the reconciliation classification date, reconciliation classification number, etc., into ASSIC code, accumulate each ASSIC code value to obtain the initial application partition identifier code value, convert the code system of the initial application partition identifier code value into the target code system, and obtain the target application partition identifier.

[0155] B. For mixed-code field data: mixed-code field data is field data mixed with different code systems. Convert the code system that does not belong to the target code system in the mixed-code field data into the target code system to obtain the target mixed-code field data. For example, if a mixed-code field data contains both UTF-8 code and EBCDIC code data, the EBCDIC code can be uniformly converted into UTF-8 code. Furthermore, in the case that the mixed-code field data includes multiple byte data, the code system of each byte data can be converted into the target code system according to the placeholder length of each byte data, and multiple converted byte data can be spliced ​​to obtain the target mixed-code field data.

[0156] Furthermore, during the code conversion process, separators will be automatically added to the first and last digits of each converted field data.

[0157] 3) Data preprocessing. Since the length ranges of the stock data to be synchronized on the host and distributed systems are different, the stock data to be synchronized on the host can be packaged to meet the length range required by the stock data to be synchronized on the distributed system.

[0158] 4) Download. Use the XCOM interface function to send the pre-processed stock data to be synchronized to the distributed system.

[0159] 5) Storage. After receiving the stock data to be synchronized, the distributed system stores the stock data to be synchronized into the database of the distributed system according to the delimiter.

[0160] For incremental data to be synchronized:

[0161] In order to ensure the timeliness of data, the incremental data to be synchronized requires additional regular data synchronization.

[0162] 1) The host sends a message queue. When operations such as adding, deleting, and modifying the host data occur, the host sends the SQL statement records of the operation to the server in the form of a message queue. The server uses this part of the data as the incremental data to be synchronized and performs data operations on the incremental data to be synchronized.

[0163] 2) Special rule conversion. The code conversion in distributed systems can be roughly divided into three categories, including:

[0164] A. Application partition identification: Same as the above existing data to be synchronized, no further details are given here.

[0165] B. For mixed code field data: Same as the above existing data to be synchronized, no further details will be given here.

[0166] C. For business-related fields: determine the business-related fields of the incremental data to be synchronized, determine the business type of the incremental data to be synchronized based on the business-related fields, and based on the business type, adopt a conversion method that matches the business type to determine the target field data to be converted in the business-related fields, and convert the code system of the target field data into the target code system.

[0167] 3) Data preprocessing: This is the same as the above-mentioned existing data to be synchronized, and will not be described in detail here.

[0168] 4) Data storage and storage. The processed data is stored and stored in the warehouse.

[0169] 2. Consistency test after data synchronization is completed:

[0170] For the stock data to be synchronized: call the preset data consistency detection script to batch detect whether the stock data to be synchronized in the host is consistent with the updated stock data to be synchronized in the distributed system.

[0171] For the incremental data to be synchronized: obtain the first incremental data to be synchronized in the distributed system with the same primary key and the second incremental data to be synchronized in the host, and detect whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized. Detect whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized, including: converting the first incremental data to be synchronized and the second incremental data to be synchronized into data with the same coding system, and then detecting whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

[0172] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiment of the present application also provides a data synchronization device for implementing the data synchronization method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more data synchronization device embodiments provided below can refer to the limitations on the data synchronization method above, and will not be repeated here.

[0174] In an exemplary embodiment, Figure 7 As shown, a data synchronization device 900 is provided, comprising: an acquisition module 200, a screening module 400, a processing module 600 and a synchronization module 800, wherein:

[0175] The acquisition module 200 is used to acquire the data to be synchronized from the host;

[0176] The screening module 400 is used to screen mixed code field data with different code systems and application partition identifiers under different application partitions from the data to be synchronized;

[0177] The processing module 600 is used to convert the code system of the application partition identifier and the mixed code field data into a target code system, obtain a target application partition identifier and a target mixed code field data, so as to update the data to be synchronized, and the target code system matches the code system of the distributed system;

[0178] The synchronization module 800 is used to synchronize the updated data to be synchronized to the distributed system.

[0179] In one of the embodiments, the processing module is also used to convert each character in the field data associated with the application partition identifier into an ASSI code value, accumulate the converted ASSI code values ​​to obtain an initial application partition identifier code value, convert the code system of the initial application partition identifier code value into a target code system to obtain a target application partition identifier.

[0180] In one of the embodiments, the processing module is also used to convert the coding system of each byte data into the target coding system according to the placeholder length of each byte data when the mixed code field data includes multiple byte data, and splice the multiple converted byte data to obtain the target mixed code field data.

[0181] In one of the embodiments, the data synchronization device also includes a business association module, which is used to determine the business association field of the incremental data to be synchronized when the data to be synchronized is incremental data to be synchronized, determine the business type of the incremental data to be synchronized according to the business association field, determine the target field data to be converted in the business association field based on the business type, and convert the code system of the target field data into the target code system.

[0182] In one of the embodiments, the data synchronization device also includes a consistency detection module, which is used to detect whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system. If it is detected that the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, a synchronization success message is pushed; if it is detected that the data to be synchronized in the host is inconsistent with the updated data to be synchronized in the distributed system, a synchronization failure message is pushed.

[0183] In one of the embodiments, the consistency detection module is also used to call a preset data consistency detection script when the data to be synchronized is existing data to be synchronized, batch detect whether the existing data to be synchronized in the host is consistent with the updated existing data to be synchronized in the distributed system, and when the data to be synchronized is incremental data to be synchronized, obtain the first incremental data to be synchronized in the distributed system and the second incremental data to be synchronized in the host, the second incremental data to be synchronized has the same primary key as the first incremental data to be synchronized, and detect whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

[0184] Each module in the above data synchronization device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0185] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as the data to be synchronized of the host. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data synchronization method is implemented.

[0186] Those skilled in the art will understand that Figure 8The structure shown in the figure is a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0187] In one of the embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0188] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0189] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0190] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0191] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0192] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data synchronization method, characterized in that: The method comprises: Get the host's data to be synchronized; From the data to be synchronized, filter mixed field data containing different coding systems and application partition identifiers under different application partitions; Convert the coding system of the application partition identifier and the mixed code field data into a target coding system to obtain a target application partition identifier and a target mixed code field data to update the data to be synchronized, wherein the target coding system matches the coding system of the distributed system; The updated data to be synchronized is synchronized to the distributed system.

2. The method according to claim 1, characterized in that: Converting the code system of the application partition identifier into a target code system to obtain a target application partition identifier includes: Convert each character in the field data associated with the application partition identifier into an ASSI code value; The ASSIC code values ​​obtained by the cumulative conversion are obtained to obtain the initial application partition identification code value; The coding system of the initial application partition identification code value is converted into a target coding system to obtain a target application partition identification.

3. The method according to claim 1, characterized in that Converting the code system of the mixed code field data into a target code system to obtain target mixed code field data, further comprising: In the case where the mixed code field data includes a plurality of byte data, converting the code system of each byte data into the target code system according to the bit length of each byte data; Concatenate multiple converted byte data to obtain the target mixed code field data.

4. The method according to claim 1, characterized in that: The method further comprises: In a case where the data to be synchronized is incremental data to be synchronized, determining a service-related field of the incremental data to be synchronized; Determining the business type of the incremental data to be synchronized according to the business association field; Based on the business type, determining target field data to be converted in the business-related field; The code system of the target field data is converted into a target code system.

5. The method according to any one of claims 1 to 4, characterized in that: After the updated data to be synchronized is synchronized to the distributed system, the method further includes: Detecting whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system; If it is detected that the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, a synchronization success message is pushed; If it is detected that the data to be synchronized in the host is inconsistent with the updated data to be synchronized in the distributed system, a synchronization failure message is pushed.

6. The method according to claim 5, characterized in that The detecting whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system includes: In the case where the data to be synchronized is stock data to be synchronized, calling a preset data consistency detection script to batch detect whether the stock data to be synchronized in the host is consistent with the updated stock data to be synchronized in the distributed system; In the case where the data to be synchronized is incremental data to be synchronized, obtaining first incremental data to be synchronized in the distributed system and second incremental data to be synchronized in the host, wherein the second incremental data to be synchronized has the same primary key as the first incremental data to be synchronized; Detect whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

7. The method according to claim 1, characterized in that The step of obtaining the data to be synchronized from the host comprises: Filter the existing data to be synchronized from the host data in the downloaded historical time period; The incremental data to be synchronized is obtained from the host data update message queue, wherein the incremental data to be synchronized includes the incremental data to be synchronized and the stock data to be synchronized.

8. A data synchronization device, characterized in that: The device comprises: An acquisition module is used to acquire the data to be synchronized from the host; A screening module, used for screening mixed code field data mixed with different code systems and application partition identifiers under different application partitions from the data to be synchronized; A processing module, used for converting the coding system of the application partition identifier and the mixed code field data into a target coding system, obtaining a target application partition identifier and a target mixed code field data, so as to update the data to be synchronized, wherein the target coding system matches the coding system of the distributed system; A synchronization module is used to synchronize the updated data to be synchronized to the distributed system.

9. The device according to claim 8, characterized in that The processing module is also used to convert each character in the field data associated with the application partition identifier into an ASSI code value, accumulate the converted ASSI code values ​​to obtain an initial application partition identifier code value, convert the code system of the initial application partition identifier code value into a target code system to obtain a target application partition identifier.

10. The device according to claim 8, characterized in that The processing module is also used to, when the mixed code field data includes multiple byte data, convert the coding of each byte data into the target coding according to the occupancy length of each byte data, and splice the multiple converted byte data to obtain the target mixed code field data.

11. The device according to claim 8, characterized in that The data synchronization device also includes a business association module, which is used to determine the business association field of the incremental data to be synchronized when the data to be synchronized is incremental data to be synchronized, determine the business type of the incremental data to be synchronized according to the business association field, determine the target field data to be converted in the business association field based on the business type, and convert the coding of the target field data into the target coding.

12. The device according to any one of claims 8 to 11, characterized in that: The data synchronization device also includes a consistency detection module, which is used to detect whether the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system. If it is detected that the data to be synchronized in the host is consistent with the updated data to be synchronized in the distributed system, a synchronization success message is pushed; if it is detected that the data to be synchronized in the host is inconsistent with the updated data to be synchronized in the distributed system, a synchronization failure message is pushed.

13. The device according to claim 12, characterized in that The consistency detection module is also used to call a preset data consistency detection script when the data to be synchronized is existing data to be synchronized, batch detect whether the existing data to be synchronized in the host is consistent with the updated existing data to be synchronized in the distributed system, and when the data to be synchronized is incremental data to be synchronized, obtain the first incremental data to be synchronized in the distributed system and the second incremental data to be synchronized in the host, the second incremental data to be synchronized has the same primary key as the first incremental data to be synchronized, and detect whether the first incremental data to be synchronized is consistent with the second incremental data to be synchronized.

14. The device according to claim 8, characterized in that The acquisition module is used to filter out the stock data to be synchronized from the downloaded host data in the historical time period, and obtain the incremental data to be synchronized from the host data update message queue. The data to be synchronized includes the incremental data to be synchronized and the stock data to be synchronized.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images