Data synchronization method and apparatus, computer device, and readable storage medium

Abstract

The application relates to the field of big data, and specifically discloses a data synchronization method and device, computer equipment, a computer readable storage medium, and a computer program product. The method comprises the following steps: obtaining to-be-synchronized data of a host; screening mixed code field data mixed with different codes and application partition identifiers under different application partitions from the to-be-synchronized data; converting the application partition identifiers and the code of the mixed code field data into a target code to obtain target application partition identifiers and target mixed code field data, so as to update the to-be-synchronized data, wherein the target code matches the code of a distributed system; and synchronizing the updated to-be-synchronized data to the distributed system. The method can improve the efficiency of data synchronization.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and in particular to a data synchronization method and device, computer equipment, a computer readable storage medium, and a computer program product. BACKGROUND

[0002] With the development of science and technology, many original old systems will be updated and iterated. In order to preserve the data in the old system, the data in the old system can be synchronized to the database of the new system.

[0003] Taking the synchronization of host data to a distributed system as an example, in the traditional technology, the commonly used data synchronization methods include but are not limited to manual Structured Query Language (SQL) script synchronization, Extract-Transform-Load (ETL) tool migration, etc. Specifically, the manual SQL script synchronization refers to manually writing a data script to synchronize the data in the host to the distributed system through the data script; the ETL tool migration refers to regularly extracting the data in the host and loading it to the distributed system through a professional ETL tool. However, the above data synchronization methods still have the problem of low efficiency of data synchronization. SUMMARY

[0004] Therefore, it is necessary to provide an efficient data synchronization method, device, computer equipment, computer readable storage medium, and computer program product in view of the above technical problems.

[0005] In a first aspect, the present application provides a data synchronization method, comprising:

[0006] obtaining to-be-synchronized data of a host;

[0007] screening mixed code field data mixed with different codes and application partition identifiers under different application partitions from the to-be-synchronized data;

[0008] converting the application partition identifiers and the code of the mixed code field data into a target code to obtain target application partition identifiers and target mixed code field data, so as to update the to-be-synchronized data, and the target code matches the code of the distributed system;

[0009] synchronizing the updated to-be-synchronized data to the distributed system.

[0010] In one embodiment, converting the code of the application partition identifier into the target code to obtain the target application partition identifier comprises:

[0011] converting each character in the field data associated with the application partition identifier into an ASCII code value;

[0012] accumulating the converted ASSIC code values to obtain an initial application partition identification code value;

[0013] converting the code system of the initial application partition identification code value into a target code system to obtain a target application partition identification.

[0014] In one embodiment, the method further comprises:

[0015] In the case where the mixed field data comprises a plurality of byte data, the code system of each byte data is converted into the target code system according to the length of the placeholder of each byte data.

[0016] The plurality of converted byte data are spliced to obtain the target mixed field data.

[0017] In one embodiment, the method further comprises:

[0018] In the case where the to-be-synchronized data is to-be-synchronized incremental data, the service association field of the to-be-synchronized incremental data is determined.

[0019] The service type of the to-be-synchronized incremental data is determined according to the service association field.

[0020] Based on the service type, the target field data to be converted in the service association field is determined.

[0021] The code system of the target field data is converted into the target code system.

[0022] In one embodiment, after the updated to-be-synchronized data is synchronized to the distributed system, the method further comprises:

[0023] It is detected whether the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent.

[0024] If it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, a synchronization success message is pushed.

[0025] If it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are inconsistent, a synchronization failure message is pushed.

[0026] In one embodiment, the detection of whether the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent comprises:

[0027] In the case where the to-be-synchronized data is to-be-synchronized stock data, a preset data consistency detection script is called to batch detect whether the to-be-synchronized stock data in the host and the updated to-be-synchronized stock data in the distributed system are consistent.

[0028] In a case where the to-be-synchronized data is to-be-synchronized incremental data, the first to-be-synchronized incremental data in the distributed system and the second to-be-synchronized incremental data in the host are obtained, and the second to-be-synchronized incremental data has the same primary key as the first to-be-synchronized incremental data;

[0029] It is detected whether the first to-be-synchronized incremental data is consistent with the second to-be-synchronized incremental data.

[0030] In one of the embodiments, the to-be-synchronized data of the host is obtained, including:

[0031] The to-be-synchronized stock data is filtered from the downloaded host data in the historical time period;

[0032] The to-be-synchronized incremental data is obtained from the host data update message queue, and the to-be-synchronized data includes the to-be-synchronized incremental data and the to-be-synchronized stock data.

[0033] In a second aspect, the application further provides a data synchronization device, including:

[0034] The obtaining module is configured to obtain the to-be-synchronized data of the host;

[0035] The filtering module is configured to filter the mixed code field data having different code systems and the application partition identifier under different application partitions from the to-be-synchronized data;

[0036] The processing module is configured to convert the application partition identifier and the code system of the mixed code field data into a target code system to obtain a target application partition identifier and target mixed code field data, so as to update the to-be-synchronized data, and the target code system matches the code system of the distributed system;

[0037] The synchronization module is configured to synchronize the updated to-be-synchronized data to the distributed system.

[0038] In one of the embodiments, the processing module is further configured to convert each character in the field data associated with the application partition identifier into an ASSIC code value, accumulate the converted ASSIC code values to obtain an initial application partition identifier code value, convert the code system of the initial application partition identifier code value into the target code system to obtain the target application partition identifier.

[0039] In one of the embodiments, the processing module is further configured to, in a case where the mixed code field data includes a plurality of byte data, convert the code system of each byte data into the target code system according to the length of the placeholder of each byte data, splice the plurality of converted byte data to obtain the target mixed code field data.

[0040] In one of the embodiments, the data synchronization device further comprises a service association module, which is configured to, in the case that the to-be-synchronized data is to-be-synchronized incremental data, determine a service association field of the to-be-synchronized incremental data, determine a service type of the to-be-synchronized incremental data according to the service association field, determine target field data to be converted in the service association field based on the service type, and convert a code system of the target field data to a target code system.

[0041] In one of the embodiments, the data synchronization device further comprises a consistency detection module, which is configured to detect whether the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, push a synchronization success message if it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, and push a synchronization failure message if it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are inconsistent.

[0042] In one of the embodiments, the consistency detection module is further configured to, in the case that the to-be-synchronized data is to-be-synchronized inventory data, call a preset data consistency detection script to batch detect whether the to-be-synchronized inventory data in the host and the updated to-be-synchronized inventory data in the distributed system are consistent, and in the case that the to-be-synchronized data is to-be-synchronized incremental data, acquire first to-be-synchronized incremental data in the distributed system and second to-be-synchronized incremental data in the host, the primary keys of the second to-be-synchronized incremental data and the first to-be-synchronized incremental data being the same, and detect whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent.

[0043] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program:

[0044] acquiring to-be-synchronized data of a host;

[0045] screening mixed code field data mixed with different code systems and application partition identifiers under different application partitions from the to-be-synchronized data;

[0046] converting the application partition identifiers and the code systems of the mixed code field data to a target code system to obtain target application partition identifiers and target mixed code field data, so as to update the to-be-synchronized data, the target code system matching a code system of the distributed system;

[0047] synchronizing the updated to-be-synchronized data to the distributed system.

[0048] In a fourth aspect, the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the following steps:

[0049] acquiring to-be-synchronized data of a host;

[0050] filtering mixed code field data mixed with different code systems and application partition identifiers under different application partitions from the to-be-synchronized data;

[0051] converting the application partition identifiers and the code system of the mixed code field data into a target code system to obtain target application partition identifiers and target mixed code field data, so as to update the to-be-synchronized data, the target code system being matched with the code system of the distributed system;

[0052] synchronizing the updated to-be-synchronized data into the distributed system.

[0053] In a fifth aspect, the present application further provides a computer program product comprising a computer program which, when executed by a processor, implements the following steps:

[0054] obtaining to-be-synchronized data of a host;

[0055] filtering mixed code field data mixed with different code systems and application partition identifiers under different application partitions from the to-be-synchronized data;

[0056] converting the application partition identifiers and the code system of the mixed code field data into a target code system to obtain target application partition identifiers and target mixed code field data, so as to update the to-be-synchronized data, the target code system being matched with the code system of the distributed system;

[0057] synchronizing the updated to-be-synchronized data into the distributed system.

[0058] The data synchronization method, device, computer device, computer readable storage medium and computer program product described above are different from the traditional method of directly synchronizing host data into a distributed system. A code value conversion mechanism with strong applicability and flexibility is proposed. Specifically, first, mixed code field data mixed with different code systems and application partition identifiers under different application partitions are filtered from the to-be-synchronized data, which can reduce unnecessary data code system conversion, improve the efficiency of data synchronization, and then the application partition identifiers and the code system of the mixed code field data are uniformly converted into a code system matched with the distributed system to update the to-be-synchronized data, which can simplify the data synchronization processing between different systems, improve the consistency and compatibility of data in the distributed system, reduce the phenomenon of garbled code caused by inconsistent code systems, further improve the data synchronization efficiency, and finally synchronize the updated to-be-synchronized data into the distributed system, which can make the synchronized host data applicable to the requirements of the distributed system. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained on the basis of these drawings without creative labor.

[0060] Figure 1 An application environment diagram of a data synchronization method in an embodiment;

[0061] Figure 2 A flowchart of a data synchronization method in an embodiment;

[0062] Figure 3 A flowchart of a data synchronization method in another embodiment;

[0063] Figure 4 A flowchart of a data synchronization method in still another embodiment;

[0064] Figure 5 A flowchart of a data synchronization method in yet another embodiment;

[0065] Figure 6 A detailed flowchart of a data synchronization method in an embodiment;

[0066] Figure 7 A structural block diagram of a data synchronization device in an embodiment;

[0067] Figure 8 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0068] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the present application with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are used to explain the present application, and are not used to limit the present application. It should be noted that in the embodiments of the present application, some software, groups, models and other industry existing solutions may be mentioned, which should be considered as exemplary, and the purpose is to illustrate the feasibility of the technical solution implementation of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0069] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to host data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations. The acquisition, storage, use, processing, etc. of data in the technical solutions of the present application comply with the relevant regulations of national laws and regulations.

[0070] The data synchronization method provided by the embodiments of the present application can be applied to an application environment as shown in the figure. Figure 1 The host 102 communicates with the server 104 through the network, and the distributed system 106 communicates with the server 104. The data synchronization system 108 is deployed on the server 104, and the server 104 can obtain the to-be-synchronized data sent by the host 102 through the data synchronization system 108 to perform code conversion of the to-be-synchronized data, obtain updated to-be-synchronized data, and then synchronize the updated to-be-synchronized data to the distributed system 106. The data storage system can be integrated on the server 104.

[0071] The host 102 sends to-be-synchronized data to the server 104 in response to a data synchronization demand initiated by a user or when the host 102 meets a preset data synchronization demand condition. The preset data synchronization demand condition includes but is not limited to a preset time condition, a preset location condition, etc.

[0072] The server 104 obtains the to-be-synchronized data of the host, filters mixed code field data with different codes and application partition identifiers under different application partitions from the to-be-synchronized data, converts the application partition identifiers and the code of the mixed code field data into a target code suitable for the requirements of the distributed system 106, obtains target application partition identifiers and target mixed code field data, updates the to-be-synchronized data, and synchronizes the updated to-be-synchronized data to the distributed system 106.

[0073] The terminal 102 can be but is not limited to various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0074] In one exemplary embodiment, as shown in Figure 2 a data synchronization method is provided, which is applied to the server 104 in Figure 1 for example. In which:

[0075] S200, obtaining the to-be-synchronized data of the host.

[0076] Among them, the to-be-synchronized data is a plurality of data in the host, based on the business requirement, this part of data needs to be retained in the distributed system.

[0077] Specifically, the host sends the to-be-synchronized data to the server in response to the data synchronization demand initiated by the user, or when the host meets the preset data synchronization demand condition, wherein the preset data synchronization demand condition includes but is not limited to the preset time condition, the preset place condition, etc.

[0078] S400, from the to-be-synchronized data, screening mixed code field data mixed with different code systems and application partition identifiers under different application partitions.

[0079] Among them, in order to facilitate memory and search, certain rules must be followed when compiling codes, these rules are called code systems, such as code systems including but not limited to binary, hexadecimal, and octal, etc. The mixed code field data is mixed with data of different code systems, such as a certain field data may contain binary data, and may also have hexadecimal data, then the field data can be called mixed code field data.

[0080] The application partition refers to the division area of the application data, and the application partition identifier in the present application generally consists of five letters, which is used to distinguish the type of application partition, such as A country, B country, etc. National type.

[0081] Specifically, the to-be-synchronized data is not only one, and each to-be-synchronized data contains data of a plurality of fields. In order to meet the data format requirement of distributed storage, it is necessary to judge whether the data of each field is mixed code field data mixed with different code systems, if yes, it is determined that the field data is mixed code field data, and subsequent code system conversion is performed on the mixed code field data, if not, it is determined that the field data is not mixed code field data, and subsequent code system conversion is not required for the field data.

[0082] The application partition identifier is an identifier data related to the business in the to-be-synchronized data, and therefore, the application partition identifier also needs to be converted to meet the data format requirements of the distributed system application and storage. Further, the specific code bit of the application partition identifier in the to-be-synchronized data has been pre-set, for example, the code bit of the application partition identifier is set to belong to the 0-4 bits in each code bit of the to-be-synchronized data. Based on the pre-set code bit of the application partition identifier, the application partition identifier under the application partition where the to-be-synchronized data is located is filtered from the to-be-synchronized data.

[0083] S600, converting the application partition identifier and the code system of the mixed code field data into a target code system to obtain target application partition identifier and target mixed code field data to update the to-be-synchronized data.

[0084] The target code system matches the code system of the distributed system application partition identifier. Generally, the target code system in the distributed system is UTF-8 (8-bit, Universal Character Set / Unicode Transformation Format) code data. UTF-8 is a variable-length character encoding for Unicode, which is used to represent any character in the Unicode standard, and the first byte in its encoding is still compatible with ASCII, so that the original software processing ASCII characters can continue to be used without or only a small part of modification.

[0085] Specifically, the to-be-synchronized data of the host has a data storage format and storage manner unique to the host. In order to successfully store the data in the distributed system, the mixed code field data mixed with different code systems and the application partition identifier related to the business can be processed through the job number conversion mode of JCL (Job Control Language, Job Control Language), that is, the code system of the application partition identifier and the mixed code field data is converted to a code system matching the distributed system application partition identifier to obtain target application partition identifier and target mixed code field data to update the to-be-synchronized data. The code system of the target application partition identifier and the target mixed code field data is the target code system.

[0086] S800, synchronizing the updated to-be-synchronized data to the distributed system.

[0087] Specifically, the data synchronization system deployed on the server synchronizes the updated to-be-synchronized data to the distributed system through the XCOM (Xtensiable Communication Of Middleware, Middleware Extensible Communication) interface function.

[0088] In one embodiment, before the updated to-be-synchronized data is synchronized into the distributed system, because the total length of the data of the host and the distributed system is not the same, the updated to-be-synchronized data needs to be packaged in terms of data length, so that the data length of the updated to-be-synchronized data is converted to a target length that meets the data length requirement of the distributed system, so that the distributed system can receive the data, and subsequent data storage, processing, and the like of the converted to-be-synchronized data are performed.

[0089] In another embodiment, after the code conversion is completed, a separator symbol “|” is automatically added at the front and the end of each converted field data, and after the updated to-be-synchronized data is received by the distributed system, the updated to-be-synchronized data falls into the database of the distributed system according to the separator. That is, the field data between each two separators is found to be in the corresponding field of the database of the distributed system, and is cut off and stored according to the field.

[0090] In the above data synchronization method, unlike the traditional method of directly synchronizing the data of the host into the distributed system, a code value conversion mechanism with strong applicability and flexibility is proposed. Specifically, first, the mixed code field data mixed with different codes and the application partition identifier under different application partitions are screened from the to-be-synchronized data, which can reduce unnecessary data synchronization, improve the efficiency of data synchronization, and then the application partition identifier and the code of the mixed code field data are uniformly converted into the code of the distributed system, the to-be-synchronized data is updated, which can simplify the data synchronization processing between different systems, improve the consistency and compatibility of the data in the distributed system, reduce the garbled code phenomenon caused by inconsistent codes, further improve the data synchronization efficiency, and finally synchronize the updated to-be-synchronized data into the distributed system, which can make the synchronized host data applicable to the requirements of the distributed system.

[0091] In one exemplary embodiment, as shown in FIG. 6, Figure 3 S600 includes:

[0092] S620, converting each character in the field data associated with the application partition identifier into an ASSIC code value.

[0093] S640, accumulating the converted ASSIC code values to obtain an initial application partition identifier code value.

[0094] S660, converting the code of the initial application partition identifier code value into a target code to obtain a target application partition identifier.

[0095] S680, converting the code of the mixed code field data into a target code to obtain target mixed code field data to update the to-be-synchronized data.

[0096] Wherein, the ASSIC code refers to using a specified 7-bit or 8-bit binary combination to represent 128 or 256 possible characters.

[0097] Specifically, the application partition identifier can be composed of several letters. Due to the needs of the business, it is necessary to convert the application partition identifier of the host into an application partition identifier that meets the needs of the distributed system. Specifically, the code conversion of the application partition identifier not only converts the code system of the application partition identifier into the target code system, but also needs the participation of the field data associated with the application partition identifier, such as the account classification number, the account classification date, etc., that is, each character of the associated field data is converted into an ASSIC code, and each ASSIC code value is superimposed, and the superimposed ASSIC code value is taken as the initial application partition identifier code value. The code system of the initial application partition identifier code value is converted from the ASSIC code to the target code system, and the result obtained after the conversion is determined as the target application partition identifier. That is, the conversion of the code system of the application partition identifier into the target code system is actually a conversion according to the field data associated with the application partition identifier, and an updated target application partition identifier is obtained.

[0098] Further, in the process of converting each character of the preset field data of the other preset field into an ASSIC code, the data synchronization system first converts the code system of each character of the preset field data into hexadecimal, and then converts it from hexadecimal to ASSIC code.

[0099] In the above embodiment, by converting the code system of each character of the field data associated with the application partition identifier into an ASSIC code, accumulating each ASSIC code value, obtaining an initial application partition identifier code value, and converting the initial application partition identifier code value from the ASSIC code to the target code system, the target application partition identifier is obtained. Accurate application partition identifier can be accurately obtained based on target field data, and since the code system of the field data associated with the application partition identifier has been converted, the application partition identifier that can be accurately synchronized to the distributed system can be obtained.

[0100] In an exemplary embodiment, converting the code system of the mixed code field data into the target code system to obtain the target mixed code field data further includes:

[0101] In the case where the mixed code field data includes a plurality of byte data, the code system of each byte data is converted into the target code system according to the length of each byte data, and a plurality of converted byte data is spliced to obtain the target mixed code field data.

[0102] Wherein, the mixed code field can be a large field, and the large field refers to a field containing multiple bytes.

[0103] Specifically, in order to improve the efficiency and accuracy of the mixed field data code conversion, the mixed field data can be divided into multiple byte data according to the field byte code of the data. Then, the code of each byte data is converted into the target code according to the length of the byte data, and the target mixed field data is obtained by splicing the converted byte data.

[0104] For example, the ABC corresponds to the mixed field data of nine bytes, wherein A corresponds to 0-2 byte data, B corresponds to 3-5 byte data, and C corresponds to 6-8 byte data. During the code conversion, 0-2 byte data, 3-5 byte data, and 6-8 byte data are converted into three letters, and then the three letters are spliced into complete target mixed field data.

[0105] In one of the embodiments, the code of each byte data is converted into the target code, including: screening out the byte data whose code is not the target code, and converting the code of the byte data into the target code.

[0106] For example, the target code is UTF-8 code, and there are mixed code data of UTF-8 code and EBCDIC code in the mixed field data. During the data synchronization to the distributed system, first, the mixed field data is divided into multiple byte data according to the length of each byte data in the mixed field data. Some byte data is in UTF-8 code, and the other byte data is in EBCDIC code. The code of the byte data in EBCDIC code is converted into UTF-8 code.

[0107] In the above embodiment, by converting the code of each byte data into the target code according to the length of each byte data when the mixed field data includes multiple byte data, and splicing the converted byte data, the codes of multiple byte data are converted into the target code, which reduces the conversion time and improves the efficiency of data synchronization. Further, compared with converting the code of longer byte data, the accuracy of the target mixed field data obtained by converting the code of shorter byte data is higher.

[0108] In one exemplary embodiment, as shown in Figure 4 the method further includes:

[0109] S120, in the case that the to-be-synchronized data is to-be-synchronized incremental data, determining the business association field of the to-be-synchronized incremental data.

[0110] S140, determining the business type of the to-be-synchronized incremental data according to the business association field.

[0111] S160, determining the target field data to be converted in the business association field based on the business type.

[0112] S180, converting the code system of the target field data into a target code system.

[0113] The service-related field refers to a field related to a service. Different services obtain different data and are stored in a data table of the host.

[0114] Specifically, the to-be-synchronized incremental data includes a plurality of fields, some of which are service-related fields. Different services correspond to different field truncation positions, and the service-related fields of the to-be-synchronized incremental data are determined according to the field truncation positions.

[0115] Since different service types correspond to different service-related fields, the service type of the to-be-synchronized incremental data can be determined according to the service-related fields, that is, the service type of the to-be-synchronized incremental data is determined.

[0116] Further, among the plurality of service-related fields corresponding to different service types, some are target fields that need to be converted in code system. Therefore, the target field to be converted in the service-related field can be determined through the data conversion rule corresponding to the service type, and then the target field data of the target field is converted into data with a target code system. Specifically, the target field to be converted in the service-related field is determined, that is, the field whose data code system is not the target code system in the service-related field is determined.

[0117] For example, the service-related field of the to-be-synchronized incremental data is determined to be a field related to card opening through the field truncation position. The service type of the to-be-synchronized incremental data is determined to be a card opening type according to the service-related field. For the card opening type, the field in the service-related field that needs to be converted in code system is determined, for example, the data corresponding to the card opening time and the card opening account number needs to be converted in code system, and the field data that needs to be converted in code system is converted into data with a target code system.

[0118] In the above embodiment, when the to-be-synchronized data is to-be-synchronized incremental data, the service type of the to-be-synchronized incremental data is determined, and the target field data to be converted can be accurately converted in code system based on the service type, thereby improving the accuracy of data synchronization.

[0119] In one exemplary embodiment, as shown in FIG. 8, after S800, the method further includes: Figure 5

[0120] S920, detecting whether the to-be-synchronized data in the host is consistent with the updated to-be-synchronized data in the distributed system.

[0121] S940, if it is detected that the to-be-synchronized data in the host is consistent with the updated to-be-synchronized data in the distributed system, a synchronization success message is pushed.​

[0122] S960, if it is detected that the to-be-synchronized data in the host is inconsistent with the updated to-be-synchronized data in the distributed system, a synchronization failure message is pushed.

[0123] Specifically, after the data synchronization is completed, it is needed to detect whether the to-be-synchronized data in the host is consistent with the updated to-be-synchronized data in the distributed system, so as to verify the process of the data synchronization.

[0124] Further, it is needed to detect whether the to-be-synchronized data in the host is consistent with the updated to-be-synchronized data in the distributed system, that is, whether the to-be-synchronized volume data in the host is consistent with the updated to-be-synchronized volume data in the distributed system, and whether the to-be-synchronized incremental data in the host is consistent with the updated to-be-synchronized incremental data in the distributed system.

[0125] In the case that both are consistent, a synchronization success message is generated and pushed. In the case that the to-be-synchronized volume data in the host is inconsistent with the updated to-be-synchronized volume data in the distributed system, or the to-be-synchronized volume incremental data in the host is inconsistent with the updated to-be-synchronized volume incremental data in the distributed system, a synchronization failure message is generated and pushed.

[0126] In the above embodiment, by detecting whether the to-be-synchronized data in the host is consistent with the updated to-be-synchronized data in the distributed system, the process of the data synchronization can be accurately verified, so as to further increase the reliability of the data synchronization process.

[0127] In an exemplary embodiment, as shown in FIG. 9, S920 comprises: Figure 6

[0128] S922, in the case that the to-be-synchronized data is to-be-synchronized volume data, a preset data consistency detection script is called to batch detect whether the to-be-synchronized volume data in the host is consistent with the updated to-be-synchronized volume data in the distributed system.

[0129] S924, in the case that the to-be-synchronized data is to-be-synchronized incremental data, first to-be-synchronized incremental data in the distributed system and second to-be-synchronized incremental data in the host are obtained, and the primary keys of the second to-be-synchronized incremental data and the first to-be-synchronized incremental data are the same.

[0130] S926, it is detected whether the first to-be-synchronized incremental data is consistent with the second to-be-synchronized incremental data.

[0131] Specifically, the data consistency detection process of the to-be-synchronized volume data and the to-be-synchronized incremental data is different.

[0132] ​For the to-be-synchronized inventory data, a preset data consistency detection script is called to batch detect whether the to-be-synchronized inventory data in the host and the updated to-be-synchronized inventory data in the distributed system are consistent.

[0133] For the to-be-synchronized incremental data, the data consistency is ensured through single comparison of sql statements. Specifically, in the process of data synchronization, the host contains a plurality of second to-be-synchronized incremental data, and the distributed system contains a plurality of first to-be-synchronized incremental data after the data synchronization is completed.

[0134] The primary keys of each first to-be-synchronized incremental data in the distributed system and each second to-be-synchronized incremental data in the host are recorded, and the first to-be-synchronized incremental data and the second to-be-synchronized incremental data corresponding to each same primary key are queried. Whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data of the same primary key are consistent is detected.

[0135] For example, the code system of the first to-be-synchronized incremental data is UTF-8 code, and the code system of the second to-be-synchronized incremental data is EBCDIC code. The specific detection process of detecting whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data of the same primary key are consistent includes the following three cases:

[0136] Firstly, the first to-be-synchronized incremental data is converted into data with the code system of EBCDIC code, and whether the converted first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent is detected.

[0137] Secondly, the second to-be-synchronized incremental data is converted into data with the code system of UTF-8 code, and whether the converted second to-be-synchronized incremental data and the first to-be-synchronized incremental data are consistent is detected.

[0138] Thirdly, the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are collectively converted into data with an intermediate code system, and whether the converted first to-be-synchronized incremental data and the converted second to-be-synchronized incremental data are consistent is detected.

[0139] In one embodiment, in the process of detecting whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data of the same primary key are consistent, not only the conversion of the code system is needed, but also some special business field data needs to be converted into field data with the same field format.

[0140] In one embodiment, when detecting whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data of the same primary key are consistent, the fields of the to-be-synchronized incremental data can be compared one by one to ensure that the converted data of each field is correct. In actual application, some specific field can be set not to be compared.

[0141] In the above embodiment, in the case that the to-be-synchronized data is to-be-synchronized stock data, whether the to-be-synchronized stock data in the host and the updated to-be-synchronized stock data in the distributed system are consistent is detected in batches, thereby improving the efficiency of detecting the data synchronization process of the to-be-synchronized stock data. In the case that the to-be-synchronized data is to-be-synchronized incremental data, consistency detection is performed on each to-be-synchronized incremental data with the same primary key in the host and the distributed system, thereby improving the accuracy of data synchronization verification.

[0142] In an exemplary embodiment, the to-be-synchronized data of the host is obtained, including:

[0143] The to-be-synchronized stock data is filtered from the downloaded host data in the historical time period, and the to-be-synchronized incremental data is obtained from the host data update message queue, and the to-be-synchronized data includes the to-be-synchronized incremental data and the to-be-synchronized stock data.

[0144] The to-be-synchronized incremental data includes the updated data of the host in the current time period. The to-be-synchronized data includes the to-be-synchronized incremental data and the to-be-synchronized stock data.

[0145] Specifically, in this embodiment, data synchronization needs to be performed on both the to-be-synchronized incremental data and the to-be-synchronized stock data.

[0146] For the to-be-synchronized stock data in the historical time period, the host data in the historical time period is downloaded by a down program and stored in a file, and the server filters the to-be-synchronized stock data from the file. Commonly used down programs include but are not limited to GBAFGUNL program.

[0147] For the to-be-synchronized incremental data in the current time period, in order to ensure the timeliness of the data, the to-be-synchronized incremental data needs to be synchronized through the message queue. That is, when the host data is increased, deleted, or modified, the host sends the sql statement recording each increase, deletion, or modification operation to the server through the message queue.

[0148] In this embodiment, by obtaining the to-be-synchronized incremental data and the to-be-synchronized stock data of the host, the to-be-synchronized data of the host can be fully synchronized to the distributed system, the possibility of some important data not being synchronized is reduced, and the accuracy of data synchronization is improved.

[0149] In a specific application embodiment, the to-be-synchronized data of the host is synchronized to the distributed system, and the data synchronization method used includes:

[0150] I. Data synchronization

[0151] For to-be-synchronized stock data:

[0152] 1) Use a down program to download the to-be-synchronized stock data.

[0153] 2) The server obtains the downloaded to-be-synchronized inventory data, and modifies the to-be-synchronized inventory data, including:

[0154] A. For application partition identification: convert each character in the field data associated with the application partition identification, such as the reconciliation classification date, the reconciliation classification number, etc., into an ASSIC code, accumulate each ASSIC code value, obtain an initial application partition identification code value, convert the code system of the initial application partition identification code value into a target code system, and obtain a target application partition identification.

[0155] B. For mixed code field data: the mixed code field data is mixed with field data of different code systems. Convert the code system of the mixed code field data that does not belong to the target code system into the target code system to obtain target mixed code field data. For example, if a certain mixed code field data contains both UTF-8 code and EBCDIC code data, the EBCDIC code can be uniformly converted into UTF-8 code. Further, in the case where the mixed code field data includes multiple byte data, the code system of each byte data can be converted into the target code system according to the length of each byte data, and the multiple converted byte data can be spliced to obtain the target mixed code field data.

[0156] Further, in the process of code system conversion, a separator is automatically added to the front and back of each converted field data.

[0157] 3) Data preprocessing. Since the length range required to be met by the to-be-synchronized inventory data of the host and the distributed system is different, the to-be-synchronized inventory data of the host can be packaged to meet the length range required to be met by the to-be-synchronized inventory data of the distributed system.

[0158] 4) Download. The to-be-synchronized inventory data after preprocessing is sent to the distributed system by using the XCOM interface function.

[0159] 5) Database storage. After receiving the to-be-synchronized inventory data, the distributed system stores the to-be-synchronized inventory data in the database of the distributed system according to the separator.

[0160] For to-be-synchronized incremental data:

[0161] In order to ensure the timeliness of the data, the to-be-synchronized incremental data needs to be synchronized with additional rules.

[0162] 1) Host message queue. In the case where the host data is added, deleted, modified, etc., the host sends the operation sql statement record to the server in the form of a message queue, and the server takes this part of data as to-be-synchronized incremental data and performs data operations on the to-be-synchronized incremental data.

[0163] 2) Special rule conversion. The code conversion of the distributed system is roughly divided into three categories, including:

[0164] A, for application partition identification: as described above, the same as the to-be-synchronized inventory data, which will not be described here.

[0165] B, for mixed code field data: as described above, the same as the to-be-synchronized inventory data, which will not be described here.

[0166] C, for business association field: determine the business association field of the to-be-synchronized incremental data, determine the business type of the to-be-synchronized incremental data according to the business association field, based on the business type, adopt the conversion mode matched with the business type, determine the target field data to be converted in the business association field, and convert the code system of the target field data to the target code system.

[0167] 3) Data preprocessing. As described above, the same as the to-be-synchronized inventory data, which will not be described here.

[0168] 4) Data storage and database. The processed data is stored and landed.

[0169] Two, consistency detection after data synchronization is completed:

[0170] For to-be-synchronized inventory data: call the preset data consistency detection script to batch detect whether the to-be-synchronized inventory data in the host and the updated to-be-synchronized inventory data in the distributed system are consistent.

[0171] For to-be-synchronized incremental data: obtain the first to-be-synchronized incremental data in the distributed system with the same primary key and the second to-be-synchronized incremental data in the host, and detect whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent. Detecting whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent includes: converting the first to-be-synchronized incremental data and the second to-be-synchronized incremental data into data with the same code system, and then detecting whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent.

[0172] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiments of the present application also provide a data synchronization device for implementing the data synchronization method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more data synchronization device embodiments provided below can refer to the limitations of the data synchronization method described above, which will not be repeated here.

[0174] In one exemplary embodiment, as shown in Figure 7 A data synchronization device 900 is provided, comprising an acquisition module 200, a screening module 400, a processing module 600 and a synchronization module 800, wherein:

[0175] The acquisition module 200 is configured to acquire the to-be-synchronized data of the host;

[0176] The screening module 400 is configured to screen mixed code field data and application partition identifiers under different application partitions from the to-be-synchronized data;

[0177] The processing module 600 is configured to convert the application partition identifier and the code system of the mixed code field data into a target code system to obtain a target application partition identifier and target mixed code field data, so as to update the to-be-synchronized data, and the target code system matches the code system of the distributed system;

[0178] The synchronization module 800 is configured to synchronize the updated to-be-synchronized data to the distributed system.

[0179] In one embodiment, the processing module is further configured to convert each character in the field data associated with the application partition identifier into an ASSIC code value, accumulate the converted ASSIC code values to obtain an initial application partition identifier code value, convert the code system of the initial application partition identifier code value into a target code system to obtain a target application partition identifier.

[0180] In one embodiment, the processing module is further configured to, in the case that the mixed code field data comprises a plurality of byte data, convert the code system of each byte data into a target code system according to the length of each byte data, splice the plurality of converted byte data to obtain target mixed code field data.

[0181] In one embodiment, the data synchronization device further comprises a service association module, the service association module is configured to, in the case that the to-be-synchronized data is to-be-synchronized incremental data, determine a service association field of the to-be-synchronized incremental data, determine a service type of the to-be-synchronized incremental data according to the service association field, determine target field data to be converted in the service association field based on the service type, and convert the code system of the target field data into a target code system.

[0182] In one of the embodiments, the data synchronization apparatus further comprises a consistency detection module, which is configured to detect whether the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, and push a synchronization success message if the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, or push a synchronization failure message if the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are inconsistent.

[0183] In one of the embodiments, the consistency detection module is further configured to, in a case where the to-be-synchronized data is to-be-synchronized inventory data, invoke a preset data consistency detection script to batch detect whether the to-be-synchronized inventory data in the host and the updated to-be-synchronized inventory data in the distributed system are consistent, and in a case where the to-be-synchronized data is to-be-synchronized incremental data, acquire first to-be-synchronized incremental data in the distributed system and second to-be-synchronized incremental data in the host, the second to-be-synchronized incremental data having the same primary key as the first to-be-synchronized incremental data, and detect whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent.

[0184] The modules in the data synchronization apparatus described above can be implemented in whole or in part by software, hardware, or a combination thereof. The modules described above can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform the operations corresponding to the modules.

[0185] In one exemplary embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in Figure 8 The computer device comprises a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store to-be-synchronized data of a host and other data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement a data synchronization method.

[0186] Those skilled in the art can understand that, Figure 8The structure shown in the figure is a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0187] In one of the embodiments, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0188] In one of the embodiments, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0189] In one of the embodiments, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0190] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0191] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0192] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data synchronization method, characterized by, The method comprises: acquiring to-be-synchronized data of a host; from the to-be-synchronized data, screening mixed-code field data mixed with different codes and application partition identifiers under different application partitions; converting the code of the application partition identifier and the mixed-code field data into a target code to obtain a target application partition identifier and target mixed-code field data, so as to update the to-be-synchronized data, wherein the target code matches the code of a distributed system, and converting the code of the application partition identifier into a target code to obtain a target application partition identifier comprises: converting each character in the field data associated with the application partition identifier into an ASSIC code value; accumulating the obtained ASSIC code values to obtain an initial application partition identifier code value; and converting the code of the initial application partition identifier code value into a target code to obtain a target application partition identifier; synchronously updating the updated to-be-synchronized data into the distributed system.

2. The method of claim 1, wherein, Converting the code of the mixed-code field data into a target code to obtain target mixed-code field data further comprises: in the case where the mixed-code field data comprises multiple byte data, converting the code of each byte data into a target code according to the length of the position occupied by each byte data; splicing the multiple converted byte data to obtain target mixed-code field data.

3. The method of claim 1, wherein, The method further comprises: in the case where the to-be-synchronized data is to-be-synchronized incremental data, determining a service association field of the to-be-synchronized incremental data; determining the service type of the to-be-synchronized incremental data according to the service association field; based on the service type, determining target field data to be converted in the service association field; converting the code of the target field data into a target code.

4. The method according to any one of claims 1 to 3, characterized in that, After the updated to-be-synchronized data is synchronously updated into the distributed system, the method further comprises: detecting whether the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent; if it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, pushing a synchronization success message; if it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are inconsistent, pushing a synchronization failure message.

5. The method of claim 4, wherein, The detection of whether the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent comprises: in the case where the to-be-synchronized data is to-be-synchronized inventory data, calling a preset data consistency detection script to batch-detect whether the to-be-synchronized inventory data in the host and the updated to-be-synchronized inventory data in the distributed system are consistent; in the case where the to-be-synchronized data is to-be-synchronized incremental data, acquiring first to-be-synchronized incremental data in the distributed system and second to-be-synchronized incremental data in the host, wherein the primary keys of the second to-be-synchronized incremental data and the first to-be-synchronized incremental data are the same; detecting whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent.

6. The method of claim 1, wherein, The acquisition of the to-be-synchronized data of the host comprises: screening to-be-synchronized inventory data from downloaded historical host data in a historical time period; Obtaining to-be-synchronized incremental data from a host data update message queue, the to-be-synchronized data including to-be-synchronized incremental data and to-be-synchronized stock data.

7. A data synchronization apparatus, characterized by comprising: The device comprises: An obtaining module for obtaining to-be-synchronized data of a host; A screening module for screening mixed code field data of different codes and application partition identifiers under different application partitions from the to-be-synchronized data; A processing module for converting the code of the application partition identifier and the code of the mixed code field data into a target code to obtain a target application partition identifier and target mixed code field data, so as to update the to-be-synchronized data, wherein the target code matches the code of a distributed system, and the conversion of the code of the application partition identifier into the target code to obtain the target application partition identifier comprises converting each character in the field data associated with the application partition identifier into an ASSIC code value, accumulating the converted ASSIC code values to obtain an initial application partition identifier code value, and converting the code of the initial application partition identifier code value into the target code to obtain the target application partition identifier; A synchronization module for synchronizing the updated to-be-synchronized data to the distributed system.

8. The apparatus of claim 7, wherein, The processing module is further configured to, in the case where the mixed code field data includes multiple byte data, convert the code of each byte data into the target code according to the length of the placeholder of each byte data, splice the multiple converted byte data to obtain the target mixed code field data.

9. The apparatus of claim 7, wherein, The data synchronization device further comprises a service association module configured to, in the case where the to-be-synchronized data is to-be-synchronized incremental data, determine a service association field of the to-be-synchronized incremental data, determine the service type of the to-be-synchronized incremental data according to the service association field, determine the target field data to be converted in the service association field based on the service type, and convert the code of the target field data into the target code.

10. The apparatus of any of claims 7-9, wherein, The data synchronization device further comprises a consistency detection module configured to detect whether the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, push a synchronization success message if it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are consistent, and push a synchronization failure message if it is detected that the to-be-synchronized data in the host and the updated to-be-synchronized data in the distributed system are inconsistent.

11. The apparatus of claim 10, wherein, The consistency detection module is further configured to, in the case where the to-be-synchronized data is to-be-synchronized stock data, call a preset data consistency detection script to batch detect whether the to-be-synchronized stock data in the host and the updated to-be-synchronized stock data in the distributed system are consistent, and, in the case where the to-be-synchronized data is to-be-synchronized incremental data, obtain first to-be-synchronized incremental data in the distributed system and second to-be-synchronized incremental data in the host, the primary keys of the first to-be-synchronized incremental data and the second to-be-synchronized incremental data being the same, and detect whether the first to-be-synchronized incremental data and the second to-be-synchronized incremental data are consistent.

12. The apparatus of claim 7, wherein, The acquisition module is configured to filter out the to-be-synchronized inventory data from the downloaded host data in the historical time period, and acquire the to-be-synchronized incremental data from the host data update message queue, wherein the to-be-synchronized data comprises the to-be-synchronized incremental data and the to-be-synchronized inventory data.

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor, when executing the computer program, implements the steps of the method of any one of claims 1 to 6.

14. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

15. A computer program product comprising a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6. The computer program, when executed by the processor, implements the steps of the method of any one of claims 1 to 6.

Images