Data synchronization method and device

Abstract

The invention discloses a data synchronization method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the following steps: setting a data change annotation, wherein the data change annotation comprises a data change operation record; and when the data change annotation contains a new data change operation record, sending a data change message to a system, so that the system performs a data updating operation based on the data change message. According to the embodiment, real-time synchronization of the data can be realized, the data synchronization efficiency is improved, and the accuracy of a system business process is guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a method for data synchronization. Background Art

[0002] Currently, when building large platforms, due to the diversity of functions to be implemented, data is usually stored in different service systems, and each system presents function points according to different modules. However, when it comes to paging queries, it is impossible to assemble interfaces to meet the requirements, and it is necessary to process and integrate the data of each system.

[0003] A common method for data processing is data heterogeneity. Data synchronization between different systems is achieved by means of timed data pulling. However, the real-time performance of the data obtained by this method is very poor, real-time data synchronization cannot be performed, and the situation of untimely data updates easily occurs, affecting the processes of subsequent business systems. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a method for data synchronization, which can achieve real-time synchronization of data in each system, and further meet the business requirements of each system.

[0005] To achieve the above object, according to the first aspect of the embodiments of the present invention, a method for data synchronization is provided, including:

[0006] Set a data change annotation, where the data change annotation includes a data change operation record;

[0007] When the data change annotation contains a new data change operation record, send a data change message to the system for the system to perform a data update operation based on the data change information.

[0008] Optionally, the step of when the data change annotation contains a new data change operation record, sending a data change message to the system includes:

[0009] When the data change annotation contains a new data change operation record, determine the data information corresponding to the new data change operation record;

[0010] Send the data change message to the system that subscribes to the data change message of the data information.

[0011] According to the second aspect of the embodiments of the present invention, a method for data synchronization is provided, including:

[0012] Obtain a data change message, where the data change message is the message obtained by the method of the first aspect;

[0013] Obtain the changed data corresponding to the data information from the sending end of the data change message based on the data information in the data change message;

[0014] Store the changed data.

[0015] Optionally, before obtaining the changed data corresponding to the data information from the sending end of the data change message based on the data information in the data change message, the method further includes:

[0016] Filter and deduplicate the data change message;

[0017] Store the processed data change message in a remote dictionary service (redis) queue;

[0018] Aggregate the data change messages in the redis queue to obtain an aggregated message;

[0019] The obtaining the changed data corresponding to the data information from the sending end of the data change message based on the data information in the data change message includes: obtaining the changed data from the sending end of the data change message according to the aggregated message.

[0020] Optionally, the filtering and deduplicating the data change message includes:

[0021] Filter the data change message according to a preset filtering condition to obtain a first message;

[0022] Generate an identifier of the message content of the first message through a hash function;

[0023] Deduplicate the messages corresponding to the identifiers that are the same as a preset identifier, where the preset identifier is an identifier calculated by a hash algorithm for historical messages.

[0024] Optionally, the obtaining the changed data corresponding to the data information from the sending end of the data change message based on the data information in the data change message includes:

[0025] Obtain the data corresponding to the data information from the sending end of the data change message based on the data information in the data change message;

[0026] Assemble the data corresponding to the data information and the data information to obtain the changed data in the ES data structure.

[0027] According to the third aspect of the embodiments of the present invention, there is provided a data synchronization device, including:

[0028] A setting module, configured to set a data change annotation, where the data change annotation includes a data change operation record;

[0029] A sending module, configured to send a data change message to a system when the data change annotation contains a record of a new data change operation, so that the system performs a data update operation based on the data change information.

[0030] According to a fourth aspect of the embodiments of the present invention, there is provided a data synchronization device, including:

[0031] An obtaining module, configured to obtain a data change message, where the data change message is a message obtained by the method according to the first aspect; and based on the data information in the data change message, obtain changed data corresponding to the data information from the sending end of the data change message;

[0032] A storage module, configured to store the changed data.

[0033] According to a fifth aspect of the embodiments of the present invention, there is provided a data simulation device, including:

[0034] One or more processors; a storage device, configured to store one or more programs,

[0035] When the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the first or second aspect of the embodiments of the present invention.

[0036] According to a sixth aspect of the embodiments of the present invention, there is provided a computer-readable medium, on which a computer program is stored, and when the program is executed by a processor, the method provided in the first or second aspect of the embodiments of the present invention is implemented.

[0037] One embodiment of the above invention has the following advantages or beneficial effects: It can obtain the change situation of the data of each system in real time through the set data change annotation, and send a change message to the system that needs to perform synchronization to perform corresponding data update operations, realizing real-time data synchronization, improving the data synchronization efficiency, and ensuring the accuracy of the system business process.

[0038] The further effects of the above non-conventional optional manners will be described in conjunction with specific embodiments below. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings are used to better understand the present invention and do not constitute an improper limitation to the present invention. Among them:

[0040] Figure 1 is a schematic diagram of the main process of the data synchronization method according to the embodiments of the present invention;

[0041] Figure 2 is a schematic diagram of the main process of another data synchronization method according to the embodiments of the present invention;

[0042] Figure 3 is a flowchart of a data synchronization method provided by an embodiment of the present invention;

[0043] Figure 4 is a schematic flowchart of data processing provided by an embodiment of the present invention;

[0044] Figure 5 is a schematic diagram of the main modules of a data synchronization device according to an embodiment of the present invention;

[0045] Figure 6 is a schematic diagram of the main modules of another data synchronization device according to an embodiment of the present invention;

[0046] Figure 7 is an exemplary system architecture diagram to which an embodiment of the present invention can be applied;

[0047] Figure 8 is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. Detailed implementation manners

[0048] The following describes exemplary embodiments of the present invention with reference to the accompanying drawings. Various details of the embodiments of the present invention are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted below.

[0049] It should be noted that in the technical solutions of the present disclosure, in terms of the collection, acquisition, update, analysis, processing, use, transmission, storage, etc. of user personal information, they all comply with the provisions of relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs. Necessary measures are taken for user personal information to prevent illegal access to user personal information data, and to safeguard user personal information security, network security, and national security.

[0050] Figure 1 is a data synchronization method according to an embodiment of the present invention, which is applied to the first system of the platform, and specifically can be any system in the platform where there is a data change operation, such as Figure 1 As shown, it includes:

[0051] Step S101, set a data change annotation, and the data change annotation includes a data change operation record.

[0052] Add a general data change annotation, such as @DataChanged, to the computer programming language (Java) code of each business service system to identify the change operations of data objects in the system. Specifically, when @DataChanged is set, the data recorded by @DataChanged is empty. When data changes occur in the system, a data change record will be generated, recording the changed data and the data change operations. Among them, the data change can be the user's operation of changing data, or the data change operation generated when the system conducts business, etc., which is not limited here.

[0053] Step S102, when the data change annotation contains a new data change operation record, send a data change message to the system for the system to perform a data update operation based on the data change information.

[0054] When a new data change operation record is added to the data change annotation @DataChanged, trigger the action of sending a change message and send a data change message to other systems that need to synchronize data. The data change message stores information about the changed data, such as the primary key field of the changed data, for other systems to perform data update operations on the corresponding data according to the data change message.

[0055] According to the data synchronization method provided by the embodiment of the present invention, it is possible to obtain the change situation of the data of each system in real time through the set data change annotation, and send a change message to the system that needs to be synchronized to perform the corresponding data update operation, realizing real-time data synchronization, improving the data synchronization efficiency, and ensuring the accuracy of the system business process.

[0056] Optionally, when the data change annotation contains a new data change operation record, sending a data change message to the system includes: when the data change annotation contains a new data change operation record, determining the data information corresponding to the new data change operation record; sending a data change message to the system that subscribes to the data change message of the data information. When there is a record of a new data change operation in @DataChanged, trigger the sending of a data change message. First, determine the information of the data that has changed according to the record of the new data change operation, determine the system that subscribes to the change message of this data according to the information, and send a change message containing the primary key field of the data to the subscribed system.

[0057] Specifically, determine the changed data according to the record of the new data change operation, and then determine the identifier of the changed data. According to the pre-stored association relationship between the data identifier and the system that subscribes to the data identifier, send the change message of this data to the corresponding system that subscribes to this data identifier.

[0058] Optionally, when the data change annotation contains a record of a new data change operation, a data change message is sent to all systems of platforms other than the system containing the new data change operation, or a data change message is sent to the systems that need to receive the change message according to the preset settings.

[0059] According to the data synchronization method provided by the embodiments of the present invention, the change situation of the data of each system can be obtained in real time through the set data change annotation, and a change message is sent to the systems that need to be synchronized and subscribe to the change messages of the corresponding data, so as to perform corresponding data update operations, realize real-time data synchronization, improve the data synchronization efficiency, and ensure the accuracy of the system business process. Based on the mechanism of sending data change messages, the triggering and processing of data change operations can be carried out asynchronously, improving the throughput and response performance of the system. The system does not need to wait for the processing results of other systems and can continue to execute other business logics.

[0060] Figure 2 It is a data synchronization method according to an embodiment of the present invention, applied to a second system, specifically a system other than the first system in the platform, such as Figure 2 as shown, including:

[0061] Step S201, obtain a data change message, where the data change message is a message obtained through the above data synchronization method.

[0062] According to the subscription information, obtain the data change message sent by the first system. The subscription information is the information of the data of other systems in the platform subscribed by the second system according to actual needs; the data change message is a message containing the information of the changed data. For example, the information of data 1 in system B subscribed by system A.

[0063] Step S202, based on the data information in the data change message, obtain the changed data corresponding to the data information from the sending end of the data change message.

[0064] Determine the system with data change and the changed data information according to the obtained data change message, and obtain the data content corresponding to the changed data information from the system with data change.

[0065] Specifically, the data information stored in the data change message is the primary key field of the data. According to the primary key field, the changed data can be determined, and by means of reverse lookup, the complete data corresponding to the primary key field is obtained from the system with data change.

[0066] Step S203, store the changed data.

[0067] Update and store the data of the obtained system in the database of this system to complete the data synchronization operation.

[0068] According to the data synchronization method provided by the embodiments of the present invention, it is possible to determine the system with data changes and the changed data information based on the obtained data change messages, and then obtain the changed data from the changed system. Real-time synchronization of system data is achieved, ensuring the business requirements of the system and the accuracy of the system business process.

[0069] Optionally, before obtaining the changed data corresponding to the data information from the sending end of the data change message based on the data information in the data change message, the method further includes: performing filtering and duplicate removal processing on the data change message; storing the processed data change message in a redis queue; aggregating the data change messages in the redis queue to obtain an aggregated message; obtaining the changed data corresponding to the data information from the sending end of the data change message based on the data information in the data change message, including: obtaining the changed data from the sending end of the data change message according to the aggregated message. Before obtaining the changed data according to the data change message, it is necessary to perform preprocessing on the data change message. Specifically, according to the pre-set message filtering conditions, the data change message is filtered, and the filtered message is subjected to duplicate removal processing using a hash algorithm. Among them, the message filtering conditions can be set through a configuration file or an Application Programming Interface (API) interface, and the content of the message filtering conditions can be restrictions on the message content, message type, message tags, etc. For example, the setting of the message tag filtering condition can be to filter the data change message and retain the message with the tag a. The filtered message contents are compared, and the data change messages with the same message content are subjected to duplicate removal processing. The data after duplicate removal processing is stored in the redis queue. By scanning the redis queue through a timer, the data change messages of each business service system in the queue are aggregated according to the business type to obtain a unified type of message body. The aggregated messages are further de-duplicated, that is, when different business systems perform the same business, there may be the same processing of data, and then two data change messages are generated, and these messages are de-duplicated. The de-duplicated messages are stored in a Message Queue (MQ).

[0070] According to the data synchronization method provided by the embodiments of the present invention, it is possible to obtain the change situation of the data of each system in real time through the set data change annotation, and send a data change message to the system that needs to be synchronized to perform corresponding data update operations, so as to achieve real-time synchronization of data, improve the data synchronization efficiency, and ensure the accuracy of the system business process. At the same time, filtering and duplicate removal of the obtained data change messages according to the preset rules can greatly reduce the amount of data transmitted and improve the transmission efficiency.

[0071] Optionally, filter and deduplicate the data change messages in the Redis queue, including: filtering the data change messages in the Redis queue according to preset filtering conditions to obtain the first messages; generating an identifier for the message content of the first messages through a hash function; deduplicating the messages corresponding to the same identifier as the preset identifier, where the preset identifier is the identifier calculated by generating through a hash function for historical messages through a hashing algorithm. Deduplicate the data change messages filtered according to the filtering conditions. Specifically: calculate and generate a unique identifier for the content of the filtered data change messages through a hash function, and match it with the pre-stored message identifier set. When a consistent message identifier is matched, determine that the data change message is a duplicate message and delete the data change message. If no consistent message identifier is matched, determine that the message is a new data change message, retain the message, and store the message identifier of the data change message in the message identifier set. Among them, the message identifiers in the pre-stored message identifier set are the identifiers calculated by historical data change messages through a hash function.

[0072] Optionally, based on the data information in the data change message, obtain the changed data corresponding to the data information from the sending end of the data change message, including: based on the data information in the data change message, obtain the data corresponding to the data information from the sending end of the data change message; assemble the data corresponding to the data information and the data information to obtain the changed data in the ES data structure. According to the data information stored in the obtained change message, such as data primary key fields, data identifiers, etc., query the corresponding data information from the database of the sending end of the data change message, and then determine the data to be obtained and obtain the data. For example, if the data information stored in the data change message is the data primary key field a1, query the a1 field from the sending end of the data change message and obtain the data under the a1 field. Assemble the obtained data and the corresponding data information to obtain the data of the search server (Elastic Search, ES).

[0073] Optionally, store the changed data in the ES database, specifically including: determining whether there is target information in the ES database that is consistent with the data information in the changed data; when there is target information in the ES database that is consistent with the data information in the changed data, update the data corresponding to the target information based on the changed data; when there is no target information in the ES database that is consistent with the data information in the changed data, store the changed data in the ES database.

[0074] Optionally, when any system within the platform is the first system, any other system can be the second system; when any other system is the first system, any of the above systems is the second system.

[0075] Optionally, when processing the acquired data, an exception compensation operation is provided to ensure data consistency. Specifically, when an exception occurs during the data processing, corresponding compensation operations can be performed, such as program error reporting, incomplete data, interface timeout, etc. Different types of error reports are processed differently. For example, in case of timeout, message retry is performed; in case of program error reporting or incomplete data, the message is pushed to the error retry queue and an alarm is sent for manual troubleshooting. After the problem is solved, the error queue is retried.

[0076] Optionally, a monitoring function is provided during the heterogeneous process based on the data change message, and the progress and status of the heterogeneity can be viewed in real time. Specifically, the queue message has a visual interface, and the message backlog, consumption, and retry situations can be viewed. The program has different levels of alarms and interface performance metrics.

[0077] According to the data synchronization method provided by the embodiments of the present invention, the change situation of the data of each system can be obtained in real time through the set data change annotation, and a change message is sent to the system that needs to perform synchronization, so that the system that needs to be synchronized can obtain the changed data from the message sending end according to the data change message to perform corresponding data update operations, realizing real-time data synchronization, improving the data synchronization efficiency, and ensuring the accuracy of the system business process. Based on the mechanism of sending data change messages, the triggering and processing of data change operations can be performed asynchronously, improving the throughput and response performance of the system. The system does not need to wait for the processing results of other systems and can continue to execute other business logics. At the same time, the filtering and deduplication processing of data change messages can reduce the data transmission volume, improve the data transmission efficiency, and further improve the data synchronization efficiency.

[0078] Figure 3 A flowchart of a data synchronization method is provided, as shown in the figure, including:

[0079] Services A, B, and C are different service systems on the same platform. Each system identifies changes in system operations through the general annotations it accesses, specifically changes in business data. Each system triggers itself to send business change messages to other systems on the same platform based on the business change situation. The target system that receives the business change message filters and deduplicates the message, and stores the processed message in the redis queue. The target system scans the redis queue through a timer to obtain the change message and aggregates the message. Among them, the data primary key that needs to be heterogeneous is stored in the message body, that is, the fields in the message body are used as query conditions to obtain the primary key of the master data that needs to be heterogeneous. The target system deduplicates the aggregated message and stores the deduplicated message in the MQ queue. The target system processes the message in the MQ queue, including initializing the queue with data obtained from the service system according to the message, initializing the data in the queue with the data primary key and the obtained data in the message, and assembling it through the business interface to obtain data in the ES data structure. Among them, data initialization can be performed by incrementing the primary key in the database to extract data. Before assembling the data, the target system also needs to delete the data in the exception queue. The target system performs ES processing on the obtained ES-structured data.

[0080] Figure 4 A schematic diagram of a data processing flow is provided, as shown in the figure, including: judging the message in the MQ queue and judging the message type. When the message is to delete data, lock the database data. If the lock acquisition fails, go to retry. If successful, assemble the data, operate on the ES database, and release the lock after the operation ends. When the message is to update data, compare it with the version of the data in the ES to judge whether the message needs to be processed. If the version of the data in the message is lower than the version of the data in the ES, it means the message is a historical message and does not need to be processed, so the message is ignored. If the version of the data in the message is higher, it means it needs to be processed, and data is locked. If the lock acquisition fails, go to retry. Otherwise, judge whether the data primary key in the message exists in the ES. If it exists, set the data as the update type; if it does not exist, set the data as the new type. Assemble the data and operate on the ES. Release the lock after the operation ends.

[0081] Figure 5 It is a schematic diagram of a data synchronization device 500 provided by an embodiment of the present invention, including:

[0082] A setting module 501, used to set data change annotations, and the data change annotations include data change operation records;

[0083] A sending module 502, used to send a data change message to the system when the data change annotation contains a new data change operation record, so that the system can perform a data update operation based on the data change information.

[0084] The data synchronization device provided by an embodiment of the present invention can obtain the change situations of the data of each system in real time through the set data change annotations, and send change messages to the systems that need to perform synchronization to perform corresponding data update operations, realizing real-time data synchronization, improving the data synchronization efficiency, and ensuring the accuracy of the system business process. Based on the mechanism of sending data change messages, the triggering and processing of data change operations can be performed asynchronously, improving the throughput and response performance of the system. The system does not need to wait for the processing results of other systems and can continue to execute other business logics.

[0085] Optionally, a sending module 502 is configured to send a data change message to a system when the data change annotation contains a new data change operation record, including: a determining module 503 is configured to determine the data information corresponding to the new data change operation record when the data change annotation contains a new data change operation record; the sending module 502 is configured to send a data change message to the system that subscribes to the data change message of the data information.

[0086] The data synchronization device provided by an embodiment of the present invention can obtain the change situations of the data of each system in real time through the set data change annotations, and send change messages to the systems that need to perform synchronization and subscribe to the change messages of the corresponding data to perform corresponding data update operations, realizing real-time data synchronization, improving the data synchronization efficiency, and ensuring the accuracy of the system business process.

[0087] Figure 6 is a schematic diagram of a data synchronization device 600 provided by an embodiment of the present invention, including:

[0088] An obtaining module 601 is configured to obtain a data change message, where the data change message is a message obtained through the above data synchronization method; based on the data information in the data change message, obtain the changed data corresponding to the data information from the sending end of the data change message;

[0089] A storage module 602 is configured to store the changed data.

[0090] The data synchronization device provided by an embodiment of the present invention can determine the system where the data change occurs and the changed data information based on the obtained data change message, and then obtain the changed data from the changed system. Realize real-time synchronization of system data and ensure the business requirements of the system and the accuracy of the system business process.

[0091] Optionally, the device further includes: a processing module 603 for filtering and deduplicating the data change message; a storage module 602 for storing the processed data change message in a redis queue; a processing module 603 for aggregating the data change messages in the redis queue to obtain an aggregated message; an obtaining module 601 for obtaining, based on the data information in the data change message, the changed data corresponding to the data information from the sending end of the data change message, including: the obtaining module 601 for obtaining the changed data according to the aggregated message from the sending end of the data change message.

[0092] According to the data synchronization device provided by the embodiment of the present invention, it can obtain the change situation of the data of each system in real time through the set data change annotation, and send a data change message to the system that needs to perform synchronization to perform corresponding data update operations, so as to achieve real-time data synchronization, improve the data synchronization efficiency, and ensure the accuracy of the system business process. At the same time, filtering and deduplicating the obtained data change messages according to preset rules can greatly reduce the amount of data transmitted and improve the transmission efficiency.

[0093] Optionally, the processing module 603 for filtering and deduplicating the data change message includes: the processing module 603 for filtering the data change message according to a preset filtering condition to obtain a first message; a calculation module 604 for generating an identifier of the message content of the first message through a hash function; the processing module 603 for deduplicating the messages corresponding to the same identifier as the preset identifier, and the preset identifier is the identifier calculated by a hash algorithm for historical messages.

[0094] Optionally, the obtaining module 601 for obtaining, based on the data information in the data change message, the changed data corresponding to the data information from the sending end of the data change message includes: the obtaining module 601 for obtaining, based on the data information in the data change message, the data corresponding to the data information from the sending end of the data change message; an assembling module 605 for assembling the data corresponding to the data information and the data information to obtain the changed data in the ES data structure.

[0095] The data synchronization device provided by an embodiment of the present invention can obtain the change situation of each system's data in real time through the set data change annotation, and send a change message to the system that needs to perform synchronization, so that the system that needs to be synchronized can obtain the changed data from the message sending end according to the data change message to perform corresponding data update operations, realizing real-time data synchronization, improving the data synchronization efficiency, and ensuring the accuracy of the system business process. Based on the mechanism of data change message sending, the triggering and processing of data change operations can be carried out asynchronously, improving the throughput and response performance of the system. The system does not need to wait for the processing results of other systems and can continue to execute other business logics. At the same time, the filtering and deduplication processing of data change messages can reduce the data transmission volume, improve the data transmission efficiency, and further improve the data synchronization efficiency.

[0096] Figure 7 FIG. 700 shows an exemplary system architecture to which the data synchronization method or the data synchronization device according to an embodiment of the present invention can be applied.

[0097] As Figure 7 shown, the system architecture 700 may include terminal devices 701, 702, 703, a network 704, and a server 705. The network 704 is used to provide a medium for communication links between the terminal devices 701, 702, 703 and the server 705. The network 704 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0098] Users can use the terminal devices 701, 702, 703 to interact with the server 705 through the network 704 to receive or send messages, etc. Various communication client applications may be installed on the terminal devices 701, 702, 703, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0099] The terminal devices 701, 702, 703 may be various electronic devices with a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop portable computers, and desktop computers, etc.

[0100] The server 705 may be a server providing various services, such as a background management server (only for example) that supports the shopping websites browsed by users using the terminal devices 701, 702, 703. The background management server may analyze and process data such as product information query requests received, and feedback the processing results (such as target push information, product information - only for example) to the terminal devices.

[0101] It should be noted that the data synchronization method provided by the embodiments of the present invention is generally executed by the server 705. Correspondingly, the data synchronization device is generally disposed in the server 705.

[0102] It should be understood that Figure 7 the number of terminal devices, networks, and servers in

[0103] Reference will be made below to Figure 8 which shows a schematic structural diagram of a computer system 800 of a terminal device suitable for implementing the embodiments of the present invention. Figure 8 The terminal device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present invention.

[0104] As Figure 8 shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage section 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the system 800 are also stored. The CPU 801, ROM 802, and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0105] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, etc.; an output section 807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, a modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed so that a computer program read from it can be installed into the storage section 808 as needed.

[0106] In particular, according to the embodiments disclosed in the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present invention include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 809 and / or installed from the removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the above-described functions defined in the system of the present invention are performed.

[0107] It should be noted that the computer-readable medium shown in the present invention can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. And in the present invention, a computer-readable signal medium can include a data signal propagated in a baseband or as part of a carrier wave, which carries the computer-readable program code. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted by any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0108] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a portion of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as combinations of blocks in the block diagram or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0109] The modules described in the embodiments of the present invention can be implemented in software or in hardware. The described modules can also be provided in a processor. For example, it can be described as: a processor includes a setting module and a sending module. Among them, the names of these modules do not constitute a limitation on the module itself in some cases. For example, the setting module can also be described as "the module for setting data change annotations".

[0110] As another aspect, the present invention also provides a computer-readable medium. This computer-readable medium can be included in the device described in the above embodiments; or it can exist separately without being assembled into the device. The above computer-readable medium carries one or more programs. When the one or more programs are executed by a device, the device includes: setting data change annotations, where the data change annotations include data change operation records; when a new data change operation record is included in the data change annotations, sending a data change message to the system for the system to perform a data update operation based on the data change information.

[0111] Or it includes: obtaining a data change message, where the data change message is a message obtained by the above data synchronization method; based on the data information in the data change message, obtaining the changed data corresponding to the data information from the sending end of the data change message; storing the changed data.

[0112] According to the technical solution of the embodiments of the present invention, the change situations of the data of each system are obtained in real time through the set data change annotations, and a change message is sent to the system that needs to perform synchronization for corresponding data update operations, realizing real-time data synchronization, improving the data synchronization efficiency, and ensuring the accuracy of the system business process.

[0113] The above specific embodiments do not constitute a limitation to the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for data synchronization, characterized in that, Including: Set a data change annotation, where the data change annotation includes a data change operation record; When the data change annotation contains a new data change operation record, send a data change message to the system for the system to perform a data update operation based on the data change information.

2. The method according to claim 1, wherein The step of, when the data change annotation contains a new data change operation record, sending a data change message to the system includes: When the data change annotation contains a new data change operation record, determine the data information corresponding to the new data change operation record; Send the data change message to the system that subscribes to the data change message of the data information.

3. A method for data synchronization, characterized in that, Including: Obtain a data change message, where the data change message is the message obtained by the method according to any one of claims 1-2; Based on the data information in the data change message, obtain the changed data corresponding to the data information from the sender of the data change message; Store the changed data.

4. The method according to claim 3, characterized in that Before the step of, based on the data information in the data change message, obtaining the changed data corresponding to the data information from the sender of the data change message, the method further includes: Perform filtering and duplicate removal processing on the data change message; Store the processed data change message in a redis queue; Aggregate the data change messages in the redis queue to obtain an aggregated message; The step of, based on the data information in the data change message, obtaining the changed data corresponding to the data information from the sender of the data change message includes: obtaining the changed data from the sender of the data change message according to the aggregated message.

5. The method according to claim 4, wherein The step of performing filtering and duplicate removal processing on the data change message includes: Filter the data change message according to a preset filtering condition to obtain a first message; Generate an identifier of the message content of the first message through a hash function; Perform duplicate removal processing on the messages corresponding to the same identifier as the preset identifier, where the preset identifier is the identifier calculated by a hash algorithm for historical messages.

6. The method according to claim 3, wherein The step of, based on the data information in the data change message, obtaining the changed data corresponding to the data information from the sender of the data change message includes: Based on the data information in the data change message, obtain the data corresponding to the data information from the sender of the data change message; Assemble the data corresponding to the data information and the data information to obtain the changed data in the ES data structure.

7. A device for data synchronization, characterized in that, Including: A setting module for setting a data change annotation, where the data change annotation includes a data change operation record; A sending module for, when the data change annotation contains a new data change operation record, sending a data change message to the system for the system to perform a data update operation based on the data change information.

8. A device for data synchronization, characterized in that, Including: An obtaining module for obtaining a data change message, where the data change message is the message obtained by the method according to any one of claims 1-2; Based on the data information in the data change message, obtain the changed data corresponding to the data information from the sender of the data change message; A storage module for storing the changed data.

9. An electronic device, characterized in that, Including: One or more processors; A storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-6.

10. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, the method according to any one of claims 1-6 is implemented.

Images