A method and device based on data synchronization between user systems

Abstract

The present invention discloses a method and device based on data synchronization between user systems, which relates to the technical field of big data processing, and solves the problem that the existing technology needs to rely on the communication status between the user system and product services for data synchronization, resulting in too high product coupling to ensure data consistency. The method includes: encapsulating a method for obtaining data from different event types of the user system, receiving a registration event request from the user system through the product service; constructing a custom data synchronization component for synchronizing the user system data to the product service, monitoring the events in the queue through the custom data synchronization component and pulling the data that needs to be synchronized in the user system; synchronizing the data that needs to be synchronized pulled by the custom data synchronization component to the product service and recording the update time, which not only reduces the coupling between the user system and the product service, but also ensures the consistency of the data.

Description

Technical Field

[0001] The present invention belongs to the technical field of big data processing, and in particular to a method and device based on data synchronization between user systems. Background Art

[0002] In some fields, the user system of services is maintained independently from the products. It only needs to synchronize the user system data to the product services. At present, the interaction between the user system and product services is mainly completed by the SDK that encapsulates the Http request, or directly calls Http. If the user system sender does not need the service receiver to respond, the receiver may fail to process the request, and the sender will not be aware of it, so data consistency cannot be guaranteed. If the sender needs the receiver to respond, although data consistency is guaranteed, it will cause the product to be too highly coupled. Summary of the invention

[0003] The purpose of the present invention is to overcome the problem that the prior art relies on the communication status between the user system and the product service for data synchronization, which results in too high product coupling to ensure data consistency, and to provide a method and device based on data synchronization between user systems.

[0004] The technical solution to achieve the purpose of the present invention is as follows: In a first aspect, a method for synchronizing data between user systems is provided, the method comprising:

[0005] Encapsulate the method of obtaining data from different event types of the user system;

[0006] Receive registration event requests from user system through product services;

[0007] Construct a custom data synchronization component for synchronizing user system data to product services; wherein the custom data synchronization component has an automatic retry strategy for synchronization failure, and registers events according to the registration event request, so as to monitor events in the queue through the custom data synchronization component and pull the data that needs to be synchronized in the user system;

[0008] Synchronize the data that needs to be synchronized pulled by the custom data synchronization component to the product service and record the update time.

[0009] Furthermore, the method for encapsulating different event types of the user system to obtain data includes:

[0010] Declare methods for obtaining data of different data types;

[0011] Based on the amount of data being queried, encapsulate a request to batch query the data change time of each piece of data.

[0012] Furthermore, the product service provides a data synchronization collector for receiving events.

[0013] Furthermore, the construction of a custom data synchronization component for synchronizing user system data to product services specifically includes:

[0014] Creating a custom event manager; the custom event manager includes a first queue and a second queue, and the events submitted by the product service are stored in the first queue;

[0015] Create a custom event manager for registering events in the first and second queues;

[0016] Call the user system interface to obtain data in the user system; among them, the failed events are stored in the second queue, and the events are registered with the custom event manager after waiting for the preset events. If the event is executed successfully, the data that needs to be synchronized and the data change time are returned to the product service.

[0017] Furthermore, the data in the user system includes users, roles and authority points.

[0018] Furthermore, the construction of a custom data synchronization component for synchronizing user system data to product services also includes:

[0019] The number of retries of the event is recorded, and the events that fail to execute and whose number of retries is less than the preset number are stored in the second queue.

[0020] Furthermore, synchronizing the data to be synchronized pulled by the custom data synchronization component to the product service and recording the update time specifically includes:

[0021] Create a custom event operator to process the data and data change time that need to be synchronized and returned to the product service;

[0022] Insert the data that needs to be synchronized and returned to the product service into the business table of the product service, and update the corresponding update time in the data record table to the latest data change time.

[0023] In a second aspect, a device for synchronizing data between user systems is provided, comprising:

[0024] Encapsulation module, used to encapsulate the method of obtaining data of different event types in the user system;

[0025] The request module is used to receive registration event requests from the user system through product services;

[0026] A data pulling module is used to build a custom data synchronization component for synchronizing user system data to product services, wherein the custom data synchronization component has an automatic retry strategy for synchronization failure, and registers events according to the registration event request, so as to monitor events in the queue through the custom data synchronization component and pull the data that needs to be synchronized in the user system;

[0027] The synchronization module is used to synchronize the data that needs to be synchronized pulled by the custom data synchronization component to the product service and record the update time.

[0028] According to a third aspect, a computer program product is provided, wherein the computer program product stores computer instructions, and when the computer instructions are executed by a processor, the method for synchronizing data between user systems is implemented.

[0029] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a program code for execution by a device, wherein the program code includes a method for executing the method based on data synchronization between user systems.

[0030] Compared with the prior art, the present invention has the following significant advantages:

[0031] (1) A custom data synchronization component is defined for synchronizing user system data to product services. It can obtain the required data from the user system and synchronize it to the product services without relying on the communication status between the user system and the product services.

[0032] (2) An automatic retry strategy for synchronization failure is set up, which can automatically retry after data synchronization is identified, reducing the coupling between the user system and product services while ensuring data consistency.

[0033] The present invention is further described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the method for data synchronization between user systems according to the present invention.

[0035] Figure 2 It is a flowchart of the method for data synchronization between user systems according to the first embodiment.

[0036] Figure 3 It is a flowchart of the custom data synchronization component in the method based on data synchronization between user systems in Example 1.

[0037] Figure 4 This is a block diagram of the structure of the device based on data synchronization between user systems in Example 2.

[0038] Figure 5 It is a schematic diagram of the internal structure of the electronic device of the fifth embodiment. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example 1

[0041] A method for synchronizing data between user systems according to Embodiment 1 of the present application is as follows: Figure 1 As shown, including:

[0042] Encapsulate the method of obtaining data from different event types of the user system;

[0043] Receive registration event requests from user system through product services;

[0044] Construct a custom data synchronization component for synchronizing user system data to product services, wherein the custom data synchronization component has an automatic retry strategy for synchronization failure, and registers events according to the registration event request, so as to listen to events in the queue through the custom data synchronization component and pull the data that needs to be synchronized in the user system;

[0045] Synchronize the data that needs to be synchronized pulled by the custom data synchronization component to the product service and record the update time

[0046] This method provides a custom data synchronization component for synchronizing user system data to product services. It can obtain the required synchronized data from the user system and synchronize it to the product services without relying on the communication status between the user system and the product services. At the same time, an automatic retry strategy for synchronization failure is set, which can automatically retry after data synchronization is identified, thereby reducing the coupling between the user system and the product services while ensuring data consistency.

[0047] Specifically, Figure 2 A flowchart of a method for synchronizing data between user systems in application embodiment 1 is shown, including:

[0048] S100, encapsulates a method for obtaining data from different event types of the user system, where event types include: user, role, and permission, etc. An event is the smallest unit of data synchronization. For example, a change in a user role is an event, and specifically includes the following steps:

[0049] S101, declare methods for obtaining data of different data types, based on the latest Update_time (the update time used to record each piece of data in the user system (such as the user table)) and support batch query in the form of Limit (page query).

[0050] S102: When the amount of data to be queried is greater than Limit (Limit is the number of pages), Pagination needs to be encapsulated (encapsulating the request for the next batch of queries), and this is repeated until the query quantity is reached. <limit。

[0051] S200, receiving a registration event request from a user system through a product service, specifically including the following steps:

[0052] S201, first the product service provides an EventClient (EventClient is a data synchronization collector for users, roles, and permission points) for receiving Events (Event is an event, for example, a change in a user role is an event, which is the smallest unit of data synchronization).

[0053] S300, constructing a custom data synchronization component for synchronizing user system data to product services, wherein the custom data synchronization component has an automatic retry strategy for synchronization failure, and registers events according to the registration event request, so as to monitor events in the queue through the custom data synchronization component and pull the data that needs to be synchronized in the user system, such as Figure 3 The following is an architecture diagram of a custom data synchronization component, which includes the following steps:

[0054] S301, create a custom EventManager class, which has two queues Quene (i.e. the first queue) and WaitQuene (i.e. the second queue, also called the waiting queue). Events submitted by product services will be stored in Quene.

[0055] S302, create a custom EventListener class for registering events in Quene and WaitQuene.

[0056] S303, calling the custom EventListener Dispitch() method, wherein the custom EventListenerDispitch() method encapsulates the relevant operations of obtaining user, role, and permission point data from the user system through http requests, and is used to call the user system system interface to obtain the user system data (user system data includes users, roles, and permission points) collected by the data synchronization collector in the above step S201. Events that fail to execute will be stored in WaitQuene, and will register events to EventListener after waiting for a preset time or after the events in Quene are executed. RetryNumber (number of retries) records the number of executions of the current event. When RetryNumber>3 times and the execution still fails, this event is no longer stored in WaitQuene; events that succeed in execution will return the data synchronized to the product service, and the Update_time of the latest changed data (that is, the update time used to record each data in the user system), wherein RetryNumber can also be other values, for example, 3, 5, 6, etc.

[0057] S400, synchronizing the data to be synchronized pulled by the custom data synchronization component to the product service and recording the update time, specifically including the following steps:

[0058] S401, when creating a custom OrganizationOperator class, it is necessary to implement the ChangeNotifier (data processing) interface class defined in the Sync Component synchronizer, and rewrite the Chage() method to process the data returned to the product service synchronization in the above step S303.

[0059] S402, inserting data into the business table of product services, and updating the CheckPoint of the Sync_record table to the latest Update_time.

[0060] In this embodiment, data monitoring is implemented based on Listiner, and Sync Component (i.e., custom data synchronization component) is used to obtain the latest Checkpoint time of the event in the synchronization record table. Each time, data with Update_time greater than or equal to the current time is pulled from the user system. After successful synchronization, the Checkpoint is updated to the latest Update_time. If the synchronization fails, the compensation strategy will be triggered. WaitQuene will record the failed event and continue to retry, which reduces the coupling between the user system and product services while ensuring data consistency.

[0061] Compared with the prior art, this embodiment integrates http requests and writes a Sync Component. When the number of queries > the defined Limit, the url will be encapsulated in the parameters until all data is obtained. At the same time, it does not depend on the communication status between services. Even if the call fails, the event will be placed in the WaitQuene of the Sync Component. RetryNumber records the number of retries, and the maximum number of retries is 3. This avoids the situation where the program keeps trying to retry after synchronization fails due to dirty data or other abnormal operations.

[0062] Example 2

[0063] like Figure 4 As shown, a device based on data synchronization between user systems involved in Embodiment 2 of the present application includes:

[0064] Encapsulation module 100, used to encapsulate the method of obtaining data of different event types of the user system;

[0065] The request module 200 is used to receive a registration event request from a user system through a product service;

[0066] The data pulling module 300 is used to construct a custom data synchronization component for synchronizing the user system data to the product service, wherein the custom data synchronization component has an automatic retry strategy for synchronization failure, and registers the event according to the registration event request, so as to monitor the events in the queue through the custom data synchronization component and pull the data that needs to be synchronized in the user system;

[0067] The synchronization module 400 is used to synchronize the data required to be synchronized pulled by the custom data synchronization component to the product service and record the update time.

[0068] It should be noted that, for other specific implementations of the device based on data synchronization between user systems in this embodiment, reference can be made to the specific implementations of the method based on data synchronization between user systems described above, and will not be repeated here to avoid redundancy.

[0069] Example 3

[0070] A computer program product involved in Example 3 of the present application is used to store a computer program. When the computer program runs on a computer, the method in any one of the implementation modes in Example 1 of the present application is implemented.

[0071] Example 4

[0072] A computer-readable storage medium according to Embodiment 4 of the present application, wherein the computer-readable storage medium stores a program code for execution by a device, wherein the program code includes steps for executing a method in any one of the implementations in Embodiment 1 of the present application;

[0073] Among them, the computer-readable storage medium can be a read-only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM); the computer-readable storage medium can store program code, and when the program stored in the computer-readable storage medium is executed by the processor, the processor is used to execute the steps of the method in any one of the implementation methods in Example 1 of the present application.

[0074] Example 5

[0075] like Figure 5 As shown, an electronic device involved in Embodiment 5 of the present application includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, a method in any one of the implementations in Embodiment 1 of the present application is implemented;

[0076] Among them, the processor can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method in any one of the implementation methods in Example 1 of the present application.

[0077] The processor may also be an integrated circuit electronic device with signal processing capability. In the implementation process, each step of the method in any implementation of Embodiment 1 of the present application may be completed by an integrated logic circuit of hardware in the processor or by instructions in software form.

[0078] The above-mentioned processor may also be a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and combines its hardware to complete the functions required to be performed by the units included in the data processing device of the embodiments of the present application, or executes the method in any one of the implementation modes in the first embodiment of the present application.

[0079] The above are only preferred specific implementations of the present application; however, the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solution and its improved ideas of the present application within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method based on data synchronization between user systems, characterized in that: The method comprises: Encapsulate the method of obtaining data from different event types of the user system; Receive registration event requests from user system through product services; Construct a custom data synchronization component for synchronizing user system data to product services; wherein the custom data synchronization component has an automatic retry strategy for synchronization failure, and registers events according to the registration event request, so as to monitor events in the queue through the custom data synchronization component and pull the data that needs to be synchronized in the user system; Synchronize the data that needs to be synchronized pulled by the custom data synchronization component to the product service and record the update time.

2. The method for data synchronization between user systems according to claim 1, characterized in that: The method for encapsulating different event types of the user system to obtain data includes: Declare methods for obtaining data of different data types; Based on the amount of data being queried, encapsulate a request to batch query the data change time of each piece of data.

3. The method for data synchronization between user systems according to claim 1, characterized in that: The product service provides a data synchronization collector for receiving events.

4. The method for data synchronization between user systems according to claim 1, characterized in that: The construction of a custom data synchronization component for synchronizing user system data to product services specifically includes: Creating a custom event manager; the custom event manager includes a first queue and a second queue, and the events submitted by the product service are stored in the first queue; Create a custom event manager for registering events in the first and second queues; Call the user system interface to obtain data in the user system; among them, the failed events are stored in the second queue, and the events are registered with the custom event manager after waiting for the preset events. If the event is executed successfully, the data that needs to be synchronized and the data change time are returned to the product service.

5. The method for data synchronization between user systems according to claim 4, characterized in that: The data in the user system includes users, roles and authority points.

6. The method for data synchronization between user systems according to claim 4, characterized in that: The construction of a custom data synchronization component for synchronizing user system data to product services also includes: The number of retries of the event is recorded, and the events that fail to execute and whose number of retries is less than the preset number are stored in the second queue.

7. The method for data synchronization between user systems according to claim 4, characterized in that: The step of synchronizing the data required to be synchronized pulled by the custom data synchronization component to the product service and recording the update time specifically includes: Create a custom event operator to process the data and data change time that need to be synchronized and returned to the product service; Insert the data that needs to be synchronized and returned to the product service into the business table of the product service, and update the corresponding update time in the data record table to the latest data change time.

8. A device for synchronizing data between user systems based on the method according to any one of claims 1 to 7, characterized in that: include: Encapsulation module, used to encapsulate the method of obtaining data of different event types in the user system; The request module is used to receive registration event requests from the user system through product services; A data pulling module is used to build a custom data synchronization component for synchronizing user system data to product services, wherein the custom data synchronization component has an automatic retry strategy for synchronization failure, and registers events according to the registration event request, so as to monitor events in the queue through the custom data synchronization component and pull the data that needs to be synchronized in the user system; The synchronization module is used to synchronize the data that needs to be synchronized pulled by the custom data synchronization component to the product service and record the update time.

9. A computer program product, characterized in that The computer program product stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program code for execution by a device, wherein the program code includes a program code for executing the method according to any one of claims 1 to 7.

Images