User data synchronization method based on message queue

Abstract

The invention belongs to the technical field of Internet application integration, and discloses a message queue-based user data synchronization method, which comprises the following steps of: initializing modules of a unified user authentication center; configuring and starting a message queue module, setting a basic structure of a message queue, and configuring a message encryption mechanism; a message sender carries out encryption and idempotence check before sending a message, sends the message to a message queue, and records a sending log; message transmission and handshake confirmation; calling and full-amount updating of the user data synchronization API, wherein full-amount updating is carried out by calling a system-level API; the application system receives and processes data changes in real time; application and scheduling of message priorities; and monitoring and maintaining the operation state of the system. By setting the message queue to synchronize the user data, the data can be updated and processed more timely, a transaction processing method is adopted, the consistency of the message data is ensured, and the situation that the user data between application systems is inconsistent is avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of Internet application integration, and in particular to a user data synchronization method based on a message queue. Background Art

[0002] In today's era of rapid development of informatization, more and more enterprises are facing the challenges of data integration and system integration. There are more and more informatization application systems within enterprises, and user data exists relatively independently in each application system, lacking unified management. It is necessary to build a user-centric application system to manage user data in a unified manner. In a distributed system, the synchronization of user data is an important issue. In order to ensure the consistency of user data between multiple application systems, an effective synchronization mechanism needs to be adopted.

[0003] Message Queue (MQ) is a communication method between applications. Message refers to the data transmitted between applications. Messages can be very simple, such as only containing text strings, or more complex, possibly containing embedded objects. Messages can be returned immediately after being sent, and the message system ensures the reliable delivery of messages. Message publishers only need to publish messages to MQ without worrying about who will retrieve them, and message users only need to retrieve messages from MQ without worrying about who published them. In this way, publishers and users do not need to know the existence of each other.

[0004] However, at present, after building a unified user-centered application system, the traditional practice is that the user application system provides a user data API interface, which generally only provides a paged acquisition method, and does not provide a full acquisition method. Other application systems call the API interface through scheduled tasks and other methods, pull user data through paging, and store it in the database corresponding to their own application system. In this process, due to the scheduled task method, data updates may not be timely, resulting in inconsistent user data between application systems. Summary of the invention

[0005] The purpose of the present invention is to provide a user data synchronization method based on message queue in order to solve the above problems, so as to improve the problems of untimely user data update and inconsistent user data between application systems in the existing user data synchronization method.

[0006] The technical solution adopted by the present invention is as follows: a user data synchronization method based on a message queue, the method comprising the following steps: S100: module initialization of a unified user authentication center, including initialization of a user authentication module and initialization of a developer center module; S200: configuration and startup of a message queue module, setting the basic structure of a message queue, including queue topics, message priorities and message persistence, and configuring a message encryption mechanism; the message sender performs encryption and idempotence checks before sending the message, sends the message to the message queue, and records a sending log; S300: message delivery and handshake confirmation; S400: calling and fully updating a user data synchronization API, and performing a fully updating by calling a system-level API; S500: the application system receives and processes data changes in real time; S600: application and scheduling of message priorities, setting priorities for different types of messages according to different business requirements; S700: monitoring and maintaining the system operation status.

[0007] Furthermore, the step S100 includes: initialization of the user authentication module: the user registers and logs in through the unified user authentication center, and the user registration is performed according to the protocol format specified by the developer center. After the registration is completed, the unified user authentication center calls the system-level API interface to perform a full update initialization operation on the relevant user data, and then the user can log in to the unified user authentication center; initialization of the developer center module: setting a set of standard protocols and specifications to describe the requirements for application system access, and relevant personnel review the user data. If there is no problem, the application unique identifier appld and the key appSecret are issued, and appld and appSecret are provided to the application system.

[0008] Furthermore, in step S200, when the message is transmitted, the application performs a persistence operation and sets a priority for the data message so that the message queue processes the data first and then distributes the data to the application system that has subscribed to the topic message.

[0009] Furthermore, in step S300, the message queue's message delivery and processing includes the following: performing a three-way handshake confirmation operation on the message during the message delivery and reception process, and if the handshake fails, repeating the message multiple times; adding a transaction to the message at the same time, if the message delivery fails, the system rolls back the transaction and retries to deliver the information; performing persistence operations at the same time when delivering the message, and recording the message content in the storage.

[0010] Furthermore, the step S400 includes the following: when the new application system is registered with the unified user authentication center, the system calls the API interface to synchronize the full data. After the full data is updated, all user data will be sent to the new application system to change the data.

[0011] Furthermore, step S500 includes: when the full data update is in progress or completed, the application system can still receive real-time user data change messages through the subscription mechanism; then the application system parses and processes the received change messages without waiting for the next full update cycle.

[0012] Furthermore, the step S600 includes: the application sets the data message to a high level, the message queue processes the data with priority, and distributes the data to the application system that has subscribed to the topic.

[0013] Furthermore, the step S700 specifically includes: setting an alarm and log recording mechanism, and regularly monitoring the status of the message queue and message processing; expanding the capacity of the message queue and optimizing the transmission mechanism according to system usage and demand changes; and providing seamless access for new application systems.

[0014] Furthermore, in step S500, after receiving the subscribed topic message data, the application system first decrypts the message, then performs corresponding data synchronization processing, and sends a confirmation signal to the message queue, indicating that the message has been successfully synchronized; if there is no response within the specified time, it is deemed that the message has not been successfully synchronized, and the message queue in the unified user authentication center application system will retry to send the message.

[0015] Furthermore, the message persistence operation is performed by the application sending data to the subscribed message topic, the data is the data stored in the message queue, and then the message queue persists the data on the disk.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0017] The present invention synchronizes user data by setting a message queue, which can make data updating and processing more timely, and adopts a transaction processing method to ensure the consistency of message data and avoid the situation of inconsistent user data between application systems.

[0018] The present invention sets priorities for messages so that a message queue can preferentially process data with high priorities, thereby ensuring the real-time performance of key data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of the method of the present invention;

[0020] Figure 2 It is a system architecture diagram of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be described in detail below in conjunction with the accompanying drawings.

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0023] In the existing traditional user data synchronization method, there are problems such as untimely user data update and high load of user center application system. In order to improve the efficiency, consistency and security of data synchronization, a user data synchronization method based on message queue is provided.

[0024] like Figure 1 As shown, a user data synchronization method based on a message queue comprises the following steps:

[0025] S100: Unifying the module initialization of the user authentication center, including the initialization of the user authentication module and the initialization of the developer center module;

[0026] Initialization of the user authentication module: Users register and log in through the unified user authentication center. User registration is performed according to the protocol format specified by the developer center. After registration is completed, the unified user authentication center calls the system-level API interface to perform full update initialization operations on relevant user data, and then the user can log in to the unified user authentication center;

[0027] Initialization of the developer center module: Set up a set of standard protocols and specifications to describe the requirements for application system access. Relevant personnel review user data. If there are no problems, the application's unique identifier appld and key appSecret are issued, and appld and appSecret are provided to the application system.

[0028] S200: Configuration and startup of the message queue module, setting the basic structure of the message queue, including queue topic, message priority and message persistence, and configuring the message encryption mechanism; the message sender performs encryption and idempotence check before sending the message, sends the message to the message queue, and records the sending log;

[0029] Among them, a topic is created in the message queue, and the unified user authentication center publishes the topic information to the message queue; when the message is transmitted, the application performs persistence operations and sets a priority for the data message so that the message queue processes the data first, and then distributes the data to the application system that has subscribed to the topic message.

[0030] S300: message delivery and handshake confirmation;

[0031] During the message transmission and reception process, a three-way handshake confirmation operation is performed on the message. If the handshake fails, the message is sent repeatedly. At the same time, a transaction is added to the message. If the message delivery fails, the system rolls back the transaction and tries to deliver the information again. During the message delivery, persistence operations are performed and the message content is recorded in the storage.

[0032] S400: calling the user data synchronization API and performing a full update, and performing a full update by calling the system-level API;

[0033] When a new application system is registered with the unified user authentication center, the system calls the API interface to synchronize the full data. After the full data is updated, all user data will be sent to the new application system to change the data. The user data synchronization API interface is used to initialize user data for the application system that is newly connected to the message queue.

[0034] S500: The application system receives and processes data changes in real time;

[0035] When a full data update is in progress or completed, the application system can still receive real-time user data change messages through the subscription mechanism; the application system then parses and processes the received change messages without waiting for the next full update cycle to ensure that user data is always up-to-date.

[0036] S600: Application and scheduling of message priorities, setting priorities for different types of messages according to different business requirements;

[0037] The application sets the data message to a high level, and the message queue processes the data with priority and distributes the data to the application systems that have subscribed to the topic.

[0038] S700: Monitor and maintain system operation status.

[0039] Set up alarm and logging mechanisms, and regularly monitor the status of message queues and message processing; expand the capacity of message queues and optimize transmission mechanisms based on system usage and demand changes; and provide seamless access for new application systems to continuously improve the scalability and stability of the system.

[0040] The method is provided with two data transmission modes, namely, synchronous data transmission through an API interface and through a message queue. In terms of data transmission, a message queue generally limits the size of message data, but the data transmission volume of an API interface is larger than that of a message queue, so an API interface is used to supplement a message queue.

[0041] The application system registers the application through the unified user authentication center application system. The unified user authentication center application system calls the open API interface to transmit data. The API obtains the full amount of data and publishes the topic message to the message queue. The two methods are carried out simultaneously to ensure that the data is updated, thereby further ensuring the consistency of user data between application systems. The application system registers the unified user authentication center application system through the developer center, and calls the system-level API interface to fully update the relevant user data during registration; the unified user authentication center application system publishes the user data as a data message, and the application system that subscribes to the topic can obtain the latest user information data in real time, thereby avoiding the problem of untimely data updates.

[0042] like Figure 2 As shown, a user data synchronization system based on a message queue includes a unified user authentication center application system, a message queue module and an application system.

[0043] The user authentication module in the unified user authentication center application system provides user authentication functions for the entire application system, is the management center for all user sessions, and provides a unified login page. The developer center module provides a set of unified protocol formats, which stipulates the data exchange format, application information, description format, and call credential format for inter-system calls of applications connected to the unified user authentication center application system.

[0044] The message queue module provides corresponding processing for message processing flow. As a bridge between the unified user authentication center application system and the application system, the message queue module is responsible for transmitting data.

[0045] There are multiple application systems. The existing application systems and the subsequent system expansion access application systems can subscribe to a data message topic in the form of subscription and process the received data messages without considering the specific source of the data messages, thereby reducing the coupling between systems and increasing the scalability of the system.

[0046] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A user data synchronization method based on a message queue, characterized in that: The method comprises the following steps: S100: Unifying the module initialization of the user authentication center, including the initialization of the user authentication module and the initialization of the developer center module; S200: Configuration and startup of the message queue module, setting the basic structure of the message queue, including queue topic, message priority and message persistence, and configuring the message encryption mechanism; the message sender performs encryption and idempotence check before sending the message, sends the message to the message queue, and records the sending log; S300: message delivery and handshake confirmation; S400: calling the user data synchronization API and performing a full update, and performing a full update by calling the system-level API; S500: The application system receives and processes data changes in real time; S600: Application and scheduling of message priorities, setting priorities for different types of messages according to different business requirements; S700: Monitor and maintain system operation status.

2. A user data synchronization method based on a message queue according to claim 1, characterized in that: The step S100 includes: Initialization of the user authentication module: Users register and log in through the unified user authentication center. User registration is performed according to the protocol format specified by the developer center. After registration is completed, the unified user authentication center calls the system-level API interface to perform full update initialization operations on relevant user data, and then the user can log in to the unified user authentication center; Initialization of the developer center module: Set up a set of standard protocols and specifications to describe the requirements for application system access. Relevant personnel review user data. If there are no problems, the application's unique identifier appld and key appSecret are issued, and appld and appSecret are provided to the application system.

3. A user data synchronization method based on message queue according to claim 1, characterized in that: In the step S200, when the message is transmitted, the application performs a persistence operation and sets a priority for the data message so that the message queue processes the data first, and then distributes the data to the application system that has subscribed to the topic message.

4. The user data synchronization method based on message queue according to claim 1, characterized in that: In step S300, the message queue transmits and processes the message as follows: During the message transmission and reception process, a three-way handshake confirmation operation is performed. If the handshake fails, the message is sent repeatedly. At the same time, a transaction is added to the message. If the message delivery fails, the system rolls back the transaction and tries to deliver the message again. When the message is delivered, persistence operation is performed at the same time, and the message content is recorded in the storage.

5. The user data synchronization method based on message queue according to claim 1, characterized in that: The step S400 includes the following: When the new application system is registered with the unified user authentication center, the system calls the API interface to synchronize the full data. After the full data is updated, all user data will be sent to the new application system to change the data.

6. A user data synchronization method based on message queue according to claim 1, characterized in that: The step S500 includes: when the full data update is in progress or after completion, the application system can still receive real-time user data change messages through the subscription mechanism; then the application system parses and processes the received change messages without waiting for the next full update cycle.

7. The user data synchronization method based on message queue according to claim 1, characterized in that: The step S600 includes: the application sets the data message to a high level, the message queue processes the data with priority, and distributes the data to the application system that has subscribed to the topic.

8. The user data synchronization method based on message queue according to claim 1, characterized in that: The step S700 specifically includes: setting up alarm and log recording mechanisms, and regularly monitoring the status of message queues and message processing; expanding the capacity of message queues and optimizing transmission mechanisms according to system usage and demand changes; and providing seamless access for newly added application systems.

9. The user data synchronization method based on message queue according to claim 1, characterized in that: In step S500, after receiving the subscribed topic message data, the application system first decrypts the message, then performs corresponding data synchronization processing, and sends a confirmation signal to the message queue, indicating that the message has been successfully synchronized; if there is no response within the specified time, it is deemed that the message has not been successfully synchronized, and the message queue in the unified user authentication center application system will retry to send the message.

10. The user data synchronization method based on message queue according to claim 1, characterized in that: The message persistence operation is performed by the application sending data to the subscribed message topic, and the data is the data stored in the message queue, and then the message queue persists the data on the disk.

Images