A data distribution method and system based on reliable message

Abstract

The present invention discloses a data distribution method and system based on reliable messages, which includes: performing configuration processing of an API service interface, determining the basic attributes and system business attributes of each service interface; performing distribution configuration of subscription data; viewing queue data and logs of data distribution, and performing maintenance management; viewing and maintenance management of interactive message data; performing system access processing, performing system identification code and key distribution, interface authorization and maintenance management; performing queue distribution and responsible for data synchronization, and processing and converting business data according to the processing of the subscription data configuration module to generate subscription data; providing distributed business data to the outside; and receiving, pre-storing, sending, resending and confirming messages. The embodiments of the present invention can ensure the stability, efficiency and security of data synchronization and can be widely used in the field of computer technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data distribution method and system based on reliable messages. Background Art

[0002] There are numerous business systems within an enterprise. As the business continues to develop, the demand for data interaction between business systems is increasing. The coupling between systems is getting higher and higher. The efficiency of calling the data synchronization interface between systems is relatively low, which will lead to a decline in system performance. At the same time, the data consistency between systems cannot be guaranteed, affecting the normal operation of the system business. Summary of the Invention

[0003] The main purpose of the embodiments of the present invention is to provide a data distribution method and system based on reliable messages, which can ensure the stability, efficiency and security of data synchronization.

[0004] To achieve the above objectives, an embodiment of the present invention provides a data distribution method based on reliable messaging, comprising the following steps:

[0005] Configure the API service interface through the interface management module to determine the basic attributes and system business attributes of each service interface;

[0006] Configure the distribution of subscription data through the subscription data configuration module;

[0007] View the queue data and logs of data distribution and perform maintenance management through the subscription queue management module;

[0008] View, maintain and manage interactive message data through the message management module;

[0009] The application management module is used to process system access, distribute system identification codes and keys, and perform interface authorization and maintenance management;

[0010] The data service module distributes queues and synchronizes data. It also processes and converts business data based on the processing of the subscription data configuration module to generate subscription data. The processing and conversion includes matching configuration items, filtering data attributes, and filtering subscription condition rules.

[0011] Provide distributed business data externally through the API service module;

[0012] Messages are received, pre-stored, sent, resent and confirmed through the message service module.

[0013] In some embodiments, the configuration processing of the API service interface by the interface management module to determine the basic attributes and system business attributes of each service interface includes the following steps:

[0014] Configure the basic attributes of each service interface through the API service interface, including interface code, interface name, interface address, interface type, business module, business system, interface identifier, interface protocol, request type, and usage scenario;

[0015] Configure the system business attributes and whitelist of each business system, where the system business attributes include user account, user password, number of failed retries, retry frequency, scheduled acquisition time, warning time limit, and warning interval; the whitelist includes IP address and remarks;

[0016] Configure the basic information of the synchronization data source system and the basic information of the data synchronization module.

[0017] In some embodiments, the configuration of subscription data distribution by the subscription data configuration module includes the following steps:

[0018] Configure subscription data conditions and custom attribute fields through the subscription data configuration module; wherein the custom attribute fields include business entity name, attribute code, and attribute name;

[0019] The message producer sends a data synchronization processing message to the data distribution system. Specifically, the message producer adds, deletes, and modifies data. After the data is approved at the last node of the process, a message distribution data record table is generated, and the data synchronization processing message is sent asynchronously to the data distribution system.

[0020] The data distribution system receives the message sent by the message producer and checks whether the message number already exists. If so, it notifies the message producer that the message number already exists. If the message number does not exist, it sends the message to the message producer to send the message to the message queue. The message producer and the data distribution system asynchronously perform message processing.

[0021] Among them, the content of the data record table includes system code, message number, business module, process primary key, and business data primary key. One message number is associated with multiple business data. The data synchronization processing message includes message number, caller system code, message content, message type, and sending time.

[0022] In some embodiments, the method further includes: sending a consumption message and a pre-stored message of the data distribution system and generating a data distribution queue, specifically including the following steps:

[0023] Send consumption messages to the data distribution system and pre-store the consumption messages in the database message table;

[0024] When the database message table is written successfully, a data distribution queue event is generated based on the spring event mechanism;

[0025] After the subscription generates the data distribution queue event, query whether the current message number has distribution data. If not, do not do anything. If so, obtain the business module code data of the distribution data, and query the subscription data configuration data through the business module code.

[0026] If no subscription data configuration data is found, no task processing will be performed; if so, the subscription queue will be queried based on the message number to see if there is data already present.

[0027] If data exists in the subscription queue, no processing is performed; if data exists, subscription queue table data is generated, and the subscription queue table data includes one or more subtask queues, subscription queue data table data, and subscription queue message table data;

[0028] After the data is successfully written to the database message table, the main task ID and sorting score are written to the cache ordered set container asynchronously and stored in the form of key-value;

[0029] The cache ordered collection container is used to divide the data into business modules and distribute them to the designated collection container.

[0030] In some embodiments, the data service module performs queue distribution and data synchronization, and processes and converts the business data according to the processing of the subscription data configuration module to generate subscription data, including the following steps:

[0031] Construct a main task thread pool for processing cached ordered set containers, responsible for processing the main task ordered set; obtain the value of the cached ordered set container, use a distributed lock to preempt the main task primary key, and through a direct database connection, be responsible for obtaining the business data primary key of the message producer's distribution data record table in batches. Then, based on the business data primary key, obtain and assemble data from the business main table and related tables, and store it in the subscription queue data table. After successfully generating the subscription data, delete the main task ID in the cached ordered set container, release the main task's primary key distributed lock, and asynchronously write the main task primary key, subtask quantity, subtask primary key, and sorting score into the cached ordered set container, storing them in the form of key-value pairs;

[0032] Construct a subtask thread pool for processing the cached ordered set container to process the subtask ordered set; obtain the value of the cached ordered set container, determine the subtask primary key, use a distributed lock to preempt the subtask primary key, and based on the business data of the subscription queue data table, after matching the configuration items of the subscription data configuration, filtering the data attributes and filtering the subscription condition rules, store the data in the subscription queue data table. After successfully generating the subscription data, delete the subtask primary key from the cached ordered set container and release the distributed lock of the subtask primary key; publish events based on the spring event mechanism and reduce the number of subtasks by 1. If the result is 0, it means that all subqueues of the queue have completed data distribution, and delete the main task primary key of the cached ordered set container;

[0033] Locate problems based on the execution logs generated during business data processing;

[0034] Obtain detailed business data and save the synchronized data to the distribution system database. The synchronization system obtains the synchronized data by directly connecting to the distribution system database.

[0035] Obtain subscription data configuration. According to the configured business module and business system, obtain configuration information in the subscription data configuration. The configuration information includes interface code, interface name, business system, subscription conditions, and attribute list. The configuration information is used to determine the generation of subscription data.

[0036] Generate subscription data, the business module configured for subscription data, the business system, subscription data conditions and an attribute list according to the subscription data configuration; wherein the attribute list includes the business body name, attribute code and attribute name; when the business system receives subscription data, synchronize part of the data according to the subscription data conditions; configure the attribute fields of the synchronized data according to the attribute list; control row permissions and column permissions according to the subscription data conditions and attribute fields, and obtain the corresponding subscription data according to the corresponding permission screening.

[0037] In some embodiments, the method further comprises:

[0038] After subscribing to an event, check the legitimacy of the event, and after the time legitimacy check passes, confirm that the producer sending the message will send the pre-stored message to the message queue;

[0039] After consuming the message, the consumer who confirms that the message has been sent checks whether the maximum number of retries has been exceeded. If the maximum number of retries has been exceeded, the producer of the dead letter message sends the message to the message queue, the consumer of the dead letter message continues to consume the message, and the message is changed to dead. If the maximum number of retries has not been exceeded, the producer checks whether the message is sent successfully. If the sending fails, the producer who resends the message sends the message to the message queue, the consumer of the resending message continues to consume the message, and publishes an event based on the spring event mechanism. The producer then returns to execute the step of sending the pre-stored message to the message queue. If the sending is successful, the message is changed to sent.

[0040] In some embodiments, the method further comprises the following steps:

[0041] Receive data synchronization messages from the distribution system through the newly added interface of the target system, and save the received synchronization messages to the local database for subsequent synchronization data processing;

[0042] According to the configured API service interface, the corresponding synchronization data is obtained from the data distribution platform through the interface to complete business processing;

[0043] The data distribution system provides a confirmation message interface, through which the target system sends a confirmation message to confirm that data synchronization has been completed. If the confirmation fails, the target system needs to resend the confirmation message.

[0044] The data distribution system receives the confirmation message sent by the target system, checks the legitimacy of the message, and then confirms that the message producer sends the message to the message queue;

[0045] Confirm the consumption message of the message consumer and confirm that the message is in the consumed state.

[0046] Another aspect of the present invention provides a reliable message-based data distribution system, including:

[0047] The interface management module is used to configure the API service interface and determine the basic attributes and system business attributes of each service interface;

[0048] Subscription data configuration module, used to configure the distribution of subscription data;

[0049] Subscription queue management module, used to view queue data and logs of data distribution, as well as perform maintenance management;

[0050] Message management module, used for viewing, maintaining and managing interactive message data;

[0051] Application management module, used for system access processing, system identification code and key distribution, interface authorization and maintenance management;

[0052] The data service module is responsible for queue distribution and data synchronization. It also processes and converts business data according to the processing of the subscription data configuration module to generate subscription data. The processing and conversion includes matching configuration items, filtering data attributes, and filtering subscription condition rules.

[0053] API service module, used to provide distributed business data to the outside world;

[0054] The message service module is used to receive, pre-store, send, resend and confirm messages.

[0055] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0056] The memory is used to store programs;

[0057] The processor executes the program to implement the method described above.

[0058] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0059] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0060] The embodiments of the present invention include at least the following beneficial effects: The present invention provides a data distribution method and system based on reliable messages. The scheme configures and processes the API service interface through the interface management module to determine the basic attributes and system business attributes of each service interface; configures the distribution of subscription data through the subscription data configuration module; views the queue data and logs of data distribution and performs maintenance management through the subscription queue management module; views and maintains interactive message data through the message management module; processes system access, allocates system identification codes and keys, and performs interface authorization and maintenance management through the application management module; distributes queues and synchronizes data through the data service module, and processes and converts business data according to the processing of the subscription data configuration module to generate subscription data; the processing and conversion includes matching configuration items, filtering data attributes, and filtering subscription condition rules; provides distributed business data to the outside through the API service module; and receives, pre-stores, sends, resends, and confirms messages through the message service module. The embodiments of the present invention can ensure the stability, efficiency, and security of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a flowchart of the overall steps provided by an embodiment of the present invention;

[0062] Figure 2 This is an architecture diagram of a reliable message-based data distribution system provided by an embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0065] It will be understood that the terms "first," "second," and the like used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are merely used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination."

[0066] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include at least one, two or more, multiple, two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0068] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0069] There are four common implementation solutions:

[0070] 1. The main business system exports data as files and places them in shared storage. Each business system reads the file data from the shared storage and then updates it to the business system.

[0071] 2. The main business system actively pushes data to each business system. When the data changes (added, modified, deleted, etc.), the changed data is actively pushed to each business system in a timely manner by calling the interface.

[0072] 3. Each business system periodically polls and calls the interface of the main business system to pull data, and then updates it to the business system.

[0073] 4. The main business system publishes data to MQ (Message Queue). Each business system obtains data from MQ and then updates it to the business system.

[0074] The existing data distribution processing on the market can be implemented in the following steps:

[0075] 1. Synchronous data generation: The source system performs operations such as adding, modifying, and deleting. After the operation is completed, the distribution system is notified to synchronize data processing;

[0076] 2. Distribution system processing: After receiving the synchronization processing notification from the source system, the system goes to the source system to obtain the corresponding synchronization data, saves the synchronization data to the distribution system storage (memory or hard disk), and notifies the target system to synchronize data as needed;

[0077] 3. Target system data synchronization: After the distribution system obtains the new synchronization data, one or more target systems synchronize data regularly or according to the notification of the distribution system, synchronizing the newly added data in the distribution system storage to the target system to complete the data synchronization process.

[0078] The requirements and challenges faced by data distribution and synchronization are:

[0079] 1. Real-time requirements: Some data need to be synchronized in real time to ensure the timeliness of system data.

[0080] 2. Large data volume: It is necessary to process the synchronization and update of large amounts of data to improve synchronization efficiency and performance.

[0081] 3. Data consistency: Ensure the consistency and integrity of data between different systems.

[0082] The existing implementation solutions on the market have the following shortcomings:

[0083] 1. Data distribution processing does not take into account the data delivery confirmation mechanism;

[0084] When data is distributed to multiple target systems for data synchronization, some data may be lost when the multiple target systems receive data, and the data consistency and integrity between the target system and the source system cannot be guaranteed.

[0085] 2. Data distribution does not consider the retry mechanism;

[0086] In case of network or service interface anomalies, data distribution may fail partially or completely. A comprehensive retry mechanism and retry count should be considered for failed data, while also supporting manual retry processing.

[0087] 3. Custom configuration is not implemented;

[0088] When synchronizing data across multiple target systems, the basic configuration and synchronization fields differ. The synchronization processing configuration and logic are hard-coded into the synchronization interface code. As the business evolves, the interface code needs to be modified to accommodate business adjustments.

[0089] 4. Synchronous logging is not implemented;

[0090] When multiple data are synchronized to multiple target systems, the success or failure of the processing needs to be recorded in the synchronization log, which can be easily queried and retried.

[0091] 5. Batch processing is not considered;

[0092] When the data volume is large, synchronization of data such as addition, deletion, and modification is frequent. Excessive data synchronization affects the external services of the business system. It is necessary to consider batch data synchronization.

[0093] 6. Need to support synchronous field addition and deletion;

[0094] When fields in the source system database are added or deleted, synchronized data fields are added or deleted, and the data synchronization code of each target system needs to be modified simultaneously, increasing the difficulty of system maintenance. Synchronized fields need to be configured to facilitate addition and deletion;

[0095] 7. Support permissions are required;

[0096] The source system and the target system support data permissions. During the data synchronization process, data permissions need to be synchronized to ensure data security.

[0097] Existing data distribution systems on the market handle basic data synchronization between different systems. However, existing implementations are insufficient for multi-system synchronization and scenarios with high data synchronization requirements. This invention addresses these shortcomings and offers design and optimization to ensure the stability, efficiency, and security of data synchronization.

[0098] The data distribution method and system based on reliable messages provided by the embodiments of the present invention relate to the field of computer technology. The data distribution method based on reliable messages provided by the embodiments of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a data distribution method based on reliable messages, etc., but is not limited to the above forms.

[0099] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0100] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0101] In addition, the server can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0102] In some implementation environments, the present invention provides a data distribution method based on reliable messages, referring to Figure 1 , Figure 1 This is a flowchart of a reliable message-based data distribution method for a server provided in an embodiment of the present invention. The execution subject of this method can be any of the aforementioned computer devices (including servers or terminals). Figure 1 , the method may include the following steps:

[0103] Configure the API service interface through the interface management module to determine the basic attributes and system business attributes of each service interface;

[0104] Configure the distribution of subscription data through the subscription data configuration module;

[0105] View the queue data and logs of data distribution and perform maintenance management through the subscription queue management module;

[0106] View, maintain and manage interactive message data through the message management module;

[0107] The application management module is used to process system access, distribute system identification codes and keys, and perform interface authorization and maintenance management;

[0108] The data service module distributes queues and synchronizes data. It also processes and converts business data based on the processing of the subscription data configuration module to generate subscription data. The processing and conversion includes matching configuration items, filtering data attributes, and filtering subscription condition rules.

[0109] Provide distributed business data externally through the API service module;

[0110] Messages are received, pre-stored, sent, resent and confirmed through the message service module.

[0111] The specific implementation process of the present invention is described in detail below with reference to the accompanying drawings:

[0112] The embodiment of the present invention provides a data distribution system based on reliable messaging, which specifically includes the following modules and services:

[0113] (1) Interface management: used to configure the service interface (API application programming interface), including interface code, interface name, interface type, business module, business system, interface identifier, interface protocol, request type, interface address, etc.

[0114] (2) Subscription data configuration: used to configure the distribution configuration of subscription data, including interface code, interface name, business system, subscription conditions, data attributes, etc.

[0115] (3) Subscription queue management: queue data, log viewing and maintenance management for data distribution;

[0116] (4) Message management: used for viewing and maintaining interactive message data;

[0117] (5) Application management: used for system access allocation system identification code and key, interface authorization and maintenance management;

[0118] (6) Data service: used for queue distribution and data synchronization. It processes and converts business data according to subscription data configuration, including matching configuration items, filtering data attributes, and filtering subscription condition rules to generate subscription data.

[0119] (7) API service: used to provide and distribute business data to external parties;

[0120] (8). Message service: responsible for receiving, pre-storing, sending, resending and confirming messages.

[0121] refer to Figure 2 The method of the reliable message data distribution system provided by the embodiment of the present invention includes the following configuration and operation steps:

[0122] (1). Step 1, for configuring the API service interface (API application programming interface);

[0123] In interface management, you can configure the basic properties of each service interface, including interface code, interface name, interface address, interface type, business module, business system, interface identifier, interface protocol, request type, and usage scenario. You can then further configure the system business properties and whitelist for each business system. System business properties include user account, user password, number of failed retries, retry frequency, scheduled acquisition time, warning timeout, and warning interval. Whitelists include IP addresses and notes. Interface management is used to configure basic information for the synchronization data source system and basic information for the data synchronization module.

[0124] (2) Step 2, used to configure the distribution configuration of subscription data;

[0125] In subscription data configuration, you can configure subscription data conditions and custom attribute fields by selecting different interfaces and business systems. The business module is automatically displayed after selecting the interface. The attribute list includes the business entity name, attribute code, and attribute name. The attribute list retrieves the corresponding business attributes based on the business module. Subscription data information configuration is added to the basic data configuration and allows you to customize the synchronization attributes and fields for different systems.

[0126] The configuration data in steps 1 and 2 are used for subsequent processing.

[0127] (3) Step 3: The message producer sends a data synchronization processing message to the data distribution system;

[0128] After the message producer (source system) performs operations such as adding, deleting, and modifying data and passes the approval at the last node of the process, a message distribution data record table is generated. The record table content includes the system code, message number (must be unique), business module, process primary key (ID), business data primary key (ID), etc. One message number can be associated with multiple business data. The data is sent asynchronously and synchronously to the data distribution system for processing. The message includes the message number (unique), caller system code, message content, message type, sending time, etc. The message distribution data record table is required for the synchronization data.

[0129] (4). Step 4, the data distribution system receives the message sent by the message producer;

[0130] The data distribution system receives messages from message producers and checks whether the message number already exists. If so, it notifies the message producer that the message number already exists. If not, the message is sent to the message producer and sent to the message queue. Using a message queue decouples business code and improves system processing capabilities. The message producer and data distribution system perform message processing asynchronously.

[0131] (5) Step 5: Send message to the message consumer (data distribution system) to consume the message, pre-store the message, and generate a data distribution queue;

[0132] The message consumer (data distribution system) consumes the message and pre-stores the message in the database message table. After the write is successful, the data distribution queue event is generated based on the spring event mechanism. After the data distribution queue event is subscribed, the message number is queried to see if there is data distribution data. If not, no processing is performed. Otherwise, the business module code data of the distribution data is obtained and the subscription data configuration data is queried through the business module code. If not, no task processing is performed. Otherwise, the subscription queue is queried based on the message number to see if there is data. If so, no processing is performed. Otherwise, subscription queue table data (including one or more subtask queues), subscription queue data table data, and subscription queue message table data are generated. After the data is successfully written to the database, the main task ID and sorting score are asynchronously written to the cache (redis) ordered set container for persistence and stored in the cache (redis) in the form of key-value pairs. The key is named sync:executor:module synchronization service:main_tasks, such as material synchronization service and color synchronization service. The purpose of using the cache (redis) ordered set container is to divide the data by business module and distribute it to the specified collection container, so that the use of subscription queues can support horizontal expansion and improve processing efficiency.

[0133] (6) Step 6: Business processing, the data distribution service processes the cache (redis) ordered set container data, obtains business detailed data through direct database connection, and then processes and converts the data according to the subscription data configuration, and finally generates subscription data;

[0134] 6.1. Build a main task thread pool for processing the cache (redis) ordered set container. This thread pool is responsible for processing the main task ordered set and obtaining the main task primary key (ID) of the cache (redis) ordered set value. To prevent repeated execution, a distributed lock is used to preempt the main task primary key (ID). Through a direct database connection, the thread pool is responsible for obtaining the business data primary key of the distribution data record table of the message producer (source system) in batches. Then, based on the business data primary key, the thread pool obtains and assembles data from the business main table and associated tables and stores it in the subscription queue data table. After successfully generating the subscription data, the thread pool deletes the main task ID from the cache (redis) ordered set, releasing the distributed lock for the main task primary key (ID). Furthermore, the thread pool asynchronously writes the main task primary key (ID), the number of subtasks, the subtask primary key (ID), and the sorting score to the cache (redis) ordered set container. These are stored in the cache (redis) as key-value pairs, where the keys are named sync:executor:module synchronization service:tasks and sync:executor:task_progress, respectively.

[0135] 6.2. Build a subtask thread pool for processing cache (redis) ordered set containers, which is responsible for processing subtask ordered sets and obtaining the subtask primary key (ID) of the cache (redis) ordered set value. In order to prevent repeated execution, a distributed lock is used to preempt the subtask primary key (ID). Based on the business data of the subscription queue data table, after matching the configuration items of the subscription data configuration, filtering the data attributes and filtering the subscription condition rules, the data is stored in the subscription queue data table. After successfully generating the subscription data, the subtask primary key (ID) is deleted from the cache (redis) ordered set, the distributed lock of the subtask primary key (ID) is released, the event is published based on the spring event mechanism, and the number of subtasks is reduced by 1. If the result is 0, it means that all subqueues of the queue have completed data distribution, and the main task primary key (ID) of the redis ordered set is deleted.

[0136] 6.3. In steps 6.1 and 6.2, corresponding execution logs will be generated during the processing, which will facilitate problem location, ensure data uniqueness, and avoid duplication or omission of processing.

[0137] 6.4. Obtain detailed business data. The synchronized data has been synchronized and saved in the distribution system database (processed in step 5). The synchronization system obtains the synchronized data by directly connecting to the distribution system database;

[0138] 6.5. Obtain subscription data configuration. According to the configured business module and business system, obtain configuration information in the subscription data configuration, including interface code, interface name, business system, subscription conditions, attribute list, etc., for the next step of subscription data generation judgment;

[0139] 6.6. Generate subscription data based on subscription data configuration, including business modules, business systems, subscription data conditions and attribute lists (attribute lists include business entity names, attribute codes, and attribute names). The business system is the target system, and the business module is the synchronized data source module (materials, patterns, or graphic documents, etc.). Subscription conditions are used to filter data. When the target system receives data from the business system, it can synchronize only part of the data based on the conditions, rather than all of it. At the same time, according to the attribute list, the synchronization data fields can be configured to facilitate subsequent expansion. The generated subscription data can control row permissions and column permissions based on the filtering conditions and attribute fields, and the corresponding subscription data can be obtained based on the corresponding permission filtering.

[0140] (7). Step 7, confirm that the message producer sends the pre-stored message to the message queue;

[0141] After subscribing to an event, the event's legitimacy is checked and confirmed, and the message producer sends the pre-stored message to the message queue.

[0142] (8) Step 8: Confirm that the message consumer consumes the message and sends the message to the target system;

[0143] Confirm that the message consumer consumes the message and check whether the maximum number of retries is exceeded. If the maximum number of retries is exceeded, the dead letter message producer sends the message to the message queue, the dead letter message consumer consumes the message, and the message is changed to dead. Otherwise, check whether the sending is successful. If the sending fails, the resend message producer sends the message to the message queue, the resend message consumer consumes the message, changes the number, time, etc., and publishes events based on the spring event mechanism to go to step 7 to continue execution. Otherwise, change the message to sent.

[0144] (9). Step 9, the target system receives the message and stores it;

[0145] The target system adds a new interface to process and receive data synchronization messages from the distribution system, and save the received synchronization messages to the local database for subsequent synchronization data processing.

[0146] (10). Step 10, the target system completes business processing;

[0147] According to the API service interface configured in step 1, the target system will obtain the corresponding synchronization data from the data distribution platform through the interface to complete business processing.

[0148] (11). Step 11, the target system sends a confirmation message;

[0149] The data distribution system provides a confirmation message interface, through which the target system sends a confirmation message to confirm that data synchronization has been completed. If the confirmation fails, the target system needs to resend the confirmation message.

[0150] (12). Step 12, the data distribution system receives a confirmation message sent by the target system;

[0151] The data distribution system receives the confirmation message sent by the target system, checks the legitimacy of the message, and then confirms that the message producer sends the message to the message queue.

[0152] (13). Step 13, the data distribution system confirms the message;

[0153] Confirm that the message consumer consumes the message and confirms that the message has been consumed.

[0154] In summary, the present invention has the following characteristics:

[0155] 1. Independent microservice deployment;

[0156] Adopting independent microservices (Spring Cloud Alibaba) to implement data distribution functions without affecting the original business system functions. It is also compatible with the existing microservice architecture and can be expanded or reduced as the business develops.

[0157] 2. Custom configuration;

[0158] Add interface management and subscription data management, and customize the system configuration to synchronize basic information and fields;

[0159] 3. Synchronous log records;

[0160] Add synchronization queue management and subscription queue management, record data distribution information and logs;

[0161] 4. Retry configuration;

[0162] Added configurations such as retry times and whitelists for different systems to facilitate retry processing for different systems;

[0163] 5. Batch processing;

[0164] Data distribution and processing uses batch data processing to avoid frequent processing affecting the original system's external services;

[0165] 6. Synchronous or asynchronous processing;

[0166] Data distribution calls scheduled tasks for processing, meeting two different situations: manual triggering and scheduled triggering, and realizing two requirements of synchronous or asynchronous processing;

[0167] 7. Support synchronous field addition and deletion;

[0168] Supports adding and deleting distribution data fields. Synchronization fields can be configured in subscription data management without modifying the synchronization code, facilitating system maintenance.

[0169] 8. Support multiple synchronization methods;

[0170] Supports database intermediate table or interface data distribution, and can be configured and processed according to actual needs.

[0171] 9. Support data permissions;

[0172] Data distribution is carried out according to the data permission requirements of the source system and the target system to ensure the data security and consistency between different systems.

[0173] Compared with the prior art, the present invention has the following advantages:

[0174] 1. Through interface management configuration, provide API service interface to allow various business systems to access and obtain data.

[0175] 2. Through subscription data configuration, different business data attributes can be configured according to different interfaces and different business systems to improve business adaptation to different data scenarios. At the same time, when business data changes (addition, modification, deletion, etc.), a change message can be sent to the target system in a timely manner. When the target system receives the message notification, it will automatically call the corresponding interface to obtain data to meet the timeliness requirements of the data, while avoiding periodic polling interfaces and reducing the consumption of server resources.

[0176] In addition, the messaging function of the reliable message data distribution system in the embodiment of the present invention can adopt a messaging middleware (RabbitMQ, RocketMQ or Kafka) to increase the data analysis and processing tool (ClickHouse). These messaging middleware can provide a certain degree of reliability and stability, rich functions and flexibility, and have a huge user community and a rich ecosystem to help developers better understand and use it.

[0177] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] An embodiment of the present invention further provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-described reliable message-based data distribution method when executing the computer program. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0179] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] See also Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0181] The processor 301 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0182] The memory 302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 302 and is called by the processor 301 to execute the reliable message-based data distribution method of the embodiment of the present invention.

[0183] Input / output interface 303, used to implement information input and output;

[0184] Communication interface 304, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.);

[0185] bus 305 , which transmits information between the various components of the device (e.g., processor 301 , memory 302 , input / output interface 303 , and communication interface 304 );

[0186] The processor 301 , the memory 302 , the input / output interface 303 and the communication interface 304 are connected to each other in communication within the device via the bus 305 .

[0187] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the computer program implements the above-mentioned reliable message-based data distribution method.

[0188] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0189] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to the user's identity or characteristics, such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present invention will be obtained.

[0191] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0192] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present invention, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0194] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0195] The terms "first," "second," "third," "fourth," and the like (if any) in the description of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or apparatus.

[0196] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0197] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and other media that can store programs.

[0201] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the invention is not limited thereby. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the invention should be within the scope of the invention.

Claims

1. A data distribution method based on reliable messaging, characterized in that: The following steps are involved: Configure the API service interface through the interface management module to determine the basic attributes and system business attributes of each service interface; Configure the distribution of subscription data through the subscription data configuration module; View the queue data and logs of data distribution and perform maintenance management through the subscription queue management module; View, maintain and manage interactive message data through the message management module; The application management module is used to process system access, distribute system identification codes and keys, and perform interface authorization and maintenance management; The data service module distributes queues and synchronizes data, and processes and converts business data according to the processing of the subscription data configuration module to generate subscription data; Provide distributed business data externally through the API service module; Receive, pre-store, send, resend and confirm messages through the message service module; The steps of distributing the queue and synchronizing the data through the data service module, processing and converting the business data according to the processing of the subscription data configuration module, and generating subscription data include: Build a main task thread pool to process the main task ordered set; obtain the value of the cached ordered set container, use a distributed lock to preempt the main task primary key, and obtain the business data primary key of the message producer's distribution data record table in batches through a direct database connection. Then, based on the business data primary key, obtain and assemble data from the business main table and related tables, and store it in the subscription queue data table. In addition, asynchronously write the main task primary key, subtask quantity, subtask primary key, and sorting score into the cached ordered set container, storing them in the form of key-value pairs. Build a subtask thread pool to process the ordered set of subtasks; determine the subtask primary key from the cached ordered set container, use a distributed lock to preempt the subtask primary key, and store the data in the subscription queue data table after matching the subscription data configuration items, filtering the data attributes, and filtering the subscription condition rules based on the business data in the subscription queue data table; The cache ordered collection container is used to divide data into business modules and distribute them to designated collection containers; the subscription queue data table is data generated based on the data in the subscription queue.

2. A data distribution method based on reliable messaging according to claim 1, characterized in that: The configuration process of the API service interface is performed through the interface management module to determine the basic attributes and system business attributes of each service interface, including the following steps: Configure the basic attributes of each service interface through the API service interface, including interface code, interface name, interface address, interface type, business module, business system, interface identifier, interface protocol, request type, and usage scenario; Configure the system business attributes and whitelist of each business system, where the system business attributes include user account, user password, number of failed retries, retry frequency, scheduled acquisition time, warning time limit, and warning interval; the whitelist includes IP address and remarks; Configure the basic information of the synchronization data source system and the basic information of the data synchronization module.

3. The method for distributing reliable message data according to claim 1, wherein: The distribution configuration of subscription data by the subscription data configuration module includes the following steps: Configure subscription data conditions and custom attribute fields through the subscription data configuration module; wherein the custom attribute fields include business entity name, attribute code, and attribute name; The message producer sends a data synchronization processing message to the data distribution system. Specifically, the message producer adds, deletes, and modifies data. After the data is approved at the last node of the process, a message distribution data record table is generated, and the data synchronization processing message is sent asynchronously to the data distribution system. The data distribution system receives the message sent by the message producer and checks whether the message number already exists. If so, it notifies the message producer that the message number already exists. If the message number does not exist, it sends the message to the message producer to send the message to the message queue. The message producer and the data distribution system asynchronously perform message processing. Among them, the content of the data record table includes system code, message number, business module, process primary key, and business data primary key. One message number is associated with multiple business data. The data synchronization processing message includes message number, caller system code, message content, message type, and sending time.

4. A data distribution method based on reliable messaging according to claim 3, characterized in that: The method further includes: sending consumption messages and pre-stored messages of the data distribution system and generating a data distribution queue, specifically including the following steps: Send consumption messages to the data distribution system and pre-store the consumption messages in the database message table; When the database message table is written successfully, a data distribution queue event is generated based on the spring event mechanism; After the subscription generates the data distribution queue event, query whether the current message number has distribution data. If not, do not do anything. If so, obtain the business module code data of the distribution data, and query the subscription data configuration data through the business module code. If no subscription data configuration data is found, no task processing will be performed; if so, the subscription queue will be queried based on the message number to see if there is data already present. If data exists in the subscription queue, no processing is performed; if data exists, subscription queue table data is generated, and the subscription queue table data includes one or more subtask queues, subscription queue data table data, and subscription queue message table data; After the data is successfully written to the database message table, the main task ID and sorting score are written to the cache ordered set container asynchronously and stored in the form of key-value.

5. The data distribution method based on reliable messaging according to claim 4, characterized in that: After the steps of assembling data through the main task thread pool and storing it in the subscription queue data table, it also includes: After successfully generating the subscription data, delete the main task ID in the cache ordered set container and release the primary key distributed lock of the main task; and, After the steps of processing business data through the subtask thread pool and storing the data in the subscription queue data table, it also includes: After successfully generating subscription data, delete the subtask primary key from the cache ordered set container and release the distributed lock of the subtask primary key; publish events based on the spring event mechanism and reduce the number of subtasks by 1. If the result is 0, it means that all subqueues of the queue have completed data distribution, and delete the main task primary key from the cache ordered set container; and, The business data processing process also includes: Locate problems based on the execution logs generated during business data processing; Obtain detailed business data and save the synchronized data to the distribution system database. The synchronization system obtains the synchronized data by directly connecting to the distribution system database. Obtain subscription data configuration. According to the configured business module and business system, obtain configuration information in the subscription data configuration. The configuration information includes interface code, interface name, business system, subscription conditions, and attribute list. The configuration information is used to determine the generation of subscription data. Generate subscription data, the business module configured for subscription data, the business system, subscription data conditions and an attribute list according to the subscription data configuration; wherein the attribute list includes the business body name, attribute code and attribute name; when the business system receives subscription data, synchronize part of the data according to the subscription data conditions; configure the attribute fields of the synchronized data according to the attribute list; control row permissions and column permissions according to the subscription data conditions and attribute fields, and obtain the corresponding subscription data according to the corresponding permission screening.

6. A data distribution method based on reliable messaging according to claim 5, characterized in that: The method further comprises: After subscribing to an event, check the legitimacy of the event, and after the time legitimacy check passes, confirm that the producer sending the message will send the pre-stored message to the message queue; After consuming the message, the consumer who confirms that the message has been sent checks whether the maximum number of retries has been exceeded. If the maximum number of retries has been exceeded, the producer of the dead letter message sends the message to the message queue, the consumer of the dead letter message continues to consume the message, and the message is changed to dead. If the maximum number of retries has not been exceeded, it checks whether the sending is successful. If the sending fails, the producer of the resending message sends the message to the message queue, the consumer of the resending message continues to consume the message, and publishes an event based on the spring event mechanism. It then returns to execute the step of sending the pre-stored message to the message queue for the producer who confirms that the message has been sent. If the sending is successful, the message is changed to sent.

7. The data distribution method based on reliable messaging according to claim 6, characterized in that: The method further comprises the following steps: Receive data synchronization messages from the distribution system through the newly added interface of the target system, and save the received synchronization messages to the local database for subsequent synchronization data processing; According to the configured API service interface, the corresponding synchronization data is obtained from the data distribution platform through the interface to complete business processing; The data distribution system provides a confirmation message interface, through which the target system sends a confirmation message to confirm that data synchronization has been completed. If the confirmation fails, the target system needs to resend the confirmation message. The data distribution system receives the confirmation message sent by the target system, checks the legitimacy of the message, and then confirms that the message producer sends the message to the message queue; Confirm the consumption message of the message consumer and confirm that the message is in the consumed state.

8. A data distribution system based on reliable messaging, characterized in that: include: The interface management module is used to configure the API service interface and determine the basic attributes and system business attributes of each service interface; Subscription data configuration module, used to configure the distribution of subscription data; Subscription queue management module, used to view queue data and logs of data distribution, as well as perform maintenance management; Message management module, used for viewing, maintaining and managing interactive message data; Application management module, used for system access processing, system identification code and key distribution, interface authorization and maintenance management; The data service module is used for queue distribution and data synchronization. It also processes and converts business data according to the processing of the subscription data configuration module to generate subscription data. API service module, used to provide distributed business data to the outside world; Message service module, used for receiving, pre-storing, sending, resending and confirming messages; The data service module distributes queues and synchronizes data. It also processes and converts business data based on the subscription data configuration module. The steps to generate subscription data include: Build a main task thread pool to process the main task ordered set; obtain the value of the cached ordered set container, use a distributed lock to preempt the main task primary key, and obtain the business data primary key of the message producer's distribution data record table in batches through a direct database connection. Then, based on the business data primary key, obtain and assemble data from the business main table and related tables, and store it in the subscription queue data table. In addition, asynchronously write the main task primary key, subtask quantity, subtask primary key, and sorting score into the cached ordered set container, storing them in the form of key-value pairs. Build a subtask thread pool to process the ordered set of subtasks; determine the subtask primary key from the cached ordered set container, use a distributed lock to preempt the subtask primary key, and store the data in the subscription queue data table after matching the subscription data configuration items, filtering the data attributes, and filtering the subscription condition rules based on the business data in the subscription queue data table; The cache ordered collection container is used to divide data into business modules and distribute them to designated collection containers; the subscription queue data table is data generated based on the data in the subscription queue.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images