Reliable message-based data distribution method and system

Abstract

The invention discloses a data distribution method and system based on reliable messages, and the method comprises the steps: carrying out the configuration processing of API service interfaces, and determining the basic attributes and system business attributes of each service interface; performing distribution configuration of subscription data; checking queue data and logs of data distribution, and performing maintenance management; checking, maintaining and managing the interactive message data; system access processing, system identification code and secret key distribution, interface authorization and maintenance management are carried out; the data processing module is used for distributing queues and synchronizing data, processing and converting service data according to the processing of the subscription data configuration module, and generating subscription data; providing distributed service data to the outside; and receiving, pre-storing, sending, resending and confirming the message. The embodiment of the invention can ensure the stability, high efficiency and safety of data synchronization, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data distribution method and system based on reliable messages. Background Art

[0002] There are numerous business systems within an enterprise. As the business continues to develop, the demand for data interaction between business systems is increasing. The coupling between systems is getting higher and higher. The efficiency of calling the data synchronization interface between systems is relatively low, which will lead to a decline in system performance. At the same time, the data consistency between systems cannot be guaranteed, affecting the normal operation of the system business. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to provide a data distribution method and system based on reliable messages, which can ensure the stability, efficiency and security of data synchronization.

[0004] To achieve the above object, an embodiment of the present invention provides a data distribution method based on reliable messages, comprising the following steps:

[0005] Configure the API service interface through the interface management module to determine the basic attributes and system business attributes of each service interface;

[0006] Distribute and configure subscription data through the subscription data configuration module;

[0007] View the queue data and logs of data distribution and perform maintenance management through the subscription queue management module;

[0008] View, maintain and manage interactive message data through the message management module;

[0009] The application management module is used to process system access, distribute system identification codes and keys, and perform interface authorization and maintenance management;

[0010] The data service module distributes queues and synchronizes data, and processes and converts business data according to the processing of the subscription data configuration module to generate subscription data; the processing and conversion includes matching of configuration items, screening of data attributes and filtering of subscription condition rules;

[0011] Provide distributed business data externally through API service modules;

[0012] Messages are received, pre-stored, sent, resent and confirmed through the message service module.

[0013] In some embodiments, the configuration processing of the API service interface by the interface management module to determine the basic attributes and system business attributes of each service interface includes the following steps:

[0014] Configure the basic attributes of each service interface through the API service interface, including interface code, interface name, interface address, interface type, business module, business system, interface identifier, interface protocol, request type, and usage scenario;

[0015] Configure the system business attributes and whitelist of each business system, where the system business attributes include user account, user password, number of failed retries, retry frequency, scheduled acquisition time, warning time, and warning interval; the whitelist includes IP address and remarks;

[0016] Configure the basic information of the synchronization data source system and the basic information of the data synchronization module.

[0017] In some embodiments, the configuration of distributing subscription data by using a subscription data configuration module includes the following steps:

[0018] The subscription data conditions and the custom attribute fields are configured through the subscription data configuration module; wherein the custom attribute fields include the business entity name, attribute code, and attribute name;

[0019] The message producer sends the data synchronization processing message to the data distribution system, which specifically includes: the message producer adds, deletes and modifies the data, and after the data is approved by the last node of the process, a message distribution data record table is generated, and the data synchronization processing message is sent asynchronously to the data distribution system;

[0020] The data distribution system receives the message sent by the message producer, checks whether the message number already exists, and if so, prompts the message producer that the message number already exists; if the message number does not exist, it sends the message to the message producer to send the message to the message queue, and the message producer and the data distribution system asynchronously perform message processing;

[0021] Among them, the content of the data recording table includes system code, message number, business module, process primary key, and business data primary key. One message number is associated with multiple business data, and the data synchronization processing message includes message number, caller system code, message content, message type, and sending time.

[0022] In some embodiments, the method further includes: sending a consumption message of the data distribution system, a pre-stored message, and generating a data distribution queue, specifically including the following steps:

[0023] Send consumption messages to the data distribution system and pre-store the consumption messages in the database message table;

[0024] When the database message table is written successfully, a data distribution queue event is generated based on the spring event mechanism;

[0025] After the subscription generates the data distribution queue event, query whether the current message number has distribution data, if not, do not do anything; if yes, obtain the data of the business module code of the distribution data, and query the subscription data configuration data through the business module code;

[0026] If no subscription data configuration data is found, no task processing will be performed; if yes, the subscription queue will be queried based on the message number to see if there is data already present;

[0027] If data exists in the subscription queue, no processing is performed; if data exists, subscription queue table data is generated, and the subscription queue table data includes one or more subtask queues, subscription queue data table data, and subscription queue message table data;

[0028] After the data is successfully written into the database message table, the main task ID and sorting score are written into the cache ordered set container in an asynchronous manner and stored in the form of key-value;

[0029] The cache ordered collection container is used to divide the data into business modules and distribute them to the specified collection container.

[0030] In some embodiments, the data service module is used to distribute the queues and synchronize the data, and the service data is processed and converted according to the processing of the subscription data configuration module to generate subscription data, including the following steps:

[0031] Construct a main task thread pool for processing cache ordered set containers, which is responsible for processing the main task ordered set; obtain the value of the cache ordered set container, use the distributed lock to seize the main task primary key, and obtain the business data primary key of the distribution data record table of the message producer in batches through the database direct connection method, and then obtain and assemble data from the business main table and the related table according to the business data primary key, and store it in the subscription queue data table. After successfully generating the subscription data, delete the main task ID in the cache ordered set container, release the main task primary key distributed lock, and write the main task primary key, subtask quantity and subtask primary key, and sorting score into the cache ordered set container in an asynchronous manner, and store them in the form of key-value pairs;

[0032] Construct a subtask thread pool for processing the cache ordered set container to process the subtask ordered set; obtain the value of the cache ordered set container, determine the subtask primary key, use the distributed lock to seize the subtask primary key, based on the business data of the subscription queue data table, after matching the configuration items of the subscription data configuration, filtering the data attributes and filtering the subscription condition rules, store the data in the subscription queue data table, after successfully generating the subscription data, delete the subtask primary key from the cache ordered set container, and release the distributed lock of the subtask primary key; publish events based on the spring event mechanism and reduce the number of subtasks by 1. If the result is 0, it means that all subqueues of the queue have completed data distribution, and delete the main task primary key of the cache ordered set container;

[0033] Locate problems based on the execution logs generated during business data processing;

[0034] Obtain detailed business data and save the synchronization data to the distribution system database. The synchronization system obtains the synchronization data by directly connecting to the distribution system database.

[0035] Obtain subscription data configuration. According to the configured business module and business system, obtain configuration information in the subscription data configuration. The configuration information includes interface code, interface name, business system, subscription conditions, and attribute list. The configuration information is used for generating and judging subscription data.

[0036] Generate subscription data, business modules, business systems, subscription data conditions and attribute lists configured for subscription data according to subscription data configuration; wherein the attribute list includes business body name, attribute code and attribute name; when the business system receives subscription data, synchronize part of the data according to subscription data conditions; configure attribute fields of synchronized data according to the attribute list; control row permissions and column permissions according to subscription data conditions and attribute fields, and obtain corresponding subscription data according to corresponding permission screening.

[0037] In some embodiments, the method further comprises:

[0038] After subscribing to an event, check the legitimacy of the event, and after the time legitimacy check passes, confirm that the producer sending the message sends the pre-stored message to the message queue;

[0039] After consuming the message, the consumer who confirms that the message has been sent checks whether the maximum number of retries has been exceeded. If the maximum number of retries has been exceeded, the producer of the dead letter message sends the message to the message queue, the consumer of the dead letter message continues to consume the message, and the message is changed to dead. If the maximum number of retries has not been exceeded, check whether the sending is successful. If the sending fails, the producer of the resending message sends the message to the message queue, the consumer of the resending message continues to consume the message, and publishes an event based on the springing event mechanism. The process then returns to the step where the producer who confirms that the message has been sent sends the pre-stored message to the message queue. If the sending is successful, the message is changed to sent.

[0040] In some embodiments, the method further comprises the following steps:

[0041] Receive data synchronization messages from the distribution system through the newly added interface of the target system, and save the received synchronization messages to the local database for subsequent synchronization data processing;

[0042] According to the configured API service interface, the corresponding synchronization data is obtained from the data distribution platform through the interface to complete the business processing;

[0043] The data distribution system provides a confirmation message interface, through which the target system sends a confirmation message to confirm that data synchronization has been completed. If the confirmation fails, the target system needs to resend the confirmation message.

[0044] The data distribution system receives the confirmation message sent by the target system, checks the legitimacy of the message, and then confirms that the message producer sends the message to the message queue;

[0045] Confirm the consumption message of the message consumer and confirm that the message is in the consumed state.

[0046] Another aspect of the embodiment of the present invention further provides a data distribution system based on reliable messages, including:

[0047] The interface management module is used to configure the API service interface and determine the basic attributes and system business attributes of each service interface;

[0048] Subscription data configuration module, used to configure the distribution of subscription data;

[0049] Subscription queue management module, used to view queue data and logs of data distribution, and perform maintenance management;

[0050] Message management module, used to view, maintain and manage interactive message data;

[0051] Application management module, used for system access processing, system identification code and key allocation, interface authorization and maintenance management;

[0052] The data service module is used for queue distribution and data synchronization, and processes and converts the business data according to the processing of the subscription data configuration module to generate subscription data; the processing and conversion includes matching of configuration items, screening of data attributes and filtering of subscription condition rules;

[0053] API service module, used to provide and distribute business data to the outside world;

[0054] The message service module is used to receive, pre-store, send, resend and confirm messages.

[0055] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0056] The memory is used to store programs;

[0057] The processor executes the program to implement the method described above.

[0058] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0059] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0060] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a data distribution method and system based on reliable messages, which performs configuration processing of API service interfaces through an interface management module to determine the basic attributes and system business attributes of each service interface; performs distribution configuration of subscription data through a subscription data configuration module; checks the queue data and logs of data distribution through a subscription queue management module, and performs maintenance management; checks and maintains interactive message data through a message management module; performs system access processing through an application management module, performs system identification code and key allocation, interface authorization and maintenance management; performs queue distribution and data synchronization through a data service module, and processes and converts business data according to the processing of the subscription data configuration module to generate subscription data; the processing and conversion includes matching of configuration items, screening of data attributes and filtering of subscription condition rules; provides distributed business data to the outside through an API service module; receives, pre-stores, sends, resends and confirms messages through a message service module. The embodiments of the present invention can ensure the stability, efficiency and security of data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a flow chart of the overall steps provided by an embodiment of the present invention;

[0062] Figure 2 is an architecture diagram of a data distribution system based on reliable messaging provided by an embodiment of the present invention;

[0063] Figure 3 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0065] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0066] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0068] Before describing the embodiments of the present invention in detail, some related technologies involved in the embodiments of the present invention are first described as follows:

[0069] There are four common implementation schemes:

[0070] 1. The main business system exports data as files and puts them into shared storage. Each business system reads the file data from the shared storage and then updates it to the business system.

[0071] 2. The main business system actively pushes data to each business system. When the data changes (added, modified, deleted, etc.), the changed data is actively pushed to each business system in a timely manner by calling the interface.

[0072] 3. Each business system periodically polls and calls the interface of the main business system to pull data, and then updates it to the business system.

[0073] 4. The main business system publishes data to MQ (Message Queue), and each business system obtains data from MQ and then updates it to the business system.

[0074] The existing data distribution processing on the market can be implemented in the following steps:

[0075] 1. Synchronous data generation: The source system performs operations such as adding, modifying, and deleting. After the operation is completed, the distribution system is notified to perform synchronous data processing;

[0076] 2. Distribution system processing: After receiving the synchronization processing notification from the source system, the system goes to the source system to obtain the corresponding synchronization data, saves the synchronization data to the distribution system storage (memory or hard disk), and notifies the target system to synchronize data according to the needs;

[0077] 3. Target system data synchronization: After the distribution system obtains new synchronization data, one or more target systems synchronize data regularly or according to the notification of the distribution system, and synchronize the newly added data in the distribution system storage to the target system to complete the data synchronization process.

[0078] The requirements and challenges faced by data distribution and synchronization are:

[0079] 1. Real-time requirements: Some data need to be synchronized in real time to ensure the timeliness of system data.

[0080] 2. Large amount of data: It is necessary to process the synchronization and update of large amounts of data to improve synchronization efficiency and performance.

[0081] 3. Data consistency: Ensure the consistency and integrity of data between different systems.

[0082] The existing implementation solutions on the market have the following shortcomings:

[0083] 1. Data distribution processing does not take into account the data delivery confirmation mechanism;

[0084] When data is distributed to multiple target systems for data synchronization, some data may be lost when the multiple target systems receive data, and the data consistency and integrity between the target system and the source system cannot be guaranteed.

[0085] 2. Data distribution does not take into account the retry mechanism;

[0086] In case of network anomalies or service interface anomalies, data distribution may fail partially or completely. Failed data requires a complete retry mechanism and retry times, and manual retry processing is also supported.

[0087] 3. Custom configuration is not implemented;

[0088] The basic configuration and synchronization fields of multi-target system data synchronization are different. The synchronization processing configuration and logic are hard-coded in the synchronization interface code. As the business develops, the interface code needs to be modified to adjust the business.

[0089] 4. Synchronous logging is not implemented;

[0090] When multiple data are synchronized to multiple target systems, the success or failure of the processing needs to be recorded in the synchronization log, which can be easily queried and retried;

[0091] 5. Batch processing is not considered;

[0092] When the amount of data is large, the synchronization data is frequently added, deleted, and modified. Too frequent data synchronization affects the external services of the business system, so it is necessary to consider data synchronization in batches.

[0093] 6. Need to support synchronous field addition and reduction;

[0094] When source system database fields are added or deleted, synchronized data fields are added or deleted, and the data synchronization codes of each target system need to be modified synchronously, which increases the difficulty of system maintenance. The synchronized fields need to be configured to facilitate addition and deletion;

[0095] 7. Support permissions are required;

[0096] The source system and the target system support data permissions. During the data synchronization process, data permissions need to be processed synchronously to ensure data security.

[0097] The existing data distribution systems on the market handle basic data synchronization between different systems. For multi-system synchronization and scenarios with high requirements for data synchronization, the existing implementation methods are not satisfactory. In view of the above shortcomings, the present invention is designed and optimized to ensure the stability, efficiency and security of data synchronization.

[0098] The data distribution method and system based on reliable messages provided by the embodiments of the present invention relate to the field of computer technology. The data distribution method based on reliable messages provided by the embodiments of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a data distribution method based on reliable messages, etc., but is not limited to the above forms.

[0099] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0100] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0101] In addition, the server can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0102] In some implementation environments, the present invention provides a reliable message-based data distribution method, referring to Figure 1 , Figure 1 The flowchart of the reliable message-based data distribution method applied to the server provided in the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 1 , the method may include the following steps:

[0103] Configure the API service interface through the interface management module to determine the basic attributes and system business attributes of each service interface;

[0104] Distribute and configure subscription data through the subscription data configuration module;

[0105] View the queue data and logs of data distribution and perform maintenance management through the subscription queue management module;

[0106] View, maintain and manage interactive message data through the message management module;

[0107] The application management module is used to process system access, distribute system identification codes and keys, and perform interface authorization and maintenance management;

[0108] The data service module distributes queues and synchronizes data, and processes and converts business data according to the processing of the subscription data configuration module to generate subscription data; the processing and conversion includes matching of configuration items, screening of data attributes and filtering of subscription condition rules;

[0109] Provide distributed business data externally through API service modules;

[0110] Messages are received, pre-stored, sent, resent and confirmed through the message service module.

[0111] The specific implementation process of the present invention is described in detail below in conjunction with the accompanying drawings:

[0112] The embodiment of the present invention provides a system for data distribution based on reliable messages, which specifically includes the following modules and services:

[0113] (1) Interface management: used to configure the service interface (API application programming interface), including interface code, interface name, interface type, business module, business system, interface identifier, interface protocol, request type, interface address, etc.

[0114] (2) Subscription data configuration: used to configure the distribution configuration of subscription data, including interface code, interface name, business system, subscription conditions, data attributes, etc.

[0115] (3) Subscription queue management: queue data, log viewing and maintenance management for data distribution;

[0116] (4) Message management: used for viewing and maintaining interactive message data;

[0117] (5) Application management: used for system access allocation system identification code and key, interface authorization and maintenance management;

[0118] (6) Data service: used for queue distribution and data synchronization. It processes and converts business data according to subscription data configuration, including configuration item matching, data attribute screening and subscription condition rule filtering, and generates subscription data.

[0119] (7). API service: used to provide and distribute business data to the outside world;

[0120] (8). Message service: responsible for receiving, pre-storing, sending, resending and confirming messages.

[0121] refer to Figure 2 The method of the reliable message data distribution system provided by the embodiment of the present invention includes the following configuration and operation steps:

[0122] (1). Step 1 is used to configure the API service interface (API application programming interface);

[0123] In interface management, you can configure the basic properties of each service interface, including interface code, interface name, interface address, interface type, business module, business system, interface identifier, interface protocol, request type, and usage scenario, and then configure the system business properties and whitelist of each business system in detail. The system business properties include user account, user password, number of failed retries, retry frequency, scheduled acquisition time, warning time, and warning interval. The whitelist includes IP address and remarks. Interface management is used to configure the basic information of the synchronization data source system and the basic information of the data synchronization module.

[0124] (2). Step 2, used to configure the distribution configuration of subscription data;

[0125] In the subscription data configuration, you can configure subscription data conditions and custom attribute fields by selecting different interfaces and different business systems. The business module is automatically brought out after selecting the interface. The attribute list includes the business body name, attribute code, and attribute name. The attribute list is a list of corresponding business attributes obtained according to the business module. The subscription data information configuration is added on top of the basic data data configuration, and is used to customize the synchronization attributes and fields of different systems.

[0126] The configuration data of step 1 and step 2 are used for subsequent processing.

[0127] (3). Step 3, the message producer sends a data synchronization processing message to the data distribution system;

[0128] After the message producer (source system) adds, deletes, and modifies data and passes the approval of the last node of the process, a message distribution data record table is generated. The content of the record table includes system code, message number (must be unique), business module, process primary key (ID), business data primary key (ID), etc. One message number can be associated with multiple business data. The asynchronous data is sent to the data distribution system for synchronous processing. The message includes the message number (unique), caller system code, message content, message type, sending time, etc. The message distribution data record table needs to be used for synchronization data.

[0129] (4). Step 4, the data distribution system receives the message sent by the message producer;

[0130] The data distribution system receives the message sent by the message producer and checks whether the message number already exists. If so, it prompts the message producer that the message number already exists. If the message number does not exist, it sends the message to the message producer to send the message to the message queue. The use of the message queue can decouple the business code and improve the processing power of the system. The message producer and the data distribution system perform message processing asynchronously.

[0131] (5) Step 5, send message consumers (data distribution system) to consume messages, pre-store messages and generate data distribution queues;

[0132] The message consumer (data distribution system) consumes the message and pre-stores the message in the database message table. After the writing is successful, the data distribution queue event is published based on the spring event mechanism. After the data distribution queue event is subscribed, it is queried whether the message number has data distribution data. If not, no processing is performed. Otherwise, the business module encoding data of the distribution data is obtained, and the subscription data configuration data is queried through the business module encoding. If not, no task processing is performed. Otherwise, the subscription queue is queried according to the message number to see if there is data. If it exists, no processing is performed. Otherwise, subscription queue table data (including one or more subtask queues), subscription queue data table data, and subscription queue message table data are generated. After the data is successfully written to the database, the main task ID and sorting score are written to the cache (redis) ordered collection container for persistence in an asynchronous manner and stored in the cache (redis) in the form of key-value, where the key is named sync:executor:module synchronization service:main_tasks, such as material synchronization service and color synchronization service. The purpose of using the cache (redis) ordered collection container is to divide the data by business module and distribute it to the specified collection container, so that the use of subscription queues can support horizontal expansion and improve processing efficiency.

[0133] (6) Step 6: Business processing: the data distribution service processes the cache (redis) ordered set container data, obtains business detailed data through a direct database connection, and then processes and converts the data according to the subscription data configuration, and finally generates subscription data;

[0134] 6.1. Construct a main task thread pool for processing the cache (redis) ordered set container, which is responsible for processing the main task ordered set, obtaining the value of the cache (redis) ordered set, the main task primary key (ID), and in order to prevent repeated execution, the distributed lock is used to seize the main task primary key (ID). Through the database direct connection method, it is responsible for obtaining the business data primary key of the distribution data record table of the message producer (source system) in batches, and then obtaining and assembling data from the business main table and the associated table according to the business data primary key, and storing it in the subscription queue data table. After successfully generating the subscription data, delete the main task ID in the cache (redis) ordered set, release the main task primary key (ID) distributed lock, and write the main task primary key (ID), subtask quantity and subtask primary key (ID), and sorting score to the cache (redis) ordered set container asynchronously, and store them in the cache (redis) in the form of key-value pairs, where the keys are named sync:executor:module synchronization service:tasks and sync:executor:task_progress respectively;

[0135] 6.2. Construct a subtask thread pool for processing cache (redis) ordered set containers, which is responsible for processing subtask ordered sets and obtaining the subtask primary key (ID) of the cache (redis) ordered set value. In order to prevent repeated execution, a distributed lock is used to preempt the subtask primary key (ID). Based on the business data of the subscription queue data table, after matching the configuration items of the subscription data configuration, filtering the data attributes and filtering the subscription condition rules, the data is stored in the subscription queue data table. After successfully generating the subscription data, the subtask primary key (ID) is deleted from the cache (redis) ordered set, the distributed lock of the subtask primary key (ID) is released, the event is published based on the spring event mechanism, and the number of subtasks is reduced by 1. If the result is 0, it means that all subqueues of the queue have completed data distribution, and the main task primary key (ID) of the redis ordered set is deleted.

[0136] 6.3. In steps 6.1 and 6.2, corresponding execution logs will be generated during the processing, which will facilitate problem location, ensure data uniqueness, and avoid duplication or omission of processing.

[0137] 6.4. Obtain detailed business data. The synchronized data has been synchronized and saved in the distribution system database (processed in step 5). The synchronization system obtains the synchronized data by directly connecting to the distribution system database;

[0138] 6.5. Obtain subscription data configuration. According to the configured business module and business system, obtain configuration information in the subscription data configuration, including interface code, interface name, business system, subscription conditions, attribute list, etc., for the next step of subscription data generation judgment;

[0139] 6.6. Generate subscription data based on subscription data configuration, business module, business system, subscription data conditions and attribute list (attribute list includes business body name, attribute code, attribute name). The business system is the target system, the business module is the synchronized data source module (material, pattern or graphic document and other information), the subscription condition is to filter data use, the target system receives data from the business system, and can synchronize only part of the data according to the condition, rather than full synchronization. At the same time, according to the attribute list, the synchronization data field can be configured to facilitate subsequent expansion and use. The generated subscription data can control row permissions and column permissions according to the filtering conditions and attribute fields, and the corresponding subscription data can be obtained according to the corresponding permission filtering.

[0140] (7). Step 7, confirm that the message producer sends the pre-stored message to the message queue;

[0141] After subscribing to the event, the legitimacy of the event is checked and confirmed, and the message producer sends the pre-stored message to the message queue.

[0142] (8). Step 8, confirm that the message consumer consumes the message and sends the message to the target system;

[0143] Confirm that the message consumer consumes the message and checks whether the maximum number of retries is exceeded. If the maximum number of retries is exceeded, the dead letter message producer sends the message to the message queue, the dead letter message consumer consumes the message, and the message is changed to dead. Otherwise, check whether the sending is successful. If the sending fails, the resend message producer sends the message to the message queue, the resend message consumer consumes the message, changes the number, time, etc., and publishes events based on the spring event mechanism to go to step 7 to continue execution, otherwise change the message to sent.

[0144] (9). Step 9, the target system receives the message and stores it;

[0145] The target system adds a new interface to process and receive data synchronization messages from the distribution system, and save the received synchronization messages to the local database for subsequent synchronization data processing.

[0146] (10). Step 10, the target system completes business processing;

[0147] According to the API service interface configured in step 1, the target system will obtain the corresponding synchronization data from the data distribution platform through the interface to complete business processing.

[0148] (11). Step 11, the target system sends a confirmation message;

[0149] The data distribution system provides a confirmation message interface, through which the target system sends a confirmation message to confirm that data synchronization has been completed. If the confirmation fails, the target system needs to resend the confirmation message.

[0150] (12). Step 12, the data distribution system receives a confirmation message sent by the target system;

[0151] The data distribution system receives the confirmation message sent by the target system, checks the legitimacy of the message, and then confirms that the message producer sends the message to the message queue.

[0152] (13). Step 13, the data distribution system confirms the message;

[0153] Confirm that the message consumer consumes the message and confirms that the message has been consumed.

[0154] In summary, the present invention has the following characteristics:

[0155] 1. Independent microservice deployment;

[0156] Adopt independent microservices (Spring Cloud Al ibaba) to implement data distribution functions, which does not affect the original business system functions. At the same time, it is compatible with the existing microservice architecture and can be expanded or reduced as the business develops;

[0157] 2. Custom configuration;

[0158] Add interface management and subscription data management to customize the system configuration and synchronize basic information and fields;

[0159] 3. Synchronous log records;

[0160] Add synchronization queue management and subscription queue management, record data distribution information and logs;

[0161] 4. Retry configuration;

[0162] Added configurations such as retry times and whitelists for different systems to facilitate retry processing for different systems;

[0163] 5. Batch processing;

[0164] Data distribution processing uses batch data processing to avoid frequent processing affecting the external services of the original system;

[0165] 6. Synchronous or asynchronous processing;

[0166] Data distribution calls scheduled tasks for processing, meeting two different situations: manual triggering and scheduled triggering, and realizing two requirements of synchronous or asynchronous processing;

[0167] 7. Support synchronous field addition and subtraction;

[0168] Supports adding and deleting distribution data fields. Synchronous fields can be configured in subscription data management without modification in synchronization code, which facilitates system maintenance.

[0169] 8. Support multiple synchronization methods;

[0170] Supports database intermediate table or interface data distribution, and can be configured and processed according to actual needs.

[0171] 9. Support data permissions;

[0172] Data distribution: distributes and processes data according to the data permission requirements of the source system and the target system to ensure data security and consistency between different systems.

[0173] Compared with the prior art, the present invention has the following advantages:

[0174] 1. Through interface management configuration, provide API service interface to allow various business systems to access and obtain data.

[0175] 2. Through subscription data configuration, different business data attributes can be configured according to different interfaces and different business systems to improve business adaptation to different data scenarios. At the same time, when business data changes (addition, modification, deletion, etc.), the change message can be sent in time to notify the target system. When the target system receives the message notification, it will call the corresponding interface to obtain data to meet the timeliness requirements of the data, while avoiding periodic polling interfaces and reducing the consumption of server resources.

[0176] In addition, the messaging function of the reliable message data distribution system in the embodiment of the present invention can use message middleware (RabbitMQ, RocketMQ or Kafka) to increase the data analysis and processing tool (ClickHouse). These message middleware can provide certain reliability and stability, rich functions and flexibility, and have a huge user community and a rich ecosystem to help developers better understand and use it.

[0177] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] The embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned data distribution method based on reliable messages when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0179] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] See also Figure 3 , Figure 3 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0181] The processor 301 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0182] The memory 302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 302 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 302, and the processor 301 calls and executes the data distribution method based on reliable messages in the embodiment of the present invention;

[0183] Input / output interface 303, used to implement information input and output;

[0184] The communication interface 304 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WI FI, Bluetooth, etc.);

[0185] A bus 305 that transmits information between the various components of the device (e.g., the processor 301, the memory 302, the input / output interface 303, and the communication interface 304);

[0186] The processor 301 , the memory 302 , the input / output interface 303 and the communication interface 304 are connected to each other in communication within the device via the bus 305 .

[0187] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned data distribution method based on reliable messages is implemented.

[0188] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0189] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] It should be noted that in various specific embodiments of the present invention, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present invention needs to obtain the user's sensitive personal information, it will obtain the user's separate permission or consent through a pop-up window or jump to a confirmation page, and after clearly obtaining the user's separate permission or consent, it will obtain the necessary user-related data for the normal operation of the embodiment of the present invention.

[0191] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0192] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0193] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0195] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0196] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0197] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0198] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0199] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0201] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A data distribution method based on reliable messages, characterized in that: The following steps are involved: Configure the API service interface through the interface management module to determine the basic attributes and system business attributes of each service interface; Distribute and configure subscription data through the subscription data configuration module; View the queue data and logs of data distribution and perform maintenance management through the subscription queue management module; View, maintain and manage interactive message data through the message management module; The application management module is used to process system access, distribute system identification codes and keys, and perform interface authorization and maintenance management; The data service module is used to distribute queues and synchronize data. The module also processes and converts business data according to the processing of the subscription data configuration module to generate subscription data. The processing conversion includes matching of configuration items, screening of data attributes and filtering of subscription condition rules; Provide distributed business data externally through API service modules; Messages are received, pre-stored, sent, resent and confirmed through the message service module.

2. A data distribution method based on reliable messaging according to claim 1, characterized in that: The configuration process of the API service interface is performed through the interface management module to determine the basic attributes and system business attributes of each service interface, including the following steps: Configure the basic attributes of each service interface through the API service interface, including interface code, interface name, interface address, interface type, business module, business system, interface identifier, interface protocol, request type, and usage scenario; Configure the system business attributes and whitelist of each business system, where the system business attributes include user account, user password, number of failed retries, retry frequency, scheduled acquisition time, warning time, and warning interval; the whitelist includes IP address and remarks; Configure the basic information of the synchronization data source system and the basic information of the data synchronization module.

3. A reliable message data distribution method according to claim 1, characterized in that: The distribution configuration of subscription data by the subscription data configuration module includes the following steps: The subscription data conditions and the custom attribute fields are configured through the subscription data configuration module; wherein the custom attribute fields include the business entity name, attribute code, and attribute name; The message producer sends the data synchronization processing message to the data distribution system, which specifically includes: the message producer adds, deletes and modifies the data, and after the data is approved by the last node of the process, a message distribution data record table is generated, and the data synchronization processing message is sent asynchronously to the data distribution system; The data distribution system receives the message sent by the message producer, checks whether the message number already exists, and if so, prompts the message producer that the message number already exists; if the message number does not exist, the message is sent to the message producer to send the message to the message queue, and the message producer and the data distribution system asynchronously perform message processing; Among them, the content of the data recording table includes system code, message number, business module, process primary key, and business data primary key. One message number is associated with multiple business data, and the data synchronization processing message includes message number, caller system code, message content, message type, and sending time.

4. A data distribution method based on reliable messaging according to claim 3, characterized in that: The method further includes: sending consumption messages and pre-stored messages of the data distribution system and generating a data distribution queue, specifically including the following steps: Send consumption messages to the data distribution system and pre-store the consumption messages in the database message table; When the database message table is written successfully, a data distribution queue event is generated based on the spring event mechanism; After the subscription generates the data distribution queue event, query whether the current message number has distribution data, if not, do not do anything; if yes, obtain the data of the business module code of the distribution data, and query the subscription data configuration data through the business module code; If no subscription data configuration data is found, no task processing will be performed; if yes, the subscription queue will be queried based on the message number to see if there is data already present; If data exists in the subscription queue, no processing is performed; if data exists, subscription queue table data is generated, and the subscription queue table data includes one or more subtask queues, subscription queue data table data, and subscription queue message table data; After the data is successfully written into the database message table, the main task ID and sorting score are written into the cache ordered set container in an asynchronous manner and stored in the form of key-value; The cache ordered collection container is used to divide the data into business modules and distribute them to the specified collection container.

5. A data distribution method based on reliable messaging according to claim 4, characterized in that: The data service module is used to distribute the queues and synchronize the data, and the service data is processed and converted according to the processing of the subscription data configuration module to generate subscription data, including the following steps: Construct a main task thread pool for processing cache ordered set containers, which is responsible for processing the main task ordered set; obtain the value of the cache ordered set container, use the distributed lock to seize the main task primary key, and obtain the business data primary key of the distribution data record table of the message producer in batches through the database direct connection method, and then obtain and assemble data from the business main table and the related table according to the business data primary key, and store it in the subscription queue data table. After successfully generating the subscription data, delete the main task ID in the cache ordered set container, release the main task primary key distributed lock, and write the main task primary key, subtask quantity and subtask primary key, and sorting score into the cache ordered set container in an asynchronous manner, and store them in the form of key-value pairs; Construct a subtask thread pool for processing cache ordered set containers to process subtask ordered sets; obtain the value of the cache ordered set container, determine the subtask primary key, use a distributed lock to seize the subtask primary key, and based on the business data of the subscription queue data table, store the data in the subscription queue data table after matching the configuration items of the subscription data configuration, filtering the data attributes, and filtering the subscription condition rules. After successfully generating the subscription data, delete the subtask primary key from the cache ordered set container and release the distributed lock of the subtask primary key; publish events based on the spring event mechanism and reduce the number of subtasks by 1. If the result is 0, it means that all subqueues of the queue have completed data distribution, and delete the primary key of the main task of the cache ordered set container; Locate problems based on the execution logs generated during business data processing; Obtain detailed business data and save the synchronization data to the distribution system database. The synchronization system obtains the synchronization data by directly connecting to the distribution system database. Obtain subscription data configuration. According to the configured business module and business system, obtain configuration information in the subscription data configuration. The configuration information includes interface code, interface name, business system, subscription conditions, and attribute list. The configuration information is used for generating and judging subscription data. Generate subscription data, business modules, business systems, subscription data conditions and attribute lists configured for subscription data according to subscription data configuration; wherein the attribute list includes business body name, attribute code and attribute name; when the business system receives subscription data, synchronize part of the data according to subscription data conditions; configure attribute fields of synchronized data according to the attribute list; control row permissions and column permissions according to subscription data conditions and attribute fields, and obtain corresponding subscription data according to corresponding permission screening.

6. A data distribution method based on reliable messaging according to claim 5, characterized in that: The method further comprises: After subscribing to an event, check the legitimacy of the event, and after the time legitimacy check passes, confirm that the producer sending the message sends the pre-stored message to the message queue; After consuming the message, the consumer who confirms that the message has been sent checks whether the maximum number of retries has been exceeded. If the maximum number of retries has been exceeded, the producer of the dead letter message sends the message to the message queue, the consumer of the dead letter message continues to consume the message, and the message is changed to dead. If the maximum number of retries has not been exceeded, check whether the sending is successful. If the sending fails, the producer of the resending message sends the message to the message queue, the consumer of the resending message continues to consume the message, and publishes an event based on the spring event mechanism, and returns to execute the step of sending the producer who confirms that the message has been sent to send the pre-stored message to the message queue. If the sending is successful, the message is changed to sent.

7. A data distribution method based on reliable messaging according to claim 6, characterized in that: The method further comprises the following steps: Receive data synchronization messages from the distribution system through the newly added interface of the target system, and save the received synchronization messages to the local database for subsequent synchronization data processing; According to the configured API service interface, the corresponding synchronization data is obtained from the data distribution platform through the interface to complete the business processing; The data distribution system provides a confirmation message interface, through which the target system sends a confirmation message to confirm that data synchronization has been completed. If the confirmation fails, the target system needs to resend the confirmation message. The data distribution system receives the confirmation message sent by the target system, checks the legitimacy of the message, and then confirms that the message producer sends the message to the message queue; Confirm the consumption message of the message consumer and confirm that the message is in the consumed state.

8. A data distribution system based on reliable messages, characterized in that: include: The interface management module is used to configure the API service interface and determine the basic attributes and system business attributes of each service interface; Subscription data configuration module, used to configure the distribution of subscription data; Subscription queue management module, used to view queue data and logs of data distribution, and perform maintenance management; Message management module, used to view, maintain and manage interactive message data; Application management module, used for system access processing, system identification code and key allocation, interface authorization and maintenance management; The data service module is used for queue distribution and data synchronization, and processes and converts business data according to the processing of the subscription data configuration module to generate subscription data; The processing conversion includes matching of configuration items, screening of data attributes and filtering of subscription condition rules; API service module, used to provide and distribute business data to the outside world; The message service module is used to receive, pre-store, send, resend and confirm messages.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Images