A method, apparatus and related device for data synchronization transmission

Abstract

The application provides a data synchronization sending method and device and related equipment, target system information of subscribed data and service node information of data synchronization sending are determined, based on the service node information, the to-be-synchronized data that each service node needs to synchronize is determined, the number of Disruptor Consumer consumption threads of each service node is configured, the to-be-synchronized data is task-allocated based on the number of consumption threads, and the service node sends the allocated synchronization sending task to the corresponding target system in an asynchronous parallel mode. In the case of limited hardware resources and inability to add message queues and other external middleware, through the high-performance memory queue Disruptor, streaming query, asynchronous parallel processing and other technologies, a large amount of data can be efficiently synchronized to the business system that has subscribed to the data in a friendly manner with small memory and cpu resource occupation, resources are saved, and the data synchronization business can be efficiently and quickly completed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information processing, and in particular to a data synchronization sending method and device and related equipment. BACKGROUND

[0002] With the development of informatization, the network services provided in the society are gradually developing in the direction of mainstream technology development, such as microservices, distributed clusters, data integration platforms, big data platforms, and large system architectures. The resources required for the application and deployment of these technologies are also increasing, and the amount of data to be processed is also growing and becoming more complex. For systems with limited server resources, it is difficult to ensure that the massive data is synchronized and sent to the target system that subscribes to the data in a timely and accurate manner. In the face of such data synchronization sending requirements for massive and complex data, the industry currently implements data synchronization to multiple information subscription systems by adding middleware. This approach requires some degree of re-editing of the system's communication method, and the efficiency of synchronizing large and complex data is also relatively general. Therefore, how to provide a data synchronization sending method that does not rely on middleware technology and is efficient has become an important problem that needs to be solved in the industry. SUMMARY

[0003] Therefore, the embodiments of the present application provide a data synchronization sending method and device and related equipment to at least or partially solve the above problems.

[0004] In a first aspect, the embodiments of the present application provide a data synchronization sending method, comprising:

[0005] determining target system information for subscribing data and service node information for data synchronization sending, the service node information at least including the number of service nodes and the number information of each service node; the target system information including one or more of target system ID, target system name, target system address, and target subscription data type;

[0006] based on the service node information, determining the to-be-synchronized data that each service node needs to synchronize and send;

[0007] configuring the number of Disruptor Consumer consumer threads for each service node, and based on the number of Disruptor Consumer consumer threads, assigning tasks to the to-be-synchronized data;

[0008] using the service nodes to send the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous and parallel manner.

[0009] Optionally, in an embodiment of the present application, the determining of the to-be-synchronized data to be synchronously transmitted by each service node based on the service node information comprises:

[0010] The following SQL statement is executed to implement the batch calculation of the to-be-synchronized data:

[0011] select * from task_ data_sync where id % [service node number] = [node number],

[0012] The to-be-synchronized data is assigned to the service node with the node number determined by the batch calculation through the modulo operation on the value of the task number of the to-be-synchronized data, and each service node is configured to process different to-be-synchronized data.

[0013] Optionally, in an embodiment of the present application, the determining of the to-be-synchronized data to be synchronously transmitted by each service node based on the service node information comprises:

[0014] The to-be-synchronized data to be synchronously transmitted by each service node is determined based on the service node corresponding to the service node information by using a streaming query mode.

[0015] Optionally, in an embodiment of the present application, the determining of the to-be-synchronized data to be synchronously transmitted by each service node based on the service node corresponding to the service node information by using a streaming query mode comprises:

[0016] When the SQL statement is executed to implement the batch calculation of the to-be-synchronized data, a configuration ResultSetType.FORWARD_ONLY is added in a Mapper interface method of a Mybatis query to-be-synchronized data sql to start the streaming query mode, which is used to determine the to-be-synchronized data to be synchronously transmitted by each service node.

[0017] Optionally, in an embodiment of the present application, before the task distribution of the to-be-synchronized data based on the Disruptor Consumer thread number of each service node, the method further comprises:

[0018] According to the data type information and the corresponding subscription target system information of the to-be-synchronized data, the to-be-synchronized data is encapsulated to form a to-be-synchronized data set in the form of a JSON string message, and the encapsulated to-be-synchronized data set at least includes one or more of the data id, the original data message, the target system information and the synchronization mode of the to-be-synchronized data.

[0019] The synchronization data set is taken as target data to be synchronized, and is used to support task allocation for the data to be synchronized.

[0020] Optionally, in an embodiment of the present application, the sending of the data to be synchronized to the target system corresponding to the target system information in an asynchronous parallel manner by using the service node comprises:

[0021] The JSON string message form of the data set to be synchronized is sent to the Disruptor memory queue of the corresponding service node in a concurrent manner by using an asynchronous processing function, so as to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous parallel manner by using the service node.

[0022] Optionally, in an embodiment of the present application, the sending of the data to be synchronized to the target system corresponding to the target system information in an asynchronous parallel manner by using the service node comprises:

[0023] The JSON string message form of the data set to be synchronized is sent to the Disruptor memory queue of the corresponding service node in a concurrent manner by using an asynchronous processing function, so as to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous parallel manner by using the service node.

[0024] According to the analysis result, the target system corresponding to the original message data is determined according to the target system address.

[0025] Optionally, in an embodiment of the present application, the method further comprises: configuring a consumption waiting strategy of the Disruptor memory queue as a blocking waiting strategy.

[0026] In a second aspect, based on the method for data synchronization sending in the first aspect of the present application, an embodiment of the present application further provides a device for data synchronization sending, comprising:

[0027] A determination module is configured to determine target system information of subscribed data and service node information for data synchronization sending, wherein the service node information at least comprises the number of service nodes and the number information of each service node; and the target system information comprises one or more of target system ID, target system name, target system address, and target subscription data type.

[0028] A query module is configured to determine the data to be synchronized that needs to be synchronized and sent by each service node based on the service node information.

[0029] The distribution module is configured to configure the number of Disruptor Consumer threads of each service node, and perform task distribution on the data to be synchronized based on the number of Disruptor Consumer threads.

[0030] The synchronization module is configured to use the service nodes to send the assigned synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner.

[0031] In a third aspect, the embodiments of the present application further provide a computer storage medium, which has computer executable instructions stored thereon, and the computer executable instructions are executed to perform any one of the methods for data synchronization sending according to the first aspect of the embodiments of the present application.

[0032] In a fourth aspect, the embodiments of the present application further provide an electronic device, which comprises a processor, a memory, a communication interface and a communication bus, and the processor, the memory and the communication interface complete communication with each other through the communication bus.

[0033] The memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform any one of the methods for data synchronization sending according to the first aspect of the embodiments of the present application.

[0034] The present application provides a method and device for data synchronization sending and related equipment. The target system information of subscribed data and the service node information for data synchronization sending are determined, and the service node information at least includes the number of service nodes and the number information of each service node. The target system information includes one or more of the target system ID, the target system name, the target system address and the target subscription data type. Based on the service node information, the data to be synchronized which needs to be synchronized and sent by each service node is determined. The number of Disruptor Consumer threads of each service node is configured, and the task distribution is performed on the data to be synchronized based on the number of Disruptor Consumer threads. The assigned synchronization sending task is used to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous parallel manner. This data synchronization sending method, in the case of limited hardware resources and unable to add message queues and other external middleware, through the high-performance memory queue Disruptor, streaming query, asynchronous parallel processing and other technologies, occupies less memory and cpu resources, and supports efficiently synchronizing a large amount of data to the business system which has subscribed to the data in a friendly manner, saves resources, and also efficiently and quickly completes the data synchronization business. BRIEF DESCRIPTION OF DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art based on these drawings.

[0036] Figure 1 A workflow schematic diagram of a data synchronization sending method provided by an embodiment of the present application;

[0037] Figure 2 A structure schematic diagram of a data synchronization sending device provided by an embodiment of the present application.

[0038] Figure 3 A structure schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the technical solutions in the embodiments of the present application better understood, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art should belong to the scope of protection of the present application.

[0040] It should be understood that the various steps recorded in the method embodiments of the present application can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present application is not limited in this respect.

[0041] Embodiment one,

[0042] The embodiments of the present application provide a data synchronization sending method, as shown in Figure 1 Figure 1 A workflow schematic diagram of a data synchronization sending method provided by an embodiment of the present application, including:

[0043] Step S101, determining target system information of subscribed data and service node information for data synchronization sending. In the present embodiment, the service node is the current available system server, wherein the service node information at least includes the number of service nodes and the number information of each service node; the target system information includes one or more of target system ID, target system name, target system address, and target subscription data type. In the present embodiment, the target information to be synchronized and sent and the node information supporting synchronization sending are determined at this stage, so as to more accurately support the data synchronization sending process.​

[0044] In step S102, based on the service node information, the to-be-synchronized data that needs to be synchronized by each service node is determined. In this stage, the service node information obtained is used to preliminarily allocate the to-be-synchronized data, so as to use multiple available nodes to process a large number of data synchronization tasks in time and improve the efficiency of data synchronization. In this embodiment of the application, the process is exemplarily illustrated as follows: for example, when multiple data synchronization tasks need to be processed, it is determined that the currently available service nodes are nodes numbered 0, 1 and 2, and then the multiple data synchronization tasks are allocated to the nodes numbered 0, 1 and 2 according to the numbers, so as to preliminarily allocate the to-be-synchronized data. This processing manner can more effectively use limited servers as nodes to perform the data synchronization process and improve the ability of the system to solve a large number of data synchronization burst demands.

[0045] In step S103, the number of Disruptor Consumer consumption threads of each service node is configured, and based on the number of Disruptor Consumer consumption threads, the to-be-synchronized data is allocated. Disruptor is a high-performance message queue developed by LMAX, which is mainly used to solve the delay problem of the memory queue. Disruptor can be used as a high-performance bounded memory queue, which is very suitable for [producer-consumer] scenarios and [publish-subscribe] scenarios. The most commonly used scenario [producer-consumer] refers to the scenario of “one producer and multiple consumers” for the information of the memory queue, and requires sequential processing. The data synchronization process described in the embodiment of the application belongs to a typical [producer-consumer] scenario and also belongs to a [publish-subscribe] scenario. In this embodiment of the application, the data synchronization process performed by the Disruptor queue can significantly improve the efficiency of data synchronization and can more friendly face the work demand of synchronizing a large number of data bursts. In this embodiment of the application, when the number of Disruptor Consumer consumption threads of each service node is configured, the number of consumption threads of Disruptor is usually managed by an external thread pool, rather than being directly set in the Disruptor framework, so that the user can configure the size of the external thread pool according to the actual demand, so as to better optimize the performance and resource utilization of the high-performance queue Disruptor. Therefore, the efficiency of the data synchronization process described in the application is further improved.

[0046] Optionally, in an embodiment of the application, based on the service node information, the to-be-synchronized data that needs to be synchronized by each service node is determined, including: executing the following SQL statement to implement batch calculation of the to-be-synchronized data:

[0047] select * from task_ data_sync where id % [service node number] = [node number], by taking the modulus operation on the value of the number of tasks of the to-be-synchronized data, the to-be-synchronized data is allocated to the service node with the number determined by the batch calculation, and each service node is configured to process different to-be-synchronized data. In this stage of the embodiment of the application, by taking the modulus operation, it is realized that the same to-be-synchronized data will only be processed by the same service node, and the to-be-synchronized data processed by each service node is balanced, and the system runs more evenly. Further, different service nodes can process different to-be-processed data to more reasonably use resources and further ensure the processing efficiency of the to-be-synchronized task.

[0048] Optionally, in an optional implementation of an embodiment of the application, based on the service node information, the to-be-synchronized data that needs to be synchronized and sent by each service node is determined, comprising: based on the service node corresponding to the service node information, using a streaming query method, determining the to-be-synchronized data that needs to be synchronized and sent by each service node. In this embodiment, the to-be-synchronized data that needs to be synchronized and sent by each service node is determined by using the streaming query method, which can effectively solve the problem of OMM crash caused by too large to-be-synchronized data, so as to ensure that the system has good working stability and working efficiency.

[0049] Optionally, in an embodiment of the present application, based on the service node information corresponding to the service node, the to-be-synchronized data required to be synchronously transmitted by each service node is determined by using a streaming query mode, comprising: when performing the SQL statement to achieve batch calculation of the to-be-synchronized data, adding configuration ResultSetType.FORWARD_ONLY in a Mapper interface method of a Mybatis query to-be-synchronized data sql to start the streaming query mode, for determining the to-be-synchronized data required to be synchronously transmitted by each service node. ResultSetType.FORWARD_ONLY means that the cursor of the result set can only move forward and does not support scrolling. Since the result set can only be traversed forward, the performance is good, and it is suitable for service applications in query scenarios that need to process a large amount of data. In this embodiment of the present application, by adding configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of the Mybatis query to-be-synchronized data sql, a large amount of data is processed, the operation of data rollback or random access of the target data obtained by querying, i.e., the result set, and other complex scenarios is reduced, and better query performance is achieved. Correspondingly, the method further comprises: after adding configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of the Mybatis query to-be-synchronized data sql to start the streaming query mode, the method can further comprise: configuring a ResultHandler <t>The parameters are passed into the callback function to receive the result set processed by the streaming query. This enables the method described in the embodiments of the present application to meet more application scenarios, such as the operation of data rollback or random access on the query result, and improves the scenario applicability of the data synchronization sending method described in the embodiments of the present application. In addition, the process of querying and obtaining the data to be synchronized is relatively simple through this method, which facilitates system development and maintenance by programmers.

[0050] In step S104, the service node is used to send the assigned synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner. In this embodiment of the present application, the Disruptor Consumer thread quantity configured in the service node is used to perform the synchronization sending of data in an asynchronous parallel manner, so that the Disruptor high-performance message queue is used to support the synchronization sending of data, and the efficient synchronization business requirement for a large amount of data is stably realized with less system resource occupation, thereby improving the efficiency of data synchronization sending.

[0051] Optionally, in an embodiment of the present application, the Disruptor Consumer thread quantity of each service node is configured, and before the task allocation for the data to be synchronized based on the Disruptor Consumer thread quantity, the method further includes: encapsulating the data to be synchronized according to the data type information of the data to be synchronized and the corresponding subscription target system information, to form a data set to be synchronized in the form of a JSON string message, and the encapsulated synchronization data set at least includes one or more of the data id, the original data message, the target system information and the synchronization mode of the data to be synchronized; and the synchronization data set is used as the target data to be synchronized to support the task allocation for the data to be synchronized. In this embodiment of the present application, the data to be synchronized is encapsulated to form a data set to be synchronized in the form of a JSON string message, and then is sent to the high-performance queue for synchronization sending processing, so that the synchronization sending processing of the data to be synchronized can be more convenient without the type identification and other pre-analysis processing of the data to be synchronized in this stage, which is left to the Disruptor high-performance in the subsequent steps, thereby improving the efficiency of the task allocation for the data to be synchronized in this stage.

[0052] Specifically, in the embodiment of the present application, the implementation process of encapsulating the data to be synchronized according to the data type information of the data to be synchronized and the corresponding subscription target system information can be performed in the callback function ResultHandler <t>In the callback function, each piece of obtained to-be-synchronized data is processed to realize encapsulation of the data. In the embodiment of the present application, the use of the ResultHandler can greatly improve the efficiency of data processing. When a large amount of data needs to be processed, the ResultHandler is used to process the query result, thereby avoiding returning all the query results or the encapsulated results at one time, reducing the probability of occurrence of the waste of memory resources or the problem of memory overflow and OMM crash.

[0053] Correspondingly, in the optional implementation manner of the embodiment of the present application, the service node is used to send the assigned synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner, including: using an asynchronous processing function to send the to-be-synchronized data set in the form of a JSON string message to the Disruptor memory queue of the corresponding service node in a concurrent manner, so as to use the service node to send the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous parallel manner.

[0054] Optionally, in an embodiment of the present application, the use of the service node to send the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous parallel manner includes: identifying the JSON string message of the to-be-synchronized data set from the Disruptor memory queue through a DisruptorConsumer consumer, and parsing the data id, original message data, target system address and synchronization mode in the JSON string message; according to the parsing result, the target system corresponding to the original message data is determined according to the target system address, so that the specific execution process of data synchronization is performed by the Disruptor high-performance memory message queue, to improve the efficiency of synchronization sending while ensuring the accuracy and reliability of the synchronization sending result.

[0055] Optionally, in an embodiment of the present application, the method further includes: configuring the consumption waiting strategy of the Disruptor memory queue as a blocking waiting strategy (BlockingWaitStrategy). BlockingWaitStrategy is a commonly used waiting strategy in the Disruptor framework, and is mainly used to determine how the consumer waits for the producer to put the event object into the Disruptor queue.

[0056] The BlockingWaitStrategy uses a ReentrantLock and Condition variable to implement the blocking logic, ensuring that during event processing, the consumer can correctly wait until an event is available. It uses a ReentrantLock and Condition to implement the wait logic. When the consumer tries to get an event, if the event is not available, the consumer is blocked until the producer puts an event in the queue and notifies the consumer. This mechanism ensures thread safety, which makes the thread be suspended when waiting, instead of continuously occupying CPU resources, thus reducing the CPU occupancy of the system.

[0057] Optionally, in an implementation form of the embodiment of the application, the method further comprises periodically determining the target system information of the subscription data and the service node information of the data synchronization sending according to a preset period, so as to reduce the working strength of the automatic system performing the method while ensuring the efficiency of the data synchronization sending.

[0058] The application provides a data synchronization sending method. The target system information of the subscription data and the service node information of the data synchronization sending are determined. The service node information at least includes the number of service nodes and the number information of each service node. The target system information includes one or more of the target system ID, the target system name, the target system address, and the target subscription data type. Based on the service node information, the to-be-synchronized data that needs to be synchronized and sent by each service node is determined. The number of Disruptor Consumer consumption threads of each service node is configured, and the to-be-synchronized data is task allocated based on the number of Disruptor Consumer consumption threads. The to-be-synchronized data is sent to the target system corresponding to the target system information by using the service node in an asynchronous parallel manner. This data synchronization sending method, in the case of limited hardware resources and inability to add message queues and other external middleware, supports efficiently synchronizing a large amount of data to a business system that has subscribed to the data in a friendly manner by using high-performance memory queue Disruptor, streaming query, asynchronous parallel processing, and other technologies, and occupies less memory and cpu resources. In this way, the data synchronization business can be efficiently and quickly completed while saving resources.

[0059] Embodiment two,

[0060] Based on the data synchronization sending method provided in the first embodiment of the application, the application further provides a data synchronization sending device, as shown in Figure 2 the application further provides a data synchronization sending device, as shown in Figure 2 A structural diagram of a data synchronization sending device 20 provided in an embodiment of the present application is shown in FIG. 1. The data synchronization sending device 20 comprises:

[0061] A determining module 201 is configured to determine target system information of subscription data and service node information for data synchronization sending, wherein the service node information at least comprises the number of service nodes and the number information of each service node; and the target system information comprises one or more of the target system ID, the target system name, the target system address and the target subscription data type.

[0062] A querying module 202 is configured to determine the to-be-synchronized data that needs to be synchronized and sent by each service node based on the service node information.

[0063] An allocating module 203 is configured to configure the number of Disruptor Consumer consumption threads of each service node, and perform task allocation on the to-be-synchronized data based on the number of Disruptor Consumer consumption threads.

[0064] A synchronizing module 204 is configured to use the service nodes to send the allocated synchronization sending tasks to the target system corresponding to the target system information in an asynchronous and parallel manner.

[0065] Optionally, in an implementation manner of the embodiment of the present application, the querying module 202 is further configured to execute the following SQL statement to implement batch calculation on the to-be-synchronized data:

[0066] select * from task_ data_sync where id % [service node number] = [node number],

[0067] The to-be-synchronized data is allocated to the service node with the node number determined by the batch calculation through the modulo operation on the value of the task number of the to-be-synchronized data, and each service node is configured to process different to-be-synchronized data.

[0068] Optionally, in an embodiment of the present application, the querying module 202 is further configured to determine the to-be-synchronized data that needs to be synchronized and sent by each service node based on the service node corresponding to the service node information by using a streaming query manner.

[0069] Optionally, in an embodiment of the present application, the query module 202 is further configured to: when performing the batch calculation of the to-be-synchronized data by executing the SQL statement, adding a configuration of ResultSetType.FORWARD_ONLY in a Mapper interface method of a Mybatis query to-be-synchronized data sql, to start a streaming query mode, for determining the to-be-synchronized data that needs to be synchronized and sent by each service node.

[0070] Optionally, in an embodiment of the present application, the device 20 further comprises a packaging module (not shown in the figure), configured to: before the task allocation of the to-be-synchronized data based on the Disruptor Consumer thread quantity, according to the data type information of the to-be-synchronized data and the corresponding subscription target system information, packaging the to-be-synchronized data to form a to-be-synchronized data set in the form of a JSON string message, the synchronized data set after packaging at least including one or more of the data id, the original data message, the target system information and the synchronization mode of the to-be-synchronized data; taking the synchronized data set as the target to-be-synchronized data, for supporting the task allocation of the to-be-synchronized data.

[0071] Optionally, in an embodiment of the present application, the synchronization module 204 is further configured to: using an asynchronous processing function, concurrently sending the to-be-synchronized data set in the form of a JSON string message to the Disruptor in-memory queue of the corresponding service node, to send the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous and parallel manner by using the service node.

[0072] Optionally, in an embodiment of the present application, the synchronization module 204 is further configured to: including: identifying the JSON string message of the to-be-synchronized data set from the Disruptor in-memory queue by a Disruptor Consumer consumer, and parsing the data id, the original message data, the target system address and the synchronization mode in the JSON string message; according to the parsing result, according to the target system address, sending the target system corresponding to the original message data.

[0073] Optionally, in an embodiment of the present application, the device 20 further comprises a configuration module (not shown in the figure), configured to configure the consumption waiting strategy of the Disruptor in-memory queue as a blocking waiting strategy.

[0074] The application provides a data synchronization sending device, a setting determination module determines target system information of subscription data and service node information of data synchronization sending, the service node information at least includes quantity of service nodes and number information of each service node; the target system information includes one or more of target system ID, target system name, target system address and target subscription data type; a setting query module determines to-be-synchronized data which needs to be synchronized by each service node based on the service node information; a setting allocation module configures Disruptor Consumer thread quantity of each service node, and a synchronization module performs task allocation on the to-be-synchronized data based on the Disruptor Consumer thread quantity; and the service nodes send the to-be-synchronized data to target systems corresponding to the target system information in an asynchronous parallel mode. The data synchronization sending device can support efficient synchronization of a large amount of data to a business system which has subscribed to the data in a friendly manner under the condition that hardware resources are limited and external middleware such as a message queue cannot be added, and the data synchronization service can be efficiently and quickly completed while saving resources.

[0075] Embodiment three,

[0076] The application also provides a storage medium having a computer program stored thereon, and the program is executed by a processor to implement any one of the data synchronization sending methods in the first embodiment of the application.

[0077] Embodiment four,

[0078] The application also provides an electronic device, such as Figure 3 as shown, Figure 3 The application provides a structural schematic diagram of an electronic device 30, and the electronic device 30 includes:

[0079] one or more processors 301, a communication interface 302, a memory 303 and a communication bus 304, the processor 301, the memory 303 and the communication interface 302 complete communication with each other through the communication bus 304;

[0080] the memory 303 is used for storing one or more programs;

[0081] When the one or more programs are executed by the one or more processors 301, the one or more processors 301 implement any one of the data synchronization sending methods in the first embodiment of the application.

[0082] To this end, particular embodiments of the present subject matter have been described. In some instances, the actions recited in the claims can be performed in a different order and still achieve desirable results. Additionally, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0083] In the 1990s, it was quite obvious to distinguish whether an improvement in a technology was in hardware (e.g., improvement in circuit structures of diodes, transistors, switches, etc.) or in software (e.g., improvement in method flow). However, as technology has evolved, many improvements in method flow today can be considered as direct improvements in hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structures by programming the improved method flow into hardware circuits. Therefore, it cannot be said that an improvement in a method flow cannot be implemented by hardware entity modules. For example, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)) is an integrated circuit whose logic function is determined by user programming of the device. A designer programs a digital system layer "integrated" on a PLD by himself / herself, without having to ask a chip manufacturer to design and manufacture a special integrated circuit chip. Moreover, instead of manually manufacturing an integrated circuit chip, this programming is now mostly implemented by "logic compiler" software, which is similar to a software compiler used when developing a program, and the original code before compilation also has to be written in a specific programming language, which is called a hardware description language (HDL), and there are many types of HDL, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc., and the most commonly used are VHDL (Very-High-Speed Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also be aware that it is quite easy to obtain a hardware circuit that implements a logical method flow by only logically programming the method flow in the above-mentioned hardware description languages and programming it into an integrated circuit.

[0084] The controller can be implemented in any suitable way, for example, the controller can take the form of a microprocessor or processor and a computer readable medium storing computer readable program code, such as software or firmware, executable by the (micro)processor, logic gates, switches, an application specific integrated circuit (ASIC), a programmable logic controller and an embedded microcontroller, examples of which include but are not limited to the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20 and Silicone Labs C8051F320, the memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that, in addition to implementing the controller in pure computer readable program code, it is also possible to implement the controller in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. to perform the same functions by means of logical programming of the method steps. Such a controller can therefore be considered as a hardware component, and the means included therein for performing various functions can also be considered as structures within the hardware component. Alternatively, the means for performing various functions can even be considered as both a software module implementing the method and a structure within the hardware component.

[0085] The system layers, devices, modules or units illustrated by the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0086] For the sake of description, the above devices are described in various units by functions respectively. Of course, the functions of each unit can be implemented in one or more software and / or hardware in the implementation of the present application.

[0087] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not include only those elements in the list, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0088] Those skilled in the art will appreciate that embodiments of the present application can be readily used as a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0089] The present application can be described in the general context of computer- executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0090] Each of the embodiments described in this specification has been described taking a progressive approach, and the same or similar parts between embodiments can be mutually referred to, and each embodiment focuses on the difference from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0091] The above merely provides embodiments of the present application, but does not serve to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of the claims of the present application.< / t> < / t>

Claims

1. A method of data synchronization transmission, characterized by, The method comprises the following steps: determining target system information of subscription data and service node information of data synchronization sending, wherein the service node information at least comprises the number of service nodes and the number information of each service node; the target system information comprises one or more of target system ID, target system name, target system address and target subscription data type; based on the service node information, determining the to-be-synchronized data that each service node needs to synchronize and send; the step specifically comprises: executing the following SQL statement to realize batch calculation of the to-be-synchronized data: select * from task_data_sync where id % [number of service nodes] = [number of nodes]; by taking the modulus of the value of the task number of the to-be-synchronized data, the to-be-synchronized data is allocated to the service node with the number of nodes determined by the batch calculation, and each service node is configured to process different to-be-synchronized data; and based on the service node corresponding to the service node information, the to-be-synchronized data that each service node needs to synchronize and send is determined by using a streaming query mode; configuring the number of Disruptor Consumer consumer threads of each service node, and based on the number of Disruptor Consumer consumer threads, allocating tasks to the to-be-synchronized data; using the service node to send the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous and parallel manner.

2. The method of data synchronization transmission according to claim 1, wherein, The method for determining the to-be-synchronized data that each service node needs to synchronize and send based on the service node corresponding to the service node information by using a streaming query mode comprises: when executing the SQL statement to realize batch calculation of the to-be-synchronized data, adding configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of the Mybatis query to-be-synchronized data sql to start the streaming query mode, which is used to determine the to-be-synchronized data that each service node needs to synchronize and send.

3. The method of data synchronization transmission according to claim 1, characterized in that, Before the method of configuring the number of Disruptor Consumer consumer threads of each service node and allocating tasks to the to-be-synchronized data based on the number of Disruptor Consumer consumer threads, the method further comprises: according to the data type information of the to-be-synchronized data and the corresponding subscription target system information, encapsulating the to-be-synchronized data to form a to-be-synchronized data set in the form of a JSON string message, wherein the encapsulated synchronization data set at least comprises one or more of the data id of the to-be-synchronized data, the original data message, the target system information and the synchronization mode; the synchronization data set is used as target to-be-synchronized data to support task allocation of the to-be-synchronized data.

4. The method of data synchronization transmission according to claim 3, wherein, The method for using the service node to send the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous and parallel manner comprises: The asynchronous processing function is used to send the JSON string message form of the to-be-synchronized data set to the Disruptor memory queue of the corresponding service node in a concurrent manner, so that the service node sends the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous and parallel manner.

5. The method of data synchronization transmission according to claim 4, characterized in that The method further comprises: configuring the consumption waiting strategy of the Disruptor memory queue as a blocking waiting strategy. The method further comprises: The determining module is configured to determine target system information of subscribed data and service node information of data synchronization sending, wherein the service node information at least comprises the number of service nodes and the number information of each service node; and the target system information comprises one or more of target system ID, target system name, target system address, and target subscription data type.

6. The method of data synchronization transmission of claim 1, wherein, The query module is configured to determine to-be-synchronized data to be synchronized and sent by each service node based on the service node information.

7. An apparatus for data synchronization transmission, the apparatus comprising: The allocation module is configured to configure the number of Disruptor Consumer consumption threads of each service node, and perform task allocation on the to-be-synchronized data based on the number of Disruptor Consumer consumption threads. The synchronization module is configured to use the service node to send the to-be-synchronized data to the target system corresponding to the target system information in an asynchronous and parallel manner. The query module is further configured to execute the following SQL statement to implement batch calculation on the to-be-synchronized data: The computer storage medium stores computer executable instructions, and the computer executable instructions are executed to perform the method for data synchronization sending according to any one of claims 1-6. The computer storage medium stores computer executable instructions, and the computer executable instructions are executed to perform the method for data synchronization sending according to any one of claims 1-6. ​ ​ 8. A computer storage medium, characterized in that, ​

Images