Method and device for synchronously sending data and related equipment

Abstract

The invention provides a data synchronous sending method and device and related equipment, and the method comprises the steps: determining target system information of subscription data and service node information for carrying out the data synchronous sending, and determining to-be-synchronized data needing to be synchronously sent by each service node based on the service node information; configuring the number of Disruption Consumer consumption threads of each service node, and performing task allocation on the to-be-synchronized data based on the number of the consumption threads; and sending the distributed synchronous sending task to a corresponding target system in an asynchronous parallel mode by using the service node. Under the conditions that hardware resources are limited and external middleware such as a message queue cannot be newly added, through the technologies of high-performance memory queue Disruptor, streaming query, asynchronous parallel processing and the like, large-batch data can be efficiently synchronized to a service system of subscribed data in a friendly mode under the condition that occupied memory and cpu resources are small, resources are saved, and meanwhile the service efficiency is improved. And the data synchronization service can be efficiently and quickly completed.

Description

Technical Field

[0001] The present application relates to the field of computer information processing technology, and in particular to a method, device and related equipment for synchronous data transmission. Background Art

[0002] With the development of informatization, the network services provided in the society are gradually developing in the direction of microservices, distributed clusters, data integration platforms, big data platforms and other large-scale system architectures as the mainstream technology development direction. The application and deployment of these technologies require more and more resources, and the amount of data that needs to be processed is also becoming larger and more complex. For systems with limited server resources, the task of synchronously sending these large amounts of data with different complexities is that it is difficult to ensure that the massive data is sent to the target system that subscribes to these data in a timely and accurate manner. In the face of this situation of synchronously sending data with massive amounts of data with different complexities, the industry currently mostly synchronizes data to multiple subscription information systems by adding middleware. This method requires a certain degree of re-editing of the system's communication method, and the efficiency of synchronous processing of large and difficult-to-format data is also relatively general. Therefore, how to provide an efficient data synchronization method that does not rely on middleware technology has become an important problem that needs to be solved in the industry. Summary of the invention

[0003] In view of this, an embodiment of the present application provides a method, an apparatus and related equipment for synchronously sending data to at least or partially solve the above-mentioned problems.

[0004] In a first aspect, an embodiment of the present application provides a method for synchronously sending data, including:

[0005] Determine the target system information for subscribing data and the service node information for data synchronization.

[0006] [HZTC-HXRJGS241001][HS2411340CCN] The service node information at least includes the number of service nodes and the numbering information of each service node; the target system information includes one or more of the target system ID, target system name, target system address, and target subscription data type;

[0007] Based on the service node information, determine the to-be-synchronized data that each service node needs to send synchronously;

[0008] Configure the number of Disruptor Consumer consumption threads of each service node, and assign tasks to the data to be synchronized based on the number of DisruptorConsumer consumption threads;

[0009] The service node is used to send the allocated synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner.

[0010] Optionally, in an embodiment of the present application, determining the to-be-synchronized data that each service node needs to send synchronously based on the service node information includes:

[0011] Execute the following SQL statement to implement batch calculation of synchronized data:

[0012] select * from task_data_sync where id%[number of service nodes] = [node number],

[0013] By performing a modulo operation on the value of the task quantity of the data to be synchronized, the data to be synchronized is distributed to the service nodes with the node numbers determined by batch calculation, and each service node is configured to process different data to be synchronized.

[0014] Optionally, in an embodiment of the present application, determining the to-be-synchronized data that each service node needs to send synchronously based on the service node information includes:

[0015] Based on the service nodes corresponding to the service node information, the to-be-synchronized data that needs to be synchronously sent by each service node is determined by using a streaming query method.

[0016] Optionally, in an embodiment of the present application, the service node corresponding to the service node information determines the to-be-synchronized data that each service node needs to send synchronously by using a streaming query method, including:

[0017] When executing the SQL statement to realize batch calculation of the data to be synchronized, add the configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of using Mybatis to query the sql of the data to be synchronized to enable the streaming query mode, which is used to determine the

[0018] [HZTC-HXRJGS241001][HS2411340CCN] The nodes need to send data to be synchronized.

[0019] Optionally, in an embodiment of the present application, the number of Disruptor Consumer consumption threads of each service node is configured, and before allocating tasks for the data to be synchronized based on the number of Disruptor Consumer consumption threads, the method further includes:

[0020] According to the data type information of the data to be synchronized and the corresponding subscription target system information, the data to be synchronized is encapsulated to form a data set to be synchronized in the form of a JSON string message, wherein the encapsulated synchronization data set includes at least one or more of the data id of the data to be synchronized, the original data message, the target system information and the synchronization mode;

[0021] The synchronization data set is used as target data to be synchronized, so as to support task allocation for the data to be synchronized.

[0022] Optionally, in an embodiment of the present application, using the service node to send the assigned synchronization sending task to a target system corresponding to the target system information in an asynchronous parallel manner includes:

[0023] Using an asynchronous processing function, the data set to be synchronized in the form of a JSON string message is sent concurrently to the Disruptor memory queue of the corresponding service node, so that the service node can use the data to be synchronized to send the target system of the corresponding target system information in an asynchronous and parallel manner.

[0024] Optionally, in an embodiment of the present application, using the service node to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous and parallel manner includes:

[0025] Identify the json string message of the data set to be synchronized from the Disruptor memory queue through the DisruptorConsumer consumer, and parse the data id, original message data, target system address and synchronization mode in the json string message;

[0026] According to the analysis result, the target system corresponding to the original message data is determined according to the target system address.

[0027] Optionally, in an embodiment of the present application, the method further includes: configuring the consumption waiting strategy of the Disruptor memory queue to be a blocking waiting strategy.

[0028] [HZTC-HXRJGS241001][HS2411340CCN]

[0029] In a second aspect, based on the method for synchronously sending data described in the first aspect of the present application, an embodiment of the present application further provides a device for synchronously sending data, including:

[0030] A determination module, used to determine target system information for subscribing data and service node information for synchronously sending data, wherein the service node information at least includes the number of service nodes and the serial number information of each service node; the target system information includes one or more of a target system ID, a target system name, a target system address, and a target subscription data type;

[0031] A query module, used to determine the to-be-synchronized data that each service node needs to send synchronously based on the service node information;

[0032] An allocation module is used to configure the number of Disruptor Consumer consumption threads of each service node, and to allocate tasks for data to be synchronized based on the number of Disruptor Consumer consumption threads;

[0033] The synchronization module is used to utilize the service node to send the assigned synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner.

[0034] In a third aspect, an embodiment of the present application further provides a computer storage medium, on which computer executable instructions are stored, and when the computer executable instructions are executed, any one of the methods for synchronously sending data as described in the first aspect of the embodiment of the present application is performed.

[0035] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus;

[0036] The memory is used to store at least one executable instruction, and the executable instruction enables the processor to execute any one of the methods for synchronously sending data as described in the first aspect of the embodiments of the present application.

[0037] The present application provides a method, apparatus and related equipment for synchronous data transmission, which determines the target system information of the subscription data and the service node information for synchronous data transmission, wherein the service node information at least includes the number of service nodes and the numbering information of each service node; the target system information includes one or more of the target system ID, the target system name, the target system address and the target subscription data type; based on the service node information, determines the data to be synchronized that each service node needs to send synchronously; configures the number of Disruptor Consumer consumption threads of each service node, based on the

[0038] [HZTC-HXRJGS241001][HS2411340CCN]Disruptor Consumer consumes the number of threads and performs task allocation for the data to be synchronized; using the service node, the assigned synchronization sending task is sent to the target system corresponding to the target system information in an asynchronous and parallel manner. This data synchronization sending method, in the case of limited hardware resources and inability to add external middleware such as message queues, uses high-performance memory queue Disruptor, streaming query, asynchronous parallel processing and other technologies, while occupying less memory and CPU resources. It can friendly support the efficient synchronization of large batches of data to the business system that has subscribed to the data, while saving resources, and can also efficiently and quickly complete the data synchronization business. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0040] Figure 1 A schematic diagram of a workflow of a method for synchronously sending data provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the structure of a device for synchronously sending data provided in an embodiment of the present application.

[0042] Figure 3 A schematic structural diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the embodiments of the present application should fall within the scope of protection of the embodiments of the present application.

[0044] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0045] Embodiment 1

[0046] The present application provides a method for synchronously sending data, such as Figure 1 As shown, Figure 1 For this

[0047] [HZTC-HXRJGS241001][HS2411340CCN] Please provide a workflow diagram of a method for synchronously sending data provided in the embodiment, including:

[0048] Step S101, determine the target system information of the subscription data and the service node information for synchronous data transmission. In the embodiment of the present application, the service node is the currently available system server, wherein the service node information at least includes the number of service nodes and the numbering information of each service node; the target system information includes one or more of the target system ID, target system name, target system address, and target subscription data type. In this stage of the embodiment of the present application, the target information to be synchronously sent and the node information supporting synchronous sending are first determined, so as to be able to more accurately support the process of synchronous data transmission.

[0049] Step S102, based on the service node information, determine the data to be synchronized that each service node needs to send synchronously. In this stage, the embodiment of the present application performs targeted preliminary allocation of synchronization data through the obtained service node information, so as to timely utilize multiple available nodes to process a large number of data synchronization sending tasks, thereby improving the efficiency of data synchronization. The embodiment of the present application here exemplifies this process: for example, when it is necessary to process multiple data synchronization tasks, it is determined that the currently available service nodes are nodes numbered 0, 1, and 2, then these multiple data synchronization tasks are allocated to nodes 0, 1, and 2 according to the numbers to complete the preliminary allocation of the data to be synchronized. This processing method can more effectively use the limited servers of the system as nodes to perform data synchronization transmission, thereby improving the system's ability to solve a large number of data synchronization bursts.

[0050] Step S103, configure the number of Disruptor Consumer consumer threads of each service node, and assign tasks to the synchronized data based on the number of Disruptor Consumer consumer threads. Disruptor is a high-performance message queue developed by LMAX, which is mainly used to solve the latency problem of memory queues. Disruptor can be used as a high-performance bounded memory queue, which is very suitable for [producer-consumer] scenarios and [publish-subscribe] scenarios. Among them, producer-consumer scenario. The most commonly used scenario [producer-consumer] refers to the information scenario of the memory queue as "one producer, multiple consumers", and requires sequential processing, and the data synchronization sending process described in the embodiment of the present application belongs to the typical [producer-consumer] scenario, and also belongs to the [publish-subscribe] scenario. The process of synchronously sending data through the Disruptor queue in the embodiment of the present application can significantly improve the efficiency of data synchronization, and can also more friendly face the work requirements of synchronously sending a large amount of data bursts in the system. In the embodiment of the present application, when configuring the number of Disruptor Consumer consumer threads of each service node, since the number of Disruptor consumer threads is usually managed by an external thread pool, not

[0051] [HZTC-HXRJGS241001][HS2411340CCN] is directly set up inside the Disruptor framework, so that users can configure the size of the external thread pool according to actual needs to better optimize the performance and resource utilization of the Disruptor high-performance queue, thereby further improving the efficiency of the data synchronization transmission described in this application.

[0052] Optionally, in an embodiment of the present application, based on the service node information, determining the data to be synchronized that each service node needs to send synchronously includes: executing the following SQL statement to implement batch calculation of the data to be synchronized:

[0053] select*from task_data_sync where id%[number of service nodes]=[number of nodes], by performing a modulo operation on the value of the number of tasks of the data to be synchronized, the data to be synchronized is distributed to the service nodes of the node numbers determined by batch calculation, and each service node is configured to process different data to be synchronized. In this stage of the embodiment of the present application, through the above method and modulo operation, the same piece of data to be synchronized will only be processed by the same service node, and the data to be synchronized processed by each service node is evenly distributed, and the operation of the system is more balanced. Different service nodes are further configured to process different data to be processed, so as to make more reasonable use of resources and further ensure the processing efficiency of the tasks to be synchronized.

[0054] Optionally, in an optional implementation method of the embodiment of the present application, based on the service node information, determining the data to be synchronized that each service node needs to send synchronously includes: based on the service node corresponding to the service node information, using a streaming query method to determine the data to be synchronized that each service node needs to send synchronously. In this embodiment, the data to be synchronized that each service node needs to send synchronously is determined by a streaming query method, which can effectively solve the problem of system program memory overflow OMM crash due to excessive amount of data to be synchronized, so as to ensure that the system has good working stability and work efficiency.

[0055] Optionally, in one embodiment of the present application, the service node corresponding to the service node information uses a streaming query method to determine the data to be synchronized that each service node needs to send synchronously, including: when executing the SQL statement to implement batch calculation of the data to be synchronized, add the configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of using Mybatis to query the sql data to be synchronized to enable the streaming query method, which is used to determine the data to be synchronized that each service node needs to send synchronously. ResultSetType.FORWARD_ONLY means that the cursor of the result set can only move forward and does not support scrolling. Since the result set can only be traversed forward, the nature

[0056] [HZTC-HXRJGS241001][HS2411340CCN] can perform well and is suitable for service applications in query scenarios that require processing of large amounts of data. In this stage of the embodiment of the present application, a large amount of data is processed by adding the configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of the Mybatis querying the sql data to be synchronized, thereby reducing the operations of complex scenarios such as data rollback or random access to the queried target data, i.e., the result set, and achieving better query performance. Correspondingly, the method also includes: adding the configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of the Mybatis querying the sql data to be synchronized to enable the streaming query mode, and the method may also include: configuring ResultHandler <t>The parameters are passed into the callback function to receive the result set of the processing streaming query. This allows the method described in the embodiment of the present application to meet more application scenarios, such as the need to operate the query results and data rollback or random access, etc., to improve the scenario applicability of the method of synchronous data transmission described in the embodiment of the present application, and in this way, the process of querying and obtaining the synchronized data is also relatively simple, which is convenient for programmers to develop and maintain the system.

[0057] Step S104, using the service node, the assigned synchronous sending task is sent to the target system of the corresponding target system information in an asynchronous and parallel manner. The embodiment of the present application uses the service node in an asynchronous and parallel manner, specifically using the number of Disruptor Consumer consumer threads configured in the service node to send data synchronously, so as to more fully utilize the high-performance message queue of Disruptor to support the synchronous sending of data, so as to stably realize the efficient synchronization business needs of large batches of data while occupying fewer system resources, and improve the efficiency of data synchronization sending.

[0058] Optionally, in one embodiment of the present application, the number of Disruptor Consumer consumption threads of each service node is configured, and based on the number of Disruptor Consumer consumption threads, before allocating tasks to the data to be synchronized, the method further includes: encapsulating the data to be synchronized according to the data type information of the data to be synchronized and the corresponding subscription target system information to form a data set to be synchronized in the form of a JSON string message, wherein the encapsulated synchronization data set includes at least one or more of the data id of the data to be synchronized, the original data message, the target system information and the synchronization method; and using the synchronization data set as the target data to be synchronized to support the task allocation to the data to be synchronized. At this stage, the embodiment of the present application can be more convenient by encapsulating the data to be synchronized to form a data set to be synchronized in the form of a JSON string message, and then sending it to a high-performance queue for synchronous sending processing.

[0059] [HZTC-HXRJGS241001][HS2411340CCN] can conveniently send and process the data to be synchronized without the need for pre-analysis such as type identification of the data to be synchronized at this stage. The Disruptor high performance can be used for subsequent steps to improve the efficiency of task allocation for the data to be synchronized at this stage.

[0060] Specifically, in the embodiment of the present application, according to the data type information of the data to be synchronized and the corresponding subscription target system information, the implementation process of encapsulating the data to be synchronized can be in the callback function ResultHandler <t>In the callback function, each piece of data to be synchronized is processed to realize data encapsulation. In the embodiment of the present application, the use of ResultHandler can greatly improve the efficiency of data processing. When a large amount of data needs to be processed, ResultHandler is used to process the query results to avoid returning all the results of a query or encapsulation, so as to reduce the probability of wasting memory resources or avoid the problem of memory overflow OMM crash.

[0061] Correspondingly, in an optional implementation method of the embodiment of the present application, the service node is used to send the assigned synchronization sending task to the target system corresponding to the target system information in an asynchronous and parallel manner, including: using an asynchronous processing function to concurrently send the data set to be synchronized in the form of a JSON string message to the Disruptor memory queue of the corresponding service node, so as to use the service node to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous and parallel manner.

[0062] Optionally, in an embodiment of the present application, the use of the service node to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous and parallel manner includes: identifying the json string message of the data set to be synchronized from the Disruptor memory queue through the DisruptorConsumer consumer, parsing the data id, original message data, target system address and synchronization method in the json string message; according to the parsing result, according to the target system address, the target system corresponding to the original message data. The specific execution process of data synchronization is performed by the Disruptor high-performance memory message queue to improve the efficiency of synchronous transmission while ensuring the accuracy and reliability of the results of synchronous transmission.

[0063] Optionally, in an embodiment of the present application, the method further comprises: configuring the consumption waiting strategy of the Disruptor memory queue to be a blocking waiting strategy (BlockingWaitStrategy). BlockingWaitStrategy is a commonly used waiting strategy in the Disruptor framework, which is mainly used to determine how consumers wait for producers to put event objects into the Disruptor queue.

[0064] [HZTC-HXRJGS241001][HS2411340CCN]BlockingWaitStrategy uses ReentrantLock and Condition variables to implement blocking logic, ensuring that during event processing, consumers can correctly wait until events are available. It implements waiting logic by using reentrant locks (ReentrantLock) and condition variables (Condition). When a consumer tries to get an event, if the event is not available, the consumer will be blocked until the producer puts the event into the queue and notifies the consumer. This mechanism ensures thread safety, which allows the thread to be suspended while waiting instead of continuously occupying CPU resources, thereby reducing the system CPU occupancy rate.

[0065] Optionally, in an implementation method of an embodiment of the present application, the target system information of the subscription data and the service node information for synchronously sending the data are determined, including: according to a preset period, periodically executing the technical steps of determining the target system information of the subscription data and the service node information for synchronously sending the data, thereby reducing the workload of the automation system that executes the method described in the embodiment of the present application while ensuring the efficiency of synchronous sending of data.

[0066] The present application provides a method for synchronous data transmission, by determining the target system information of the subscribed data and the service node information for synchronous data transmission, the service node information at least includes the number of service nodes and the number information of each service node; the target system information includes one or more of the target system ID, the target system name, the target system address, and the target subscription data type; based on the service node information, determine the data to be synchronized that each service node needs to send synchronously; configure the number of Disruptor Consumer consumption threads of each service node, and assign tasks to the data to be synchronized based on the number of DisruptorConsumer consumption threads; use the service node to send the assigned synchronous transmission tasks to the target system corresponding to the target system information in an asynchronous and parallel manner. This data synchronization transmission method, in the case of limited hardware resources and inability to add external middleware such as message queues, uses high-performance memory queue Disruptor, streaming query, asynchronous parallel processing and other technologies, and in the case of small memory and CPU resource occupation, friendly supports efficient synchronization of large batches of data to the business system of the subscribed data, while saving resources, and can also efficiently and quickly complete data synchronization services.

[0067] Embodiment 2

[0068] Based on the method for synchronously sending data provided in the first embodiment of the present application, the embodiment of the present application also provides a device for synchronously sending data, such as Figure 2 As shown, Figure 2 [HZTC-HXRJGS241001][HS2411340CCN] provided in the embodiment of the present application is a structural diagram of a device 20 for synchronously sending data, and the device 20 for synchronously sending data includes:

[0069] The determination module 201 is used to determine the target system information of the subscription data and the service node information for data synchronization transmission, wherein the service node information at least includes the number of service nodes and the serial number information of each service node; the target system information includes one or more of the target system ID, the target system name, the target system address, and the target subscription data type;

[0070] A query module 202, configured to determine the to-be-synchronized data that each service node needs to send synchronously based on the service node information;

[0071] An allocation module 203 is used to configure the number of Disruptor Consumer consumption threads of each service node, and to allocate tasks for data to be synchronized based on the number of Disruptor Consumer consumption threads;

[0072] The synchronization module 204 is used to use the service node to send the assigned synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner.

[0073] Optionally, in an implementation of the embodiment of the present application, the query module 202 is further used to execute the following SQL statement to implement batch calculation of the data to be synchronized:

[0074] select * from task_data_sync where id%[number of service nodes] = [node number],

[0075] By performing a modulo operation on the value of the task quantity of the data to be synchronized, the data to be synchronized is distributed to the service nodes with the node numbers determined by batch calculation, and each service node is configured to process different data to be synchronized.

[0076] Optionally, in an embodiment of the present application, the query module 202 is further used to: determine the to-be-synchronized data that each service node needs to send synchronously by using a streaming query method based on the service node corresponding to the service node information.

[0077] Optionally, in one embodiment of the present application, the query module 202 is also used for: when executing the SQL statement to implement batch calculation of the data to be synchronized, adding the configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of using Mybatis to query the sql of the data to be synchronized, so as to enable the streaming query mode, which is used to determine the data to be synchronized that each service node needs to send synchronously.

[0078] [HZTC-HXRJGS241001][HS2411340CCN]

[0079] Optionally, in one embodiment of the present application, the device 20 further includes: an encapsulation module (not shown in the accompanying drawings), which is used to configure the number of Disruptor Consumer consumer threads of each service node, and based on the number of Disruptor Consumer consumer threads, before allocating tasks to the data to be synchronized, encapsulate the data to be synchronized according to the data type information of the data to be synchronized and the corresponding subscription target system information to form a data set to be synchronized in the form of a JSON string message, wherein the encapsulated synchronization data set includes at least one or more of the data id of the data to be synchronized, the original data message, the target system information and the synchronization mode; and use the synchronization data set as the target data to be synchronized to support the task allocation of the data to be synchronized.

[0080] Optionally, in one embodiment of the present application, the synchronization module 204 is also used to: use an asynchronous processing function to concurrently send the data set to be synchronized in the form of a JSON string message to the Disruptor memory queue of the corresponding service node, so as to utilize the service node to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous and parallel manner.

[0081] Optionally, in one embodiment of the present application, the synchronization module 204 is also used for: including: identifying the json string message of the data set to be synchronized from the Disruptor memory queue through the DisruptorConsumer consumer, parsing the data id, original message data, target system address and synchronization method in the json string message; according to the parsing result, according to the target system address, the target system corresponding to the original message data.

[0082] Optionally, in an embodiment of the present application, the device 20 further includes a configuration module (not shown in the drawings), which is used to configure the consumption waiting strategy of the Disruptor memory queue to be a blocking waiting strategy.

[0083] The present application provides a device for synchronously sending data, wherein a determination module is set to determine target system information of subscription data and service node information for synchronously sending data, wherein the service node information at least includes the number of service nodes and the numbering information of each service node; the target system information includes one or more of a target system ID, a target system name, a target system address, and a target subscription data type; a query module is set to determine the to-be-synchronized data that each service node needs to synchronously send based on the service node information; an allocation module is set to configure the number of Disruptor Consumer consumption threads of each service node, and a synchronization module is set to treat the same

[0084] [HZTC-HXRJGS241001][HS2411340CCN] step data for task allocation; using the service node, the assigned synchronization sending task is sent to the target system of the corresponding target system information in an asynchronous parallel manner. This data synchronization sending device, in the case of limited hardware resources and inability to add external middleware such as message queues, uses high-performance memory queue Disruptor, streaming query, asynchronous parallel processing and other technologies, and in the case of small memory and CPU resource usage, it can friendly support the efficient synchronization of large batches of data to the business system that has subscribed to the data, while saving resources, and can also efficiently and quickly complete the data synchronization business.

[0085] Embodiment 3

[0086] An embodiment of the present application also provides a storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods for synchronously sending data as described in the first embodiment of the present application is implemented.

[0087] Embodiment 4:

[0088] The present application also provides an electronic device, such as Figure 3 As shown, Figure 3 A schematic diagram of the structure of an electronic device 30 is provided for an embodiment of the present application. The electronic device 30 includes:

[0089] One or more processors 301, communication interface 302, memory 303 and communication bus 304, the processor 301, memory 303 and communication interface 302 communicate with each other via the communication bus 304;

[0090] Memory 303, used to store one or more programs;

[0091] When the one or more programs are executed by the one or more processors 301, the one or more processors 301 implement any one of the methods for synchronously sending data as described in Embodiment 1 of the present application.

[0092] So far, the application has described specific embodiments of the subject matter. In some cases, the actions recorded in the claims can be performed in different orders and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0093] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to circuit structures such as diodes, transistors, switches, etc.) or software improvements (improvements to method flows). However, with the development of technology, many of today's method flows are

[0094] The improvement of [HZTC-HXRJGS241001][HS2411340CCN] can be regarded as a direct improvement of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be realized by hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user programming the device. Designers can "integrate" a digital system layer on a PLD by programming themselves, without having to ask chip manufacturers to design and produce dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs. The original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0095] The controller may be implemented in any suitable manner. For example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller may also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code manner, it is entirely possible to implement the controller in a manner that is not limited to a computer-readable program code.

[0096] [HZTC-HXRJGS241001][HS2411340CCN] The method steps are logically programmed to enable the controller to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0097] The system layers, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0098] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0099] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the existence of other identical elements in the process, method, commodity or device including the element.

[0100] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, system layers or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0101] The present application may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. In general, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where remote computers connected by a communication network are connected to each other.

[0102] [HZTC-HXRJGS241001][HS2411340CCN]processing equipment to execute transactions. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0103] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system layer embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0104] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.< / t> < / t>

Claims

1. A method for synchronously sending data, characterized in that: include: Determine the target system information of the subscription data and the service node information for data synchronization transmission, wherein the service node information at least includes the number of service nodes and the serial number information of each service node; the target system information includes one or more of the target system ID, target system name, target system address, and target subscription data type; Based on the service node information, determine the to-be-synchronized data that each service node needs to send synchronously; Configure the number of Disruptor Consumer consumption threads of each service node, and assign tasks to the synchronized data based on the number of Disruptor Consumer consumption threads; The service node is used to send the allocated synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner.

2. The method for synchronously sending data according to claim 1, characterized in that: The determining, based on the service node information, the data to be synchronized that each service node needs to send synchronously includes: executing the following SQL statement to implement batch calculation of the data to be synchronized: select * from task_data_sync where id%[number of service nodes] = [node number], By performing a modulo operation on the value of the task quantity of the data to be synchronized, the data to be synchronized is distributed to the service nodes with the node numbers determined by batch calculation, and each service node is configured to process different data to be synchronized.

3. The method for synchronously sending data according to claim 2, characterized in that: The determining, based on the service node information, the to-be-synchronized data that each service node needs to synchronously send includes: Based on the service nodes corresponding to the service node information, the to-be-synchronized data that needs to be synchronously sent by each service node is determined by using a streaming query method.

4. The method for synchronously transmitting data according to claim 3, characterized in that: The service node corresponding to the service node information determines the data to be synchronized that each service node needs to send synchronously by using a streaming query method, including: When executing the SQL statement to realize batch calculation of the data to be synchronized, add the configuration ResultSetType.FORWARD_ONLY in the Mapper interface method of querying the sql of the data to be synchronized by Mybatis to enable the streaming query mode, which is used to determine the data to be synchronized that each service node needs to send synchronously.

5. The method for synchronously transmitting data according to claim 1, characterized in that: The method further comprises configuring the number of Disruptor Consumer consumption threads of each service node, and before allocating tasks for the synchronization data based on the number of Disruptor Consumer consumption threads: According to the data type information of the data to be synchronized and the corresponding subscription target system information, the data to be synchronized is encapsulated to form a data set to be synchronized in the form of a JSON string message, wherein the encapsulated synchronization data set includes at least one or more of the data id of the data to be synchronized, the original data message, the target system information and the synchronization mode; The synchronization data set is used as target data to be synchronized, so as to support task allocation for the data to be synchronized.

6. The method for synchronously transmitting data according to claim 5, characterized in that: The using the service node to send the assigned synchronization sending task in an asynchronous parallel manner to the target system of the corresponding target system information includes: Using an asynchronous processing function, the data set to be synchronized in the form of a JSON string message is sent concurrently to the Disruptor memory queue of the corresponding service node, so that the service node can use the data to be synchronized to send the target system of the corresponding target system information in an asynchronous and parallel manner.

7. The method for synchronously transmitting data according to claim 6, characterized in that , the method of using the service node to send the data to be synchronized to the target system corresponding to the target system information in an asynchronous and parallel manner includes: Identify the json string message of the data set to be synchronized from the Disruptor memory queue through the DisruptorConsumer consumer, and parse the data id, original message data, target system address and synchronization mode in the json string message; [HZTC-HXRJGS241001][HS2411340CCN] According to the analysis result, according to the target system address, the target system corresponding to the original message data is matched.

8. The method for synchronously transmitting data according to claim 1, characterized in that: The method further includes: configuring the consumption waiting strategy of the Disruptor memory queue to be a blocking waiting strategy.

9. A device for synchronously sending data, characterized in that: include: A determination module, used to determine target system information for subscribing data and service node information for synchronously sending data, wherein the service node information at least includes the number of service nodes and the serial number information of each service node; the target system information includes one or more of a target system ID, a target system name, a target system address, and a target subscription data type; A query module, used to determine the to-be-synchronized data that each service node needs to send synchronously based on the service node information; An allocation module is used to configure the number of Disruptor Consumer consumption threads of each service node, and to allocate tasks for data to be synchronized based on the number of Disruptor Consumer consumption threads; The synchronization module is used to utilize the service node to send the assigned synchronization sending task to the target system corresponding to the target system information in an asynchronous parallel manner.

10. A computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed, the method for synchronously sending data according to any one of claims 1 to 8 is executed.

Images