Publishing and subscribing multi-thread parallel transmission device based on data segmentation

Abstract

The invention provides a publishing and subscribing multi-thread parallel transmission device based on data segmentation aiming at the large-bandwidth communication requirement of mass data, high-speed and real-time distribution of data on a standard Ethernet is completed by adopting strategies such as theme data splitting and parallel transmission and receiving end zero copy, and high-throughput radar information processing is supported. According to the method, a data packet of a theme T is split into n sub-themes to be sent at a publishing end according to the parallelism degree n, parallel receiving is carried out at a receiving end according to the parallelism degree n, and original data packet splicing is completed while data fragments are received.

Description

Technical Field

[0001] The invention belongs to the technical field of data transmission, and in particular relates to a publish-subscribe multi-thread parallel transmission device based on data segmentation. Background Art

[0002] In radar communication systems with large bandwidth communication requirements, when using the single thread of DDS (Data Distribution Service) communication middleware for data transmission and reception, the performance of the communication middleware in single-thread data distribution is insufficient, and its pass rate may not meet the high-performance communication requirements of radar applications. Summary of the invention

[0003] Aiming at the demand for large-bandwidth communication of massive data, the present invention proposes a publish-subscribe multi-threaded parallel transmission device based on data segmentation. It adopts strategies such as topic data splitting and parallel sending, and zero copy at the receiving end to complete high-speed and real-time distribution of data on a standard Ethernet network, and supports high-throughput radar information processing.

[0004] The invention discloses a publish-subscribe multi-thread parallel transmission device based on data segmentation, comprising six component modules, namely: an initialization module (1), a topic creation module (2), a publisher creation module (3), a subscriber creation module (4), a parallel data transmission module (5) and a parallel data reception module (6).

[0005] The following is an example of using these modules to distribute data: Figure 1 As shown. At the data publishing end, the initialization module (1) is used to complete the necessary entity creation, memory application, and parallelism specification. Then, according to the specified parallelism, the topic creation module (2) is used to create a specified number of subtopics. Next, the publisher creation module (3) is called under each subtopic to create a publisher. After that, the parallel transmission module (5) can be called to perform parallel transmission.

[0006] On the data subscription side, the operation steps of each module are similar to those on the publishing side, but please note that the degree of parallelism must be the same as that on the publishing side.

[0007] A key to improving the single-thread data receiving and sending throughput of the present invention is that the publishing end needs to split the data packet of topic T into n sub-topics according to the parallelism n for sending, such as Figure 2As shown. Create a publisher module (3) to create a sending thread pool, the size of the thread pool is equal to the degree of parallelism. The sending threads in the thread pool will not execute the sending task immediately, but will wait for the semaphore in their respective sending subtasks to indicate whether the data to be sent is ready. When the semaphore is available, it means that the data in the sending subtask is ready, and then the sending action is executed. After the sending is completed, continue to wait for the next semaphore to be ready. The planning of the sending subtask is completed by the parallel sending module (5). Whenever a packet of data to be sent arrives, the module will split the data packet into tasks according to the degree of parallelism n, so that the length of each subtask sent is almost equal (the data length of the last subtask may be slightly smaller). Each subtask is a "(T k ,len k , offset k )”, where k is the subtask number, T k Indicates that the data shards belonging to subtask k will be in the topic T k Send, len k Indicates the length of the data segment to be sent by subtask k, offset k Indicates the offset of the data fragment to be sent in the original data packet. When each sending thread sends its own data fragment, it will encapsulate the fragment length and offset information into its own data packet header for use by the receiving end.

[0008] Figure 3 The data transmission logic of the transmitting end is shown.

[0009] Another key to improving the single-thread data receiving and sending throughput of the present invention is that the receiving end performs parallel reception according to the parallelism n, and completes the splicing of the original data packets while receiving the data fragments, such as Figure 4 As shown. Create a subscriber module (4) to register the data collection callback function. Whenever a data shard arrives, the callback function will be called. When the callback function receives a data shard, it first extracts the length of the shard, len k and offset k , then copy the data slice to the corresponding position of the final data receiving buffer according to the offset, and finally send a signal indicating that the slice has been received to notify the upper layer to take the data.

[0010] The original data packet of topic T is obtained through the parallel data collection module (6). The work of this module is to traverse and wait for the semaphores of each fragment (a total of n) to be ready. Because the data reception callback function has placed each fragment in the correct position according to the offset while completing data reception, when the semaphores of each fragment are ready, the actual packet assembly action has also been completed. The parallel data collection module only needs to return the first address of the data reception buffer to the application to complete the data delivery. The parallel data collection module itself does not need to copy the data fragments and assemble the packets. This zero-copy strategy is an important reason for the improvement of DDS data distribution throughput.

[0011] Figure 5 The data receiving logic at the receiving end is shown.

[0012] The beneficial effects of the present invention are

[0013] 1. Realized multi-core parallel sending and receiving of large blocks of data on Ethernet.

[0014] 2. Significantly improve the data distribution rate of DDS communication middleware. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of parallel transmission and reception of DDS communication middleware.

[0016] Figure 2 Publish schematic for 4-degree parallelism data.

[0017] Figure 3 Send logic flow chart for data.

[0018] Figure 4 Schematic diagram for 4-degree parallelism data subscription.

[0019] Figure 5 Logic flow chart for data reception.

[0020] Figure 6 This is a schematic diagram of parallel transmission and reception of cross-plug-in communication middleware. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] like Figure 1 As shown, a publish-subscribe multi-threaded parallel transmission device based on data segmentation of the present invention comprises:

[0023] (1) Initialization module

[0024] This module completes the creation of necessary entities (such as domain participants), memory allocation, and parallelism specification, and generates a publish-subscribe management handle.

[0025] (2) Create a theme module

[0026] This module creates multiple data sending / receiving topics according to the parallelism specified during initialization. The number of topics is equal to the parallelism.

[0027] (3) Create a publisher module

[0028] This module creates a publisher for each data publishing topic, and creates a data sending thread in the sending thread pool for the publisher to execute the data sending subtask of the publisher.

[0029] (4) Create a subscriber module

[0030] This module creates a subscriber for each data receiving topic and registers a data collection callback function for the subscriber. Whenever the subscriber receives a data shard, the callback function will be called to receive the data shard.

[0031] (5) Parallel data transmission module

[0032] This module executes the sending task planning. It splits each packet of data according to the degree of parallelism, generates a sending subtask set, and notifies the sending thread to send each data fragment in parallel.

[0033] (6) Parallel data collection module

[0034] This module collects the data fragments and delivers the assembled original data packets to the application. It is not responsible for assembly and does not copy any data.

[0035] In order to further illustrate the specific implementation of the present invention, an example of cross-insert box data distribution based on a domestic Feiteng computing board through standard Ethernet is given, such as Figure 6As shown. In the example, each plug-in box contains several Feiteng computing boards and a switch board, and both the switch board and the computing board are connected to the backplane. The computing boards in the plug-in box are interconnected through the switch board, and the data link standard is 40G 10G Ethernet. There is also a rear interface board inserted on the backplane to access the optical fiber link and interconnect with another plug-in box or optical fiber switch. The data link standard between the plug-ins is also 40G 10G Ethernet. The data sending application is deployed on computing board 1 of plug-in box 1. The application first calls the initialization module, passes in the parallelism parameter n, and obtains a publish-subscribe management handle; then calls the topic creation module, passes in the publish-subscribe management handle and the topic name template T, and obtains a multi-channel topic (the number of channels of the topic is equal to the parallelism n); next, calls the publisher creation module, passes in the publish-subscribe management handle, and creates a publisher for each channel of the topic; finally, the application's publishing loop, in which the parallel publishing module is called to publish the data, and the application does not need to care about the underlying topic data splitting and task planning details.

[0036] The data receiving application is deployed on the computing board 1 of the plug-in box 2. The application first calls the initialization module, passes in the parallelism parameter n (the parallelism must be equal to the parallelism of the sender), and obtains a publish-subscribe management handle; then calls the topic creation module, passes in the publish-subscribe management handle and the topic name template T, and obtains a multi-channel topic (the number of channels of the topic is equal to the parallelism n); next, calls the subscriber creation module, passes in the publish-subscribe management handle, and creates a subscriber for each channel of the topic; finally, the application's data collection loop is called in this loop to receive the data of the original topic template T.

[0037] The present invention is not limited to the above specific implementations, and the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made to the above implementations based on the technical essence of the present invention shall be included in the protection scope of the present invention.

Claims

1. A publish-subscribe multi-threaded parallel transmission device based on data segmentation, characterized in that: It includes 6 modules, namely: initialization module (1), topic creation module (2), publisher creation module (3), subscriber creation module (4), parallel data transmission module (5) and parallel data collection module (6); specifically, The initialization module (1) completes entity creation, memory application and parallelism specification, and generates a publish-subscribe management handle; The topic creation module (2) creates multiple data sending / receiving topics according to the parallelism specified during initialization, and the number of topics is equal to the parallelism; The publisher creation module (3) creates a publisher for each data publishing topic, and simultaneously creates a data sending thread for the publisher in the sending thread pool to execute the data sending subtask of the publisher; The subscriber creation module (4) creates a subscriber for each data receiving topic and registers a data collection callback function for the subscriber. Whenever the subscriber receives a data fragment, the callback function will be called to receive the data fragment. The parallel sending module (5) executes the sending task planning, splits each packet of data according to the parallelism, generates a sending subtask set, and notifies the sending thread to send each data fragment in parallel; The parallel data collection module (6) collects each data fragment and delivers the assembled original data packet to the application. It is not responsible for assembling the data packet itself and does not copy any data.

2. According to claim 1, a publish-subscribe multi-threaded parallel transmission device based on data segmentation is characterized in that: At the data publishing end, firstly, the entity creation, memory application and parallelism designation are completed through the initialization module (1); then, according to the designated parallelism, a designated number of sub-topics are created through the creation of topics module (2); under each sub-topic, the creation of publisher module (3) is called to create a publisher; then, the parallel transmission module (5) is called to perform parallel transmission; On the data subscription side, the initialization module (1) is first called, the parallelism parameter is passed in, and a publish-subscribe management handle is obtained; Then, the topic creation module (2) is called, and the publish-subscribe management handle and the topic name template are passed in to obtain a multi-channel topic, where the number of channels of the topic is equal to the degree of parallelism; next, the subscriber creation module (4) is called, and the publish-subscribe management handle is passed in to create a subscriber for each channel of the topic; a data collection loop is created, in which the parallel data collection module (6) is called to receive the data of the original topic template.

3. The publish-subscribe multi-threaded parallel transmission device based on data segmentation according to claim 2, characterized in that: Create a publisher module (3) to create a sending thread pool. The size of the thread pool is equal to the degree of parallelism. The sending threads in the thread pool will not execute the sending task immediately. They will wait for the semaphore used to indicate whether the data to be sent is ready in each sending subtask. When the semaphore is available, it means that the data in the sending subtask is ready, and the sending action is executed. After the sending is completed, continue to wait for the next semaphore to be ready.

4. The publish-subscribe multi-threaded parallel transmission device based on data segmentation according to claim 3, characterized in that: The planning of sending subtasks is completed by the parallel sending module (5); whenever a packet of data to be sent arrives, the parallel sending module (5) divides the data packet into tasks according to the parallelism n, so that the length of each subtask sent is almost equal. Each subtask is a triplet marked by the subtask number, including the subtask subject, the subtask data fragment length and the offset; when each sending thread sends its own data fragment, it will encapsulate the data fragment length and offset information into its own data packet header for use by the receiving end.

5. The publish-subscribe multi-threaded parallel transmission device based on data segmentation according to claim 4, characterized in that: Create a subscriber module (4) to register a data collection callback function. Whenever a data segment arrives, the callback function will be called. When receiving the data segment, the callback function first extracts the length and offset of the data segment, and then copies the data segment to the corresponding position of the final data receiving buffer according to the offset. Finally, it sends a signal indicating that the segment has been received, notifying the upper layer to take the data away.

6. The publish-subscribe multi-threaded parallel transmission device based on data segmentation according to claim 5, characterized in that: The original data packet of the subject is obtained through the parallel data receiving module (6), and the semaphores of each fragment are traversed and waited for to be ready. Because the data receiving callback function has placed each fragment in the correct position according to the offset while completing the data reception, when the semaphores of each fragment are ready, the actual packet assembly action has also been completed. The parallel data receiving module only needs to return the first address of the data receiving buffer to the application to complete the data delivery, without copying the data fragments and assembling the packets.

Images