A data transmission method and system

Abstract

The application discloses a data transmission method and system, the method comprises the following steps: a publisher registers a topic by creating a shared memory and initializing metadata; the publisher stores to-be-transmitted data into the shared memory and the metadata; a subscriber subscribes to the topic and maps the metadata under the topic to the memory space of the subscriber; and the subscriber takes out the to-be-transmitted data in the memory space through deserialization according to the metadata. Based on the technologies of shared memory, lock-free data structure and zero-copy, the application realizes high-performance data transmission between modules in the same computing unit, and has the advantages of high throughput, low delay, no blocking, zero-copy, no single point of failure and the like.

Description

Technical Field

[0001] This application relates to the field of communication technology, and more specifically, to a data transmission method and system. Background Technology

[0002] As society progresses and people's living standards improve, the importance of data is constantly increasing. From mobile phones to smart home appliances to the automotive industry, data collection and transmission have permeated every aspect of technological advancements in life. The ever-evolving sensors for information collection and the ever-growing number of connected devices are generating more data than ever before, and data transmission technologies need to keep pace.

[0003] Taking the autonomous driving industry as an example, autonomous driving requires equipment to be aware of its surroundings in real time, make correct judgments based on the collected information, and then take corresponding actions. This inevitably requires the installation of a large number of sensors in the autonomous driving system, including LiDAR, millimeter-wave radar, ultrasonic radar, and cameras. Through the flexible cooperation of these sensors, timely responses, predictive planning and decision-making are achieved, ultimately realizing automated driving and ensuring the safe arrival of all passengers at their destination. This process involves a large amount of sensor data transmission. Therefore, how to achieve high-performance data transmission between modules has become a key challenge and a pain point that needs to be addressed and overcome. Summary of the Invention

[0004] In view of this, this application provides a data transmission method and system that, based on technologies such as shared memory, lock-free data structures, and zero-copy, achieves high-performance data transmission between modules and has advantages such as high throughput, low latency, non-blocking, zero copy, and no single point of failure.

[0005] To achieve the above objectives, the following solution is proposed:

[0006] A data transmission method, comprising:

[0007] Publishers register topics by creating shared memory and initializing metadata;

[0008] The publisher stores the data to be transmitted in the shared memory and the metadata;

[0009] The subscriber subscribes to the topic and maps the metadata under the topic to the subscriber's memory space;

[0010] The subscriber retrieves the data to be transmitted from the memory space by deserialization based on the metadata.

[0011] Optionally, the process by which the publisher registers a topic by creating shared memory and initializing metadata includes:

[0012] The publisher creates shared memory and uses the shared memory as a circular buffer under the topic, and the circular buffer has head and tail pointers;

[0013] The publisher initializes metadata, which includes a message metadata array with head and tail pointers, a pre-stored information area, and a tag file.

[0014] Optionally, the process by which the publisher stores the data to be transmitted into the shared memory and the metadata includes:

[0015] The publisher stores the business data of the data to be transmitted into the pre-stored information area, and performs data decomposition on the data to be transmitted in the pre-stored information area;

[0016] The publisher stores the business data of the data to be transmitted into the shared memory ring buffer;

[0017] The publisher stores the basic information and data address of the data to be transmitted into the message metadata array, where the data address is the storage address of the business data in the shared memory ring buffer.

[0018] The publisher stores the topic name and data type corresponding to the data to be transmitted in the tag file.

[0019] Optional, also includes:

[0020] After the data to be transmitted is stored in the shared memory and the metadata, the publisher clears and releases the pre-stored information area.

[0021] Optionally, the process of a publisher registering a topic may further include:

[0022] The publisher establishes local inter-process communication to listen for connection requests;

[0023] The subscribers subscribe to the topic, including:

[0024] The subscriber uses the inotify monitoring framework to listen to the tag file of the topic;

[0025] When the subscriber detects the publisher in the tag file, it sends a connection request to the publisher and adds the callback function provided by the publisher to the polling array.

[0026] Optionally, the process by which the subscriber retrieves the data to be transmitted from the memory space through deserialization based on the metadata includes:

[0027] Search and determine the data address of the business data that corresponds to the data to be transmitted in the message metadata array of the metadata;

[0028] Based on the data address of the data to be transmitted, the service data of the data to be transmitted is extracted from the shared memory circular buffer and restored by deserialization.

[0029] The data to be transmitted is generated by calling the corresponding callback function.

[0030] Optionally, the process by which the subscriber retrieves the data to be transmitted from the memory space through deserialization based on the metadata further includes:

[0031] The subscriber confirms the validity of the data to be transmitted by using the head and tail pointer positions of the shared memory circular buffer and / or the head and tail pointer positions of the message metadata array.

[0032] Optionally, the process by which the subscriber confirms the validity of the data to be transmitted through the head and tail pointer positions of the shared memory circular buffer includes:

[0033] The subscriber determines whether the memory data of the data to be transmitted has been overwritten in the shared memory circular buffer by using the head and tail pointer positions of the shared memory circular buffer;

[0034] If the memory data is overwritten in the shared memory circular buffer, the subscriber confirms that the data to be transmitted is invalid.

[0035] If the memory data is not overwritten in the shared memory ring buffer, the subscriber confirms that the data to be transmitted is valid.

[0036] Optionally, the process by which the subscriber confirms the validity of the data to be transmitted through the head and tail pointer positions of the message metadata array includes:

[0037] The subscriber determines whether the data address of the data to be transmitted has been overwritten in the message metadata array by using the head and tail pointer positions of the message metadata array;

[0038] If the data address is overwritten in the message metadata array, the subscriber confirms that the data to be transmitted is invalid.

[0039] If the data address is not covered in the message metadata array, the subscriber confirms that the data to be transmitted is valid.

[0040] A data transmission system, comprising publishers and subscribers;

[0041] The publisher registers a topic by creating shared memory and initializing metadata, and stores the data to be transmitted in the shared memory and the metadata.

[0042] The subscriber subscribes to the topic, maps the metadata under the topic to the subscriber's memory space, and retrieves the data to be transmitted from the memory space by deserialization based on the metadata.

[0043] As can be seen from the above technical solutions, the data transmission method and system provided in this application embodiment, based on technologies such as shared memory, lock-free data structures, and zero-copy, achieves high-performance data transmission between modules within the same computing unit by creating shared memory and storing the data to be transmitted in the shared memory and the metadata. This has the following advantages:

[0044] Firstly, it has a high throughput rate. Since data is written directly into shared memory without going through general communication channels such as networks or pipes, it can transmit a large amount of data and has a high throughput rate.

[0045] Secondly, low latency. In this application, the data transmission latency depends on the speed at which data is written to shared memory and the latency of local inter-process communication, the latter of which does not contain message payload, so the overall latency is low.

[0046] Third, it is non-blocking. When a publisher publishes a new message upstream, it does not need to wait for a response from downstream or release shared memory, so it will not be affected by downstream subscribers and avoids blocking of shared resources.

[0047] Fourth, there is no memory fragmentation and low memory allocation and reclamation overhead. Due to the use of a circular buffer, messages can be continuously placed into shared memory, and the complexity of memory allocation and reclamation is O(1).

[0048] Fifth, zero copy: upstream publishers can directly write data to shared memory, and downstream subscribers can directly process the data in shared memory without additional data copying.

[0049] Sixth, there is no single point of failure. Topic discovery is achieved by listening to the tag file, which does not require a dedicated cluster control node. Furthermore, topics are completely independent of each other, so there is no single point of failure. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a flowchart of a data transmission method disclosed in this application;

[0052] Figure 2 A schematic diagram illustrating shared memory and initialization metadata provided in an embodiment of this application;

[0053] Figure 3 A schematic diagram illustrating an invalid shared memory circular buffer overwrite provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of a data transmission system disclosed in this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] This application provides a data transmission method that can be applied to various data transmission systems and computing units, enabling high-performance data transmission between modules within a data transmission system or within the same computing unit.

[0057] The following section introduces the solution proposed in this application. The technical solution is as follows, and details are provided below.

[0058] Figure 1 This is a flowchart of a data transmission method disclosed in an embodiment of this application, such as... Figure 1 As shown, the method may include:

[0059] Step S1: The publisher registers the topic by creating shared memory and initializing metadata.

[0060] Specifically, a Topic acts as a transmission intermediary between message publishers and subscribers. Devices can send and receive messages through Topics, thereby enabling communication between the server and the device. The process of a publisher registering a topic includes creating shared memory and initializing metadata, where the shared memory can be used as a circular buffer in this application.

[0061] Step S2: The publisher stores the data to be transmitted into the shared memory and the metadata.

[0062] Specifically, the publisher processes the data to be transmitted and stores it in the shared memory and the metadata. Since the data is written directly into the shared memory without going through typical communication channels such as networks or pipes, it can adapt to scenarios involving large-scale data transmission. The publisher can directly write the data to be created into the shared memory and the metadata for storage, while the subscriber can also retrieve data from the shared memory and the metadata. Therefore, during the transmission of the data from the publisher to the subscriber, no additional copying is performed, and both the publisher and the subscriber can write and retrieve data through the shared memory and the metadata.

[0063] Step S3: The subscriber subscribes to the topic and maps the metadata under the topic to the subscriber's memory space.

[0064] Specifically, when a subscriber discovers a new publisher, it creates a domain socket and connects to it to subscribe to the topic. Then, it can open the metadata for that topic via local inter-process communication and map the metadata into the subscriber's local memory space. The metadata stores basic information and data addresses of the data to be transmitted. Afterward, the subscriber needs to retrieve the data to be transmitted from the memory space based on the data address.

[0065] Step S4: The subscriber retrieves the data to be transmitted from the memory space by deserialization based on the metadata.

[0066] Specifically, the subscriber determines the address of the business data to be transmitted in the memory space based on the records in the metadata, retrieves the data at that address, and restores the data to be transmitted through deserialization. Deserialization refers to the process of converting raw data into protobuf format.

[0067] As can be seen from the above technical solutions, the data transmission method provided in this application embodiment, based on technologies such as shared memory, lock-free data structures, and zero-copy, achieves high-performance data transmission between modules within the same computing unit by creating shared memory and storing the data to be transmitted in the shared memory and the metadata. This method has the following advantages:

[0068] Firstly, it has a high throughput rate. Since data is written directly into shared memory without going through general communication channels such as networks or pipes, it can transmit a large amount of data and has a high throughput rate.

[0069] Secondly, low latency. In this application, the data transmission latency depends on the speed at which data is written to shared memory and the latency of local inter-process communication, the latter of which does not contain message payload, so the overall latency is low.

[0070] Third, it is non-blocking. When a publisher publishes a new message upstream, it does not need to wait for a response from downstream or release shared memory, so it will not be affected by downstream subscribers and avoids blocking of shared resources.

[0071] Fourth, there is no memory fragmentation and low memory allocation and reclamation overhead. Due to the use of a circular buffer, messages can be continuously placed into shared memory, and the complexity of memory allocation and reclamation is O(1).

[0072] Fifth, zero copy: upstream publishers can directly write data to shared memory, and downstream subscribers can directly process the data in shared memory without additional data copying.

[0073] Sixth, there is no single point of failure. Topic discovery is achieved by listening to the tag file, which does not require a dedicated cluster control node. Furthermore, topics are completely independent of each other, so there is no single point of failure.

[0074] In some embodiments of this application, the process by which the publisher registers a topic by creating shared memory and initializing metadata in step S1 is described, and may specifically include:

[0075] Step S11: The publisher creates shared memory and uses the shared memory as a circular buffer under the topic. The circular buffer has head and tail pointers.

[0076] Specifically, Figure 2 This is a schematic diagram of shared memory and initialization metadata provided in an embodiment of this application, such as... Figure 2 As shown, shared memory serves as a circular buffer under the topic. This circular buffer can be reused, and when new messages are filled, old messages may be overwritten, requiring updates to the corresponding message metadata. New data is filled from the tail pointer position. When the shared memory circular buffer is full, the corresponding old data will be overwritten.

[0077] For example, in Figure 3 When the shared memory circular buffer stores six data entries, it is not full, and new data will be filled from the tail pointer position. When the eighth data entry is filled, the circular buffer is full. If a ninth data entry is then needed, the first data entry at the head pointer position will be overwritten. In practical applications, considering the varying sizes of data, it's possible for a new data entry to overwrite multiple existing data entries. Therefore, adding a process to check the validity of the message data could be considered.

[0078] Step S12: The publisher initializes the metadata, which includes a message metadata array with head and tail pointers, a pre-stored information area, and a tag file.

[0079] Specifically, such as Figure 2 As shown, the publisher initializes the metadata, which is divided into three parts: a message metadata array with head and tail pointers, a pre-stored information area, and a tag file.

[0080] The message metadata array is used to store basic information about the data to be transmitted, such as data length, and the data address. The data address is the storage address of the service data in the shared memory ring buffer.

[0081] The pre-stored information area is a pre-processing area for message data. When information data needs to be transmitted, the data to be transmitted must first be placed in the pre-stored information area. The data to be transmitted is then decomposed in the pre-stored information area to achieve storage of the data to be transmitted.

[0082] The tag file is used to record the topic name and data type corresponding to the data to be transmitted. When a new publisher publishes a message, subscribers can obtain the publishing information by listening to the tag file.

[0083] Optionally, based on the above, the process of a publisher registering a topic may further include:

[0084] Step S13: The publisher creates local inter-process communication to listen for connection requests.

[0085] Conversely, in step S3, the subscriber subscribing to the topic may include:

[0086] Step S31: The subscriber uses the inotify monitoring framework to listen to the tag file of the topic.

[0087] Specifically, inotify is a high-efficiency, real-time Linux file system event monitoring framework. Based on inode-level file system monitoring technology, inotify is a powerful, fine-grained, asynchronous mechanism that meets a wide range of file monitoring needs, not just security and performance. Inotify can efficiently track changes in the Linux file system in real time. Subscribers use the inotify monitoring framework to listen to the marker files of the topic to obtain publication information. When a new publisher appears, a new marker file will appear. The appearance of a new marker file can then identify the new publisher and publication information, allowing subscribers to receive the information promptly. Therefore, topic discovery is achieved by listening to marker files, eliminating the need for a dedicated cluster control node, and topics are completely independent, thus eliminating single points of failure.

[0088] Step S32: When the subscriber detects the publisher in the tag file, it sends a connection request to the publisher and adds the callback function provided by the publisher to the polling array.

[0089] Specifically, when a subscriber detects the publisher's presence in the tag file, it connects to the publisher via local inter-process communication, sending a connection request and obtaining the publisher's callback function, which is then added to the polling array. A subscriber can receive data from multiple topics. The polling array stores the callback functions for all topics.

[0090] In some embodiments of this application, the process of the publisher storing the data to be transmitted into the shared memory and the metadata is described, and may specifically include:

[0091] Step S21: The publisher stores the business data of the data to be transmitted into the pre-stored information area, and performs data decomposition on the data to be transmitted in the pre-stored information area.

[0092] Specifically, the pre-stored information area is a pre-processing area for message data. When information data needs to be transmitted, the data to be transmitted is first placed in the pre-stored information area. The data to be transmitted is then decomposed in the pre-stored information area to achieve storage. Specifically, the data is decomposed into the business data portion stored in the shared memory circular buffer, the basic information and data address portion stored in the message metadata array, and the topic name and data type portion corresponding to the data to be transmitted stored in the tag file.

[0093] Optionally, after the publisher stores the data to be transmitted in the shared memory and the metadata, the publisher clears and releases the pre-stored information area.

[0094] After a piece of data to be transmitted is decomposed and stored sequentially in the shared memory and the metadata, the data to be transmitted has been stored. At this time, the pre-stored information area will be cleared and released for use by the next piece of data to be transmitted.

[0095] Step S22: The publisher stores the service data of the data to be transmitted into the shared memory ring buffer.

[0096] Specifically, the shared memory circular buffer is used to store the business data to be transmitted. During the storage process, the data is put into the current corresponding tail pointer position in the storage order. Since the shared memory is a circular buffer, the message data can be put into the shared memory continuously. The complexity of memory allocation and reclamation is O(1), which has the advantages of no memory fragmentation and low memory allocation and reclamation overhead.

[0097] Step S23: The publisher stores the basic information and data address of the data to be transmitted into the message metadata array, where the data address is the storage address of the business data in the shared memory ring buffer.

[0098] Specifically, the message metadata array is used to store basic information such as the data length of the data to be transmitted, as well as the data address. The data address is the storage address of the business data in the shared memory ring buffer.

[0099] Similar to the shared memory circular buffer, the message metadata array is also a circular array that can be reused, and new messages may overwrite older ones. New data is filled in from the tail pointer position, and when the message metadata array is full, the corresponding older data will be overwritten.

[0100] For example, such as Figure 2 The message metadata array described above already contains six data entries. Since the array is not yet full, new data will be added from the tail pointer position. When the tenth data entry is added, the array will be full. If an eleventh data entry is then added, the first data entry at the head pointer position will be overwritten. However, in practical applications, considering the varying sizes of the data, it's possible for a new data entry to overwrite multiple existing data entries.

[0101] Step S24: The publisher stores the topic name and data type corresponding to the data to be transmitted into the tag file.

[0102] Specifically, the tag file is used to record the topic name and data type corresponding to the data to be transmitted. When a new publisher publishes a message, subscribers can obtain the publishing information by listening to the tag file.

[0103] In some embodiments of this application, the process of the subscriber retrieving the data to be transmitted from the memory space by deserialization based on the metadata in step S4 is described, and may specifically include:

[0104] Step S41: The subscriber searches for and determines the data address of the business data that is related to the data to be transmitted in the message metadata array of the metadata.

[0105] Step S42: The subscriber extracts the business data of the data to be transmitted from the shared memory ring buffer based on the data address of the data to be transmitted, and restores the data to be transmitted by deserialization.

[0106] Specifically, after determining the address of the business data to be transmitted in the memory space based on the data address of the business data to be transmitted recorded in the message metadata array, the subscriber can extract the business data at that data address and restore the business data to be transmitted through deserialization.

[0107] Step S43: The subscriber generates the data to be transmitted by calling the corresponding callback function.

[0108] Specifically, when the subscriber detects the publisher's presence in the tag file, it sends a connection request to the publisher and obtains a callback function provided by the publisher, which is then added to the polling array. The subscriber then generates the data to be transmitted based on the business data of the data to be transmitted by calling the corresponding callback function.

[0109] Downstream subscribers can directly process the data in shared memory during the entire process of retrieving and restoring the data to be transmitted without the need for additional data copying. Furthermore, the execution of data transmission between upstream and downstream is relatively independent. When an upstream publisher publishes a new message, it does not need to wait for a response from a downstream subscriber or release shared memory, so it will not be affected by the processing speed of the downstream subscriber, thus avoiding blocking of shared resources.

[0110] In some embodiments of this application, considering the capacity limitations of the shared memory circular buffer and the message metadata array, in step S4, when the subscriber retrieves the data to be transmitted from the memory space through deserialization based on the metadata, a process for confirming the validity of the data to be transmitted can be added. Specifically, this may include:

[0111] The subscriber confirms the validity of the data to be transmitted by using the head and tail pointer positions of the shared memory circular buffer and / or the head and tail pointer positions of the message metadata array.

[0112] The process of verifying the validity of the data to be transmitted twice is described below:

[0113] first,

[0114] The process by which the subscriber confirms the validity of the data to be transmitted through the head and tail pointer positions of the message metadata array may include:

[0115] The subscriber determines whether the data address of the data to be transmitted has been overwritten in the message metadata array by using the head and tail pointer positions of the message metadata array;

[0116] If the data address is overwritten in the message metadata array, the subscriber confirms that the data to be transmitted is invalid.

[0117] If the data address is not covered in the message metadata array, the subscriber confirms that the data to be transmitted is valid.

[0118] The second time

[0119] The process by which the subscriber confirms the validity of the data to be transmitted through the head and tail pointer positions of the shared memory circular buffer may include:

[0120] The subscriber determines whether the memory data of the data to be transmitted has been overwritten in the shared memory circular buffer by using the head and tail pointer positions of the shared memory circular buffer;

[0121] If the memory data is overwritten in the shared memory circular buffer, the subscriber confirms that the data to be transmitted is invalid.

[0122] If the memory data is not overwritten in the shared memory ring buffer, the subscriber confirms that the data to be transmitted is valid.

[0123] Specifically, the shared memory and the message metadata array are a reusable circular buffer and a circular array, respectively. When a new message is filled, an old message may be overwritten, requiring the corresponding message metadata to be updated. New data is filled from the position of the tail pointer. When the shared memory circular buffer and the message metadata array are full, the corresponding old data will be overwritten. Both are stored in the same way.

[0124] The following explanation uses a shared memory circular buffer as an example. Figure 3 As shown, assuming all data items are the same size, when the shared memory circular buffer stores six data items, the buffer is not full. New data will be filled from the tail pointer position. At this time, the tail pointer has not yet passed the head pointer. When the eighth data item is filled, the shared memory circular buffer is full, and the head and tail pointer positions overlap. If a ninth data item needs to be filled, the tail pointer will pass the head pointer position for the first time, and the corresponding first data item will be overwritten. The overwritten data to be transmitted is invalid. That is, after the ninth data item is filled, the first data item is overwritten by the ninth data item, and the first data item is invalid. The validity of the data to be transmitted can be determined by judging whether the memory data of the data to be transmitted has been overwritten in the shared memory circular buffer by the head and tail pointer positions.

[0125] Understandably, if the first determination has already determined that the data to be transmitted is invalid, that is, the message metadata array can no longer find the data address of the data to be transmitted, and the data address of the data to be transmitted has been overwritten and invalidated, then the second determination does not need to be executed.

[0126] The following describes a data transmission system provided by an embodiment of this application. The data transmission system described below can be referred to in correspondence with the data transmission method described above, and can implement the various steps of the data transmission method described in any of the above claims.

[0127] See Figure 4 , Figure 4 This is a schematic diagram of the structure of a data transmission system disclosed in an embodiment of this application.

[0128] like Figure 4 As shown, the system can include publishers and subscribers.

[0129] The publisher registers a topic by creating shared memory and initializing metadata, and stores the data to be transmitted in the shared memory and the metadata.

[0130] The subscriber subscribes to the topic, maps the metadata under the topic to the subscriber's memory space, and retrieves the data to be transmitted from the memory space by deserialization based on the metadata.

[0131] As can be seen from the above technical solutions, the data transmission method and system provided in this application embodiment, based on technologies such as shared memory, lock-free data structures, and zero-copy, achieves high-performance data transmission between modules within the same computing unit by creating shared memory and storing the data to be transmitted in the shared memory and the metadata. This has the following advantages:

[0132] Firstly, it has a high throughput rate. Since data is written directly into shared memory without going through general communication channels such as networks or pipes, it can transmit a large amount of data and has a high throughput rate.

[0133] Secondly, low latency. In this application, the data transmission latency depends on the speed at which data is written to shared memory and the latency of local inter-process communication, the latter of which does not contain message payload, so the overall latency is low.

[0134] Third, it is non-blocking. When a publisher publishes a new message upstream, it does not need to wait for a response from downstream or release shared memory, so it will not be affected by downstream subscribers and avoids blocking of shared resources.

[0135] Fourth, there is no memory fragmentation and low memory allocation and reclamation overhead. Due to the use of a circular buffer, messages can be continuously placed into shared memory, and the complexity of memory allocation and reclamation is O(1).

[0136] Fifth, zero copy: upstream publishers can directly write data to shared memory, and downstream subscribers can directly process the data in shared memory without additional data copying.

[0137] Sixth, there is no single point of failure. Topic discovery is achieved by listening to the tag file, which does not require a dedicated cluster control node. Furthermore, topics are completely independent of each other, so there is no single point of failure.

[0138] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0140] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data transmission method, characterized in that, include: Publishers register topics by creating shared memory and initializing metadata, including: The publisher creates shared memory and uses the shared memory as a circular buffer under the topic, and the circular buffer has head and tail pointers; The publisher initializes metadata, which includes a message metadata array with head and tail pointers, a pre-stored information area, and a tag file; The publisher stores the data to be transmitted in the shared memory and the metadata, including: The publisher stores the data to be transmitted in the pre-stored information area and performs data decomposition on the data to be transmitted in the pre-stored information area; The publisher stores the business data of the data to be transmitted into the shared memory ring buffer; The publisher stores the basic information and data address of the data to be transmitted into the message metadata array, where the data address is the storage address of the business data in the shared memory ring buffer. The publisher stores the topic name and data type corresponding to the data to be transmitted into the tag file; The subscriber subscribes to the topic and maps the metadata under the topic to the subscriber's memory space; The subscriber retrieves the data to be transmitted from the memory space by deserialization based on the metadata. The process of registering a topic for the publisher also includes: The publisher establishes local inter-process communication to listen for connection requests; The subscribers subscribe to the topic, including: The subscriber uses the inotify monitoring framework to listen to the tag file of the topic; When the subscriber detects the publisher in the tag file, it sends a connection request to the publisher and adds the callback function provided by the publisher to the polling array.

2. The method according to claim 1, characterized in that, Also includes: After the data to be transmitted is stored in the shared memory and the metadata, the publisher clears and releases the pre-stored information area.

3. The method according to claim 1, characterized in that, The process by which the subscriber retrieves the data to be transmitted from the memory space through deserialization based on the metadata includes: The subscriber searches for and determines the data address of the business data that matches the data to be transmitted in the message metadata array of the metadata. The subscriber extracts the business data of the data to be transmitted from the shared memory ring buffer based on the data address of the data to be transmitted, and restores the data to be transmitted by deserialization; The subscriber generates the data to be transmitted by calling the corresponding callback function.

4. The method according to claim 1, characterized in that, The process by which the subscriber retrieves the data to be transmitted from the memory space through deserialization based on the metadata also includes: The subscriber confirms the validity of the data to be transmitted by using the head and tail pointer positions of the shared memory circular buffer and / or the head and tail pointer positions of the message metadata array.

5. The method according to claim 4, characterized in that, The process by which the subscriber confirms the validity of the data to be transmitted through the head and tail pointer positions of the shared memory circular buffer includes: The subscriber determines whether the memory data of the data to be transmitted has been overwritten in the shared memory circular buffer by using the head and tail pointer positions of the shared memory circular buffer; If the memory data is overwritten in the shared memory circular buffer, the subscriber confirms that the data to be transmitted is invalid. If the memory data is not overwritten in the shared memory ring buffer, the subscriber confirms that the data to be transmitted is valid.

6. The method according to claim 4, characterized in that, The process by which the subscriber confirms the validity of the data to be transmitted through the head and tail pointer positions of the message metadata array includes: The subscriber determines whether the data address of the data to be transmitted has been overwritten in the message metadata array by using the head and tail pointer positions of the message metadata array; If the data address is overwritten in the message metadata array, the subscriber confirms that the data to be transmitted is invalid. If the data address is not covered in the message metadata array, the subscriber confirms that the data to be transmitted is valid.

7. A data transmission system, characterized in that, Including publishers and subscribers; The publisher registers a topic by creating shared memory and initializing metadata, and stores the data to be transmitted in the shared memory and the metadata. The subscriber subscribes to the topic, maps the metadata under the topic to the subscriber's memory space, and retrieves the data to be transmitted from the memory space by deserialization based on the metadata; The process by which a publisher registers a topic by creating shared memory and initializing metadata includes: The publisher creates shared memory and uses the shared memory as a circular buffer under the topic, and the circular buffer has head and tail pointers; The publisher initializes metadata, which includes a message metadata array with head and tail pointers, a pre-stored information area, and a tag file; The process by which the publisher stores the data to be transmitted into the shared memory and the metadata includes: The publisher stores the data to be transmitted in the pre-stored information area and performs data decomposition on the data to be transmitted in the pre-stored information area; The publisher stores the business data of the data to be transmitted into the shared memory ring buffer; The publisher stores the basic information and data address of the data to be transmitted into the message metadata array, where the data address is the storage address of the business data in the shared memory ring buffer. The publisher stores the topic name and data type corresponding to the data to be transmitted into the tag file; The process of registering a topic for the publisher also includes: The publisher establishes local inter-process communication to listen for connection requests; The subscribers subscribe to the topic, including: The subscriber uses the inotify monitoring framework to listen to the tag file of the topic; When the subscriber detects the publisher in the tag file, it sends a connection request to the publisher and adds the callback function provided by the publisher to the polling array.

Images