Data interaction method, system and device and computer program product

Abstract

The invention discloses a data interaction method, system and device and a computer program product, the method is executed by a data interaction system, and the method comprises the steps that in the system initialization stage, a continuous physical memory pool with fixed capacity is pre-allocated, and the continuous physical memory pool comprises a plurality of memory blocks organized in an annular queue form; writing the data into memory blocks of the continuous physical memory pool based on a preset data writing strategy; and reading data from the memory blocks of the continuous physical memory pool through memory mapping. According to the method, the continuous physical memory pool with the fixed capacity is pre-allocated, data writing is performed in combination with writing strategies such as circular multiplexing of the memory blocks, and data reading is performed in combination with a memory mapping technology, so that the bottleneck of a network layer is effectively avoided, and the requirements of high throughput, low delay and deterministic communication are met; the efficiency and the real-time performance of data interaction are remarkably improved, and a reliable data interaction solution is provided for scenes such as automatic driving perception and the like which have extremely high requirements on the real-time performance.

Description

Technical Field

[0001] This application relates to the field of autonomous driving data interaction technology, and in particular to a data interaction method, system and device, and computer program product. Background Technology

[0002] In the field of autonomous driving perception, multimodal and multi-temporal data fusion technology is gradually becoming mainstream. This technology integrates data collected at different times by different sensors (such as LiDAR and cameras) to obtain more comprehensive and accurate information about the surrounding environment, thereby improving the perception capabilities and decision-making accuracy of autonomous driving systems. However, the communication latency issue of multimodal and multi-temporal data severely limits the deployment of this technology on embedded devices. In autonomous driving scenarios, the real-time requirements for data are extremely high; even millisecond-level delays can lead to incorrect system decisions and safety hazards. Therefore, developing a module capable of high data throughput and real-time communication is crucial for the practical application of this perception technology.

[0003] Currently, existing data transmission schemes mainly fall into two categories: network-based inter-process communication (IPC) mechanisms and traditional shared memory technology. However, both schemes have certain problems in practical applications such as autonomous driving. For example, network-based IPC schemes suffer from high latency and bandwidth bottlenecks, high CPU utilization, and data transmission jitter. Traditional shared memory technology suffers from inefficient memory allocation, complex and insecure synchronization mechanisms, and low resource utilization.

[0004] Therefore, it is necessary to propose an innovative technical solution to solve at least some of the problems existing in the current technology, thereby promoting the application and development of autonomous driving perception technology on embedded devices. Summary of the Invention

[0005] This application provides a data interaction method, system, device, and computer program product to improve the transmission efficiency and stability of data transmission in autonomous driving scenarios.

[0006] The embodiments of this application adopt the following technical solutions:

[0007] In a first aspect, embodiments of this application provide a data interaction method, which is executed by a data interaction system, and the data interaction method includes:

[0008] During the system initialization phase, a fixed-capacity contiguous physical memory pool is pre-allocated, which contains multiple memory blocks organized in a circular queue.

[0009] Based on a preset data writing strategy, data is written into the memory blocks of the continuous physical memory pool;

[0010] Data is read from memory blocks of the contiguous physical memory pool via memory mapping.

[0011] Optionally, the preset data writing strategy includes a memory block circular reuse strategy, and the step of writing data into the memory blocks of the contiguous physical memory pool based on the preset data writing strategy includes:

[0012] Based on the memory block cyclic reuse strategy, the data is written into the memory blocks of the contiguous physical memory pool according to the organization order among multiple memory blocks.

[0013] Optionally, the circular reuse strategy based on the memory blocks, which writes the data into the memory blocks of the contiguous physical memory pool according to the organizational order among multiple memory blocks, includes:

[0014] Based on the circular reuse strategy of the memory blocks, determine the status flag of the memory block to be written.

[0015] When the state flag of the memory block is writable, the data is written to the memory block, and the state flag of the memory block is updated to non-writable through an atomic operation;

[0016] After the write operation is completed, the status flag of the memory block is updated to a writable state through an atomic operation.

[0017] Optionally, the status flags of the multiple memory blocks are maintained through a shared atomic variable region.

[0018] Optionally, the multiple memory blocks are indexed by a key value. After writing data into the memory blocks of the contiguous physical memory pool based on a preset data writing strategy, the data interaction method further includes:

[0019] The key value of the memory block is stored in the shared atomic variable area as the key value of the published memory block.

[0020] Optionally, reading data from memory blocks of the contiguous physical memory pool via memory mapping includes:

[0021] The key value of the published memory block is obtained from the shared atomic variable region through atomic operations;

[0022] Data is read from the published memory block using memory mapping based on the Key value of the published memory block.

[0023] Secondly, embodiments of this application also provide a data interaction system, the data interaction system comprising:

[0024] The memory management unit is used to pre-allocate a fixed-capacity contiguous physical memory pool during the system initialization phase. The contiguous physical memory pool contains multiple memory blocks organized in a circular queue.

[0025] The data writing unit is used to write data into the memory blocks of the continuous physical memory pool based on a preset data writing strategy.

[0026] A data reading unit is used to read data from memory blocks of the contiguous physical memory pool via memory mapping.

[0027] Optionally, the data interaction system is provided with a standardized interface, which is used to embed the data interaction system into an external system framework.

[0028] Thirdly, embodiments of this application also provide an apparatus, comprising:

[0029] A processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the aforementioned data interaction methods.

[0030] Fourthly, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement any of the aforementioned data interaction methods.

[0031] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: The data interaction method of the embodiments of this application is executed by a data interaction system. During the system initialization phase, a continuous physical memory pool with a fixed capacity is pre-allocated. The continuous physical memory pool contains multiple memory blocks organized in the form of a circular queue. Based on a preset data writing strategy, data is written into the memory blocks of the continuous physical memory pool. Data is read from the memory blocks of the continuous physical memory pool through memory mapping. The data interaction method of the embodiments of this application, by pre-allocating a continuous physical memory pool with a fixed capacity and organizing memory blocks in the form of a circular queue, combining the circular reuse strategy of memory blocks and the lock-free lightweight synchronization protocol for data writing, and using memory mapping technology for data reading, effectively avoids network layer bottlenecks, reduces problems such as inefficient memory management and redundant synchronization mechanisms, and can meet the requirements of high throughput, low latency, and deterministic communication. It significantly improves the efficiency and real-time performance of data interaction, and provides a reliable data interaction solution for scenarios with extremely high real-time requirements, such as autonomous driving perception. Attached Figure Description

[0032] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0033] Figure 1 This is a flowchart illustrating a data interaction method in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of the structure of a pre-allocated contiguous physical memory pool in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of the structure of a data interaction system according to an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of the structure of a device according to an embodiment of this application. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0038] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0039] Currently, the data transmission schemes used in existing technologies mainly include network-based inter-process communication (IPC) mechanisms and traditional shared memory technology. The following analysis, combined with specific application scenarios such as point cloud transmission of autonomous driving LiDAR, examines their technical defects and causes.

[0040] (a) Network-based IPC mechanisms (such as ROS Topic / Service)

[0041] Frameworks such as the Robot Operating System (ROS) employ a loop-based network communication mechanism using TCP / UDP protocols, achieving inter-process interaction through data serialization, network transmission, and deserialization. In this mechanism, the data sender serializes the data and sends it to the network buffer via the network protocol stack. The receiver then receives the data from the network buffer and deserializes it for subsequent processing. This approach has the following main problems:

[0042] (1) High latency and bandwidth bottleneck: When transmitting high-frequency big data (such as LiDAR point clouds, where a single frame can reach 1600KB), network bandwidth competition is fierce, leading to a surge in latency. Actual measurements show that the latency can reach 10-50ms, which cannot meet the stringent millisecond-level real-time requirements of scenarios such as autonomous driving.

[0043] (2) Multiple memory copies increase CPU utilization: Data needs to be copied to memory multiple times during transmission, including serialization to memory, copying from memory to the network buffer, and deserialization from the network buffer to the target memory. This increases CPU utilization by 20%-30%, further exacerbating transmission latency.

[0044] (3) Protocol stack redundancy overhead leads to transmission jitter: TCP retransmission mechanism, data packet splitting / packet reassembly operations, etc. introduce uncertainty, resulting in transmission jitter. The delay fluctuation is more than ±5ms, which seriously affects the real-time performance and stability of the data.

[0045] The main reason for the aforementioned technical problems is that the network protocol stack was designed primarily for versatility rather than real-time performance, and it was not optimized for large blocks of continuous data. It inherently relies on operating system kernel scheduling and cannot avoid inherent bottlenecks in the network layer, such as network bandwidth limitations and protocol processing latency, thus leading to the aforementioned problems when handling high-frequency, large-scale data transmission.

[0046] (II) Traditional Shared Memory Technology

[0047] Traditional shared memory technology allocates physical memory through the kernel, allowing multiple processes to map the same memory region to share data. Theoretically, this avoids network transmission overhead and improves data transmission efficiency. However, this approach has the following main problems:

[0048] (1) Inefficient dynamic memory allocation: Existing solutions typically allocate / release memory blocks dynamically on demand (e.g., using system calls such as shmget / shmdt), and frequent system calls lead to memory fragmentation. In actual tests, memory management accounted for more than 15% of the time, seriously affecting system performance.

[0049] (2) Lack of pre-allocation mechanism: It cannot cope with the sudden amount of data in high-throughput scenarios. For example, during the acquisition of lidar point clouds, sudden frames may occur. Traditional shared memory technology is difficult to process in time, which can easily lead to data loss or transmission delay.

[0050] (3) The synchronization mechanism is complex and insecure: it relies on semaphores, mutexes, etc. to achieve read-write synchronization. Lock contention causes latency fluctuations of about ±5ms. Moreover, simultaneous read and write by multiple processes can easily lead to data contention. Without isolation of read and write permissions, erroneous operations at the receiving end may damage data integrity and bring security risks to the system.

[0051] (4) Low resource utilization: The memory block reuse rate is less than 50%, and the circular reuse strategy is not adopted, resulting in physical memory waste and increasing system cost and resource management difficulty.

[0052] The main reason for the aforementioned technical problems is that traditional shared memory technology is not designed for high-throughput scenarios and lacks pre-allocation management, permission isolation, and lightweight synchronization mechanisms. When handling high-frequency, large-volume data transmissions, it cannot effectively coordinate the relationship between memory allocation, data synchronization, and resource utilization, making it difficult to balance performance and security.

[0053] Existing technologies suffer from three core problems: network transmission bottlenecks, inefficient memory management, and redundant synchronization mechanisms. These issues prevent them from meeting the demands for high throughput (>1GB / s), low latency (<2ms), and deterministic communication. Specifically, network communication is limited by protocol stack constraints, hindering real-time communication; dynamic allocation and lock contention in traditional shared memory lead to resource waste and reduced real-time performance; and hybrid solutions are hampered by architectural complexity and compatibility issues, making practical implementation difficult. Therefore, it is necessary to propose an innovative technical solution to address some of the existing technical problems and promote the application and development of autonomous driving perception technology on embedded devices.

[0054] Specifically, embodiments of this application provide a data interaction method, such as... Figure 1 The diagram illustrates a data interaction method according to an embodiment of this application. The data interaction method is executed by a data interaction system and includes at least the following steps S110 to S130:

[0055] Step S110: During the system initialization phase, a fixed-capacity contiguous physical memory pool is pre-allocated, wherein the contiguous physical memory pool contains multiple memory blocks organized in the form of a circular queue.

[0056] During the system initialization phase, the requirements of the business scenario are fully considered. For example, in the LiDAR point cloud transmission scenario, there are extremely high requirements for the real-time performance and stability of data transmission. Based on factors such as the data volume and transmission frequency in this scenario, a fixed-capacity contiguous physical memory pool is pre-allocated. Specifically, the total capacity of the memory pool is determined, for example, set to 10 × 2048KB, which contains N (here N = 10) fixed-size memory blocks, each memory block being 2048KB in size. These memory blocks are organized in the form of a circular queue.

[0057] like Figure 2 The diagram shows a pre-allocated contiguous physical memory pool according to an embodiment of this application. Memory blocks 1 to 10 are arranged sequentially, and the position of the currently operated memory block is indicated by a sliding pointer.

[0058] The pre-allocated physical memory pool in this embodiment consists of a contiguous physical memory region. Its physical address can be allocated and locked via ` / proc / iomem`, effectively preventing the system from reclaiming this memory during subsequent operation and ensuring the stability and availability of the memory pool. This pre-allocated fixed-capacity contiguous memory pool replaces the traditional dynamic memory allocation method, laying the foundation for efficient data interaction later.

[0059] Step S120: Based on a preset data writing strategy, write the data into the memory blocks of the continuous physical memory pool.

[0060] For data producers, data writing operations are performed based on a preset data writing strategy. The circular reuse strategy for memory blocks is a key strategy for data writing. When the sliding pointer reaches the end of the circular queue, it returns to the beginning of the queue, reusing the memory blocks to avoid the performance overhead of frequent memory allocation and deallocation. Simultaneously, a lock-free lightweight synchronization protocol is employed, which reduces the overhead and contention of inter-process synchronization, improving the efficiency and real-time performance of data writing. During data writing, data is accurately written to the corresponding memory blocks according to the circular reuse order, ensuring orderly data storage and efficient writing.

[0061] Step S130: Read data from memory blocks of the contiguous physical memory pool through memory mapping.

[0062] For data subscribers, memory mapping (mmap) is used to read data from contiguous physical memory pool blocks. Memory mapping is a technique (following the POSIX standard) that maps file or device memory to a process's address space. In this way, processes can directly access memory blocks in the memory pool as if they were accessing their own memory space, without the need for complex data copying operations. When data needs to be read, the process quickly and efficiently retrieves the data from the memory block based on the position of the sliding pointer, achieving fast data reading and transfer.

[0063] The aforementioned producer-subscriber-based data transmission model can be flexibly applied to different data transmission scenarios, such as data transmission between any functional modules, nodes, or processes in an autonomous driving system.

[0064] The data interaction method of this application embodiment pre-allocates a continuous physical memory pool of fixed capacity and organizes memory blocks in the form of a circular queue. It combines a memory block circular reuse strategy and a lock-free lightweight synchronization protocol for data writing, and uses memory mapping technology for data reading. This effectively avoids network layer bottlenecks, reduces problems such as inefficient memory management and redundant synchronization mechanisms, and can meet the requirements of high throughput, low latency and deterministic communication. It significantly improves the efficiency and real-time performance of data interaction, and provides a reliable data interaction solution for scenarios with extremely high real-time requirements such as autonomous driving perception.

[0065] In some embodiments of this application, the preset data writing strategy includes a memory block circular reuse strategy, and writing data into the memory blocks of the contiguous physical memory pool based on the preset data writing strategy includes: writing the data into the memory blocks of the contiguous physical memory pool according to the organization order among multiple memory blocks based on the memory block circular reuse strategy.

[0066] During the data writing phase, the producer determines the memory block to be written to based on the memory block circular reuse strategy. Combined with... Figure 2 The memory blocks are arranged in a circular queue, with a sliding pointer indicating the current operation position. When writing data, the memory blocks are arranged in the order of memory block 1, memory block 2, ..., memory block 10, and the movement of the sliding pointer determines the memory block to be written to. For example, when the sliding pointer points to memory block 3, data is written to memory block 3; after writing to memory block 10, the sliding pointer will point back to memory block 1, and the cycle of writing data will continue, achieving circular reuse of memory blocks.

[0067] This circular reuse strategy for memory blocks is primarily designed based on the principle of memory continuity and the circular buffer algorithm. Based on the principle of memory continuity, a contiguous physical memory layout helps improve data access efficiency; simultaneously, the circular buffer algorithm employs an overwrite strategy to ensure data timeliness, meaning new data can overwrite older data, ensuring that the memory pool always stores the latest and most valid data.

[0068] The circular reuse strategy offers several advantages. Firstly, it eliminates the overhead of dynamic allocation. Since memory blocks are pre-allocated, system calls for dynamic memory allocation, such as shmget / shmdt, are completely avoided, significantly reducing the time spent on memory management. Secondly, it ensures high throughput stability. A fixed-capacity memory pool can cache bursty data, such as consecutive frames of LiDAR point cloud data, avoiding latency fluctuations that may occur with dynamic memory allocation. Thirdly, it improves memory utilization. The circular queue design maximizes memory block reuse, reducing both memory fragmentation and overall memory utilization compared to traditional methods.

[0069] In some embodiments of this application, the method of writing the data into the memory blocks of the contiguous physical memory pool according to the organization order among multiple memory blocks based on the circular reuse strategy of the memory blocks includes: determining the status flag of the memory block to be written based on the circular reuse strategy of the memory blocks; writing the data into the memory block when the status flag of the memory block is writable, and updating the status flag of the memory block to a non-writable state through an atomic operation; and updating the status flag of the memory block to a writable state through an atomic operation after the writing is completed.

[0070] When a producer writes data, it first determines the status flag of the memory block to be written based on the memory block's circular reuse strategy. This status flag indicates the current availability of the memory block; for example, status = 0x01 indicates writable, and status = 0x02 indicates unwritable.

[0071] After determining the memory block to be written using a memory block-based circular reuse strategy, its status flag is checked. If the memory block's status flag is writable (i.e., status = 0x01), the producer can write data to that memory block. During the data writing process, to ensure that only one producer can write data to the same memory block at a time, an atomic operation is used to update the memory block's status flag to non-writable (status = 0x02). Atomic operations ensure that modifications to the status flag are indivisible in a multi-threaded or multi-process environment, avoiding the chaos caused by multiple producers modifying the status flag simultaneously.

[0072] After the data is written, the status flag of the memory block is also updated to writable (status=0x01) through atomic operations, so that data from other producers can continue to be written to the memory block, realizing the circular reuse of the memory block.

[0073] On the one hand, lock-free synchronization is achieved through hardware support for atomic operations. The uninterruptibility of atomic operations is guaranteed at the hardware level, thus avoiding the contention issues caused by traditional locking mechanisms. On the other hand, memory barriers ensure multi-core cache consistency, guaranteeing that different processor cores access memory block status flags consistently, preventing data errors caused by cache inconsistencies. Simultaneously, a read-write permission isolation design is employed, granting the receiving end only read permissions, eliminating the risk of accidental writes and eliminating the need for additional verification mechanisms.

[0074] This application's embodiments utilize a memory block circular reuse strategy, employing atomic operations to update memory block status flags to control data writing. Combined with read / write permission isolation design, this effectively replaces the traditional semaphore / mutex lock mechanism, mitigating latency fluctuations caused by lock contention to some extent, eliminating data contention risks, and eliminating the need for additional verification mechanisms, thus improving the efficiency and stability of data writing. Simultaneously, relying on atomic operation hardware support and memory barriers to ensure consistency in multi-core environments, it can meet the high real-time and high reliability requirements of data interaction, providing an efficient data writing solution for scenarios with stringent data transmission requirements, such as autonomous driving.

[0075] In some embodiments of this application, the status flags of the multiple memory blocks are maintained through a shared atomic variable region.

[0076] The shared atomic variable region is a specific hardware storage area, for example, its address can be set to 0x7FFE0000. This region is used to store memory block status flags, and 32-bit atomic variables can be used to record the status information of each memory block. Taking the meaning of the status flag values ​​as an example, a specific binary code can represent different states such as whether the memory block is writable or not.

[0077] During system operation, when a producer needs to write data to a memory block, it accesses the shared atomic variable area to obtain the status flag of the corresponding memory block. Because this area uses atomic variables, the operations of reading and updating the status flag are atomic and will not be interrupted by other operations. If the status flag indicates that the memory block is writable, the producer writes data and, during the writing process, updates the status flag of the memory block to non-writable through atomic operations to prevent data conflicts caused by other producers writing simultaneously. After the write is complete, the status flag is also updated to writable through atomic operations, allowing the memory block to be used by subsequent data write operations. Other producers also follow the same rules when accessing memory blocks, obtaining and updating the memory block status flag through the shared atomic variable area, achieving lock-free synchronous data write control.

[0078] This application's embodiments utilize a shared atomic variable region to maintain the status flags of multiple memory blocks. Leveraging the indivisible nature of atomic operations, lock-free synchronization is achieved, effectively avoiding latency fluctuations caused by lock contention in traditional locking mechanisms and improving the real-time performance and efficiency of data writing. Simultaneously, this design simplifies the synchronization mechanism, reduces system resource consumption and overhead, and ensures accurate maintenance of memory block status flags and orderly data writing in high-throughput, high-concurrency data interaction scenarios. This provides reliable technical support for fields such as autonomous driving, which have extremely high requirements for the real-time performance and stability of data transmission.

[0079] In some embodiments of this application, multiple memory blocks are indexed by key values. After writing data into the memory blocks of the contiguous physical memory pool based on a preset data writing strategy, the data interaction method further includes: storing the key value of the memory block in a shared atomic variable area as the key value of the published memory block.

[0080] Each memory block is organized in a circular queue. For ease of management and identification, a globally unique key value identifier can be established for each memory block. The key value identifier can be generated, for example, using a file path and a project ID. Of course, those skilled in the art can flexibly configure how to generate the key value identifier according to actual needs, and no specific limitations are made here.

[0081] After writing data to a memory block, the producer stores the key value of the memory block in a shared atomic variable area based on a key-value publishing mechanism. The shared atomic variable area is a specific storage region that guarantees the atomicity of operations on the data within it. The producer writes the key value of the most recently written memory block to this shared atomic variable area, using it as the key value of the published memory block. In this way, other processes or components that need to read data can obtain the key value of the most recently written memory block by accessing the shared atomic variable area, and then locate the corresponding memory block to perform data reading operations based on that key value.

[0082] This is achieved by establishing a unique key index for each memory block and storing the key value of the memory block in a shared atomic variable area based on a key-value publishing mechanism after data is written. This design enables efficient integration of data production and subscription processes, allowing subscribers to quickly and accurately obtain information about the memory block containing the latest data. Combined with previous technical solutions, this further improves the efficiency and real-time performance of data interaction, avoiding read delays and errors caused by unclear data locations. It provides a more comprehensive data interaction solution for scenarios such as autonomous driving, which have extremely high requirements for data real-time performance and accuracy, ensuring the overall high-performance operation of the system.

[0083] In some embodiments of this application, reading data from memory blocks of the contiguous physical memory pool via memory mapping includes: obtaining the key value of the published memory block from the shared atomic variable region through atomic operations; and reading data from the published memory block via memory mapping based on the key value of the published memory block.

[0084] During the data reading phase, subscribers first need to obtain the key value of the published memory block. Since the key value of the published memory block is stored in a shared atomic variable area, subscribers can retrieve this key value from this shared atomic variable area through atomic operations. Atomic operations ensure that the key value retrieval process is not disturbed in the event of concurrent access by multiple subscribers or multiple producer-subscriber pairs, and can accurately obtain the latest published valid key value.

[0085] After successfully obtaining the key value of the published memory block, the subscriber directly accesses the corresponding published memory block using memory mapping (mmap) technology based on this key value. Because the subscriber only has read permissions, it can only read data from the memory block and will not modify the data in the memory block, ensuring data security and integrity. Simultaneously, employing a lock-free state detection mechanism, the subscriber does not need to worry about the data being in an inconsistent state after being written, ensuring that the read data is complete and valid, achieving consistent data access.

[0086] This application's embodiments obtain the key value of the published memory block through atomic operations. Combined with memory mapping technology, subscriber read-only permissions, and a lock-free state detection mechanism, it achieves efficient, secure, and consistent data reading. Subscribers can quickly and accurately obtain the memory block containing the latest data and perform read operations, avoiding problems such as lock contention and data inconsistency that may occur in traditional reading methods. This further improves the efficiency and reliability of data interaction, providing a stable data reading solution for scenarios such as autonomous driving that have stringent requirements for data real-time performance and accuracy, ensuring the overall efficient operation of the system.

[0087] In some embodiments of this application, zero-copy memory mapping is employed for data transfer. The producer and subscriber processes use the mmap system call to map a pre-allocated shared memory pool to their respective user-space virtual addresses. This process is implemented through page table mapping under the operating system's virtual memory management mechanism. In this way, producers and subscribers can directly manipulate the data in the shared memory pool in user space. Data exchange between producers and subscribers does not require copying between kernel and user space, achieving zero-copy data transfer.

[0088] To avoid performance issues caused by redundant field padding during serialization in traditional systems like ROS, memory alignment and compact data layout (such as Packed Struct) are employed. Memory alignment is based on the principle of data alignment optimization, arranging data storage according to cache line alignment. Compact data layout, on the other hand, compresses data storage space as much as possible, removing unnecessary redundant fields to make data storage in memory more efficient.

[0089] By directly mapping physical memory and leveraging virtual memory management mechanisms to achieve zero-copy data transmission, and employing serialization optimization techniques and data alignment optimization to improve memory access efficiency, this approach effectively eliminates the performance overhead caused by serialization / deserialization and network buffer copying, significantly reducing CPU utilization and avoiding TCP / IP protocol stack overhead. The memory pass-through mode ensures that the end-to-end transmission latency for point cloud data transmission scenarios remains stable within 0.8ms, with a transmission latency of <1ms. This provides a highly efficient data transmission solution for scenarios with extremely high real-time and resource utilization requirements, such as autonomous driving, ensuring rapid system response and stable operation.

[0090] To facilitate understanding of the above embodiments, the specific implementation of the technical solution of this application will be further illustrated using a laser point cloud data transmission scenario as an example:

[0091] (1) Memory pool initialization:

[0092] The server creates 10 memory blocks.

[0093]

[0094] (2) Dynamic key distribution:

[0095] Publish the current memory block key

[0096] pub.publish(key=hash(time.time()%10))

[0097] (3) Data writing and reading:

[0098] Write end:

[0099] pointcloud_multiresult buffer;

[0100] buffer.multi_result.time=get_timestamp();

[0101] memcpy(buffer.data, lidar_data, 1600*1024); / / Fill the point cloud

[0102] server.Write(buffer.data,20971480);

[0103] Reader:

[0104] share_mem_client client;

[0105] client.Getviraddressfromkey(published_key); / / Dynamic key mapping

[0106] client.Read(received_data, 20971480); / / Zero-copy read

[0107] (4) Security verification mechanism:

[0108]

[0109] On the server side, to meet the needs of LiDAR point cloud transmission, 10 memory blocks are created. Through a loop operation, utilizing the `share_mem_server` function, a virtual address is obtained for each memory block. Each memory block is 2MB in size, and the storage path can be set to, for example, ` / home / mogo / data`. These memory blocks are organized in a circular queue, providing a fixed physical memory space for subsequent point cloud data storage and transmission, avoiding performance issues caused by dynamic memory allocation.

[0110] To ensure secure data transmission and accurate retrieval, a dynamic key distribution mechanism is employed. The server publishes the key for the current memory block, for example, using `pub.publish(key = hash(time.time() % 10))`, calculating a hash value based on the current time and publishing it as the key. This allows the reader to accurately locate the corresponding memory block based on the dynamic key, guaranteeing the security and orderliness of data transmission.

[0111] During the data writing phase, the point cloud data collected by the LiDAR is first processed and filled into a buffer of the `pointcloud_multiresult` structure, recording the current timestamp. Then, the point cloud data is written to the corresponding memory block using `server.Write(buffer.data, 20971480)`. Throughout the writing process, a memory block circular reuse strategy and a state flag maintenance mechanism based on atomic operations are followed to ensure efficient and accurate data writing.

[0112] During the data reading phase, subscribers use the `share_mem_client` function to map a dynamic key to the corresponding memory block, i.e., `client.Getviraddressfromkey(published_key)`. Then, utilizing memory mapping (zero-copy read) technology, they directly read point cloud data from the memory block using `client.Read(received_data,20971480)`, eliminating the need for data copying between kernel and user space and improving data reading efficiency.

[0113] To ensure the integrity and security of transmitted point cloud data, a security verification mechanism was designed. A compact data layout was used to define the SafetyHeader structure, which includes a CRC32 checksum and a timestamp. The CRC32 checksum is used to verify data integrity and prevent data tampering during transmission; the timestamp is used to prevent replay attacks and ensure that the received data is the latest and valid data.

[0114] In LiDAR point cloud transmission scenarios, a stable storage space is provided through memory pool initialization, dynamic key distribution ensures accurate data addressing, and the data writing and reading processes utilize circular reuse strategies, atomic operations, and zero-copy technology to improve efficiency. A security verification mechanism ensures data security and integrity. This effectively solves the problems of high latency, low throughput, and data security in LiDAR point cloud transmission, achieving efficient, real-time, and secure point cloud data transmission, meeting the stringent requirements of LiDAR data transmission in scenarios such as autonomous driving.

[0115] This application also provides a data interaction system 300, such as... Figure 3 As shown, a schematic diagram of a data interaction system according to an embodiment of this application is provided. The data interaction system 300 includes: a memory management unit 310, a data writing unit 320, and a data reading unit 330, wherein:

[0116] The memory management unit 310 is used to pre-allocate a fixed-capacity contiguous physical memory pool during the system initialization phase. The contiguous physical memory pool contains multiple memory blocks organized in the form of a circular queue.

[0117] The data writing unit 320 is used to write data into the memory blocks of the continuous physical memory pool based on a preset data writing strategy.

[0118] The data reading unit 330 is used to read data from memory blocks of the contiguous physical memory pool through memory mapping.

[0119] In some embodiments of this application, the data interaction system 300 is provided with a standardized interface, which is used to embed the data interaction system into an external system framework.

[0120] To achieve good compatibility and ease of use, standardized interface encapsulation and compatibility design were adopted. Specifically, a multi-framework adaptation layer was designed, providing API interfaces compatible with mainstream robot frameworks such as ROS1 / 2 and CyberRT, such as shm_publish() and shm_subscribe(). Internally, the multi-framework adaptation layer supports automatic data format conversion, for example, converting the sensor_msgs / PointCloud2 data format in ROS to a compact format for memory pools. This design packages related functionalities into a tool-like class, allowing external systems to easily embed the data interaction system into any middleware by calling these interfaces. When an external system needs to use the data interaction system, it only needs to call the corresponding interface, and the multi-framework adaptation layer will automatically handle data format conversion and other issues, enabling the data interaction system to work collaboratively with external system frameworks.

[0121] By setting up standardized interfaces and designing a multi-framework adaptation layer, seamless integration between the data interaction system and various mainstream robot frameworks is achieved. On the one hand, this enables seamless migration of existing systems without modifying business logic code, greatly reducing the difficulty and workload of system integration. On the other hand, replacing the hybrid architecture with a single communication channel reduces system complexity, resource overhead, and memory usage, improving system resource utilization efficiency and providing an efficient and highly compatible solution for data interaction in complex systems such as autonomous driving.

[0122] In some embodiments of this application, the preset data writing strategy includes a memory block circular reuse strategy, and the data writing unit 320 is specifically used to: write the data into the memory blocks of the continuous physical memory pool according to the organization order among multiple memory blocks based on the memory block circular reuse strategy.

[0123] In some embodiments of this application, the data writing unit 320 is specifically used to: determine the status flag of the memory block to be written based on the circular reuse strategy of the memory block; write the data into the memory block when the status flag of the memory block is writable, and update the status flag of the memory block to an unwritable state through an atomic operation; after the writing is completed, update the status flag of the memory block to a writable state through an atomic operation.

[0124] In some embodiments of this application, the status flags of the multiple memory blocks are maintained through a shared atomic variable region.

[0125] In some embodiments of this application, multiple memory blocks are indexed by key values. The data interaction system 300 further includes a storage unit, which stores the key value of the memory block in a shared atomic variable area after writing data into the memory blocks of the continuous physical memory pool based on a preset data writing strategy, as the key value of the published memory block.

[0126] In some embodiments of this application, the data reading unit 330 is specifically used to: obtain the Key value of the published memory block from the shared atomic variable region through atomic operations; and read data from the published memory block through memory mapping based on the Key value of the published memory block.

[0127] It is understood that the above-described data interaction system can implement each step of the data interaction method provided in the foregoing embodiments. The relevant explanations of the data interaction method are applicable to the data interaction system and will not be repeated here.

[0128] Figure 4 This is a schematic diagram of the structure of a device according to an embodiment of this application. For example... Figure 4 As shown, the device includes one or more processors (or processing units), and may also include one or more memories coupled to the processors, and may also include a communication module coupled to the processors.

[0129] A communication module can be used to communicate with other devices or apparatuses, such as sending or receiving data and / or signals. A communication module may have at least one communication module for communication. A communication module may include any interface necessary for communicating with other devices. Exemplarily, a communication module may be a transceiver, circuit, bus, module, or other type of communication module.

[0130] The processor may include, but is not limited to, one or more of the following: a general-purpose computer, a special-purpose computer, a microcontroller, a digital signal processor (DSP), or a controller-based multi-core controller architecture. The device may have multiple processors, such as application-specific integrated circuit (ASIC) chips, which are time-dependent on a clock synchronized with the main processor.

[0131] The memory may include one or more non-volatile memories and one or more volatile memories. Examples of non-volatile memories include, but are not limited to, at least one of the following: read-only memory (ROM), electrically programmable read-only memory (EPROM), flash memory, hard disk, compact disc (CD), digital video disc (DVD), or other magnetic and / or optical storage. Examples of volatile memories include, but are not limited to, at least one of the following: random access memory (RAM), or other volatile memories that do not persist during the duration of a power outage.

[0132] A computer program consists of computer-executable instructions that are executed by an associated processor. Programs can be stored in ROM. A processor can perform any appropriate action and processing by loading the program into RAM.

[0133] Possible implementations of this application can be achieved through a program, enabling the communication device to execute any of the processes discussed in the foregoing embodiments. Possible implementations of this application can also be achieved through hardware or a combination of software and hardware.

[0134] In some implementations, the program may be tangibly contained in a computer-readable storage medium, which may include in a device (such as in memory) or other storage device accessible by the device. The program may be loaded from the computer-readable storage medium into RAM for execution. The computer-readable storage medium may include any type of tangible non-volatile memory, such as ROM, EPROM, flash memory, hard disk, CD, DVD, etc.

[0135] This application also provides a computer-readable storage medium storing computer instructions or program code thereon, which, when executed by a processor, causes the processor to perform the methods and functions involved in any of the above embodiments. A computer-readable medium can be any tangible medium that contains or stores a program for or relating to an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. More detailed examples of computer-readable storage media include electrical connections with one or more wires, magnetic media (e.g., disks, floppy disks, hard disks, magnetic tapes, magnetic storage devices), optical media (e.g., optical storage devices, DVDs), semiconductor media (e.g., solid-state drives), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), or any suitable combination thereof.

[0136] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. Embodiments of this application also provide at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. This computer program product includes one or more computer-executable instructions, such as instructions included in a program module, which execute in a device on a target real or virtual processor to perform the processes, methods, and functions involved in any of the above embodiments. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0137] This application also proposes a computer program product, including a computer program or instructions that, when run on a computer, cause the computer to perform the processes, methods, and functions described in the above embodiments. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided as needed. The machine-executable instructions for the program modules can be executed locally or in a distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0138] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software, which can be executed by a controller, microprocessor, or other computing device. Although various aspects of the embodiments of this disclosure are shown and described as block diagrams, flowcharts, or represented using some other illustration, it should be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0139] It should be noted that although embodiments of this application have been described above with reference to the accompanying drawings, these embodiments are not independent of each other, and they can be combined to obtain other embodiments. The methods, situations, categories, and classifications of embodiments in this application are only for the convenience of description and should not constitute a special limitation. Various methods, categories, situations, and features in embodiments can be combined with each other if logically consistent. The various embodiments of this application can be arbitrarily combined to achieve different technical effects. The embodiments of this application will not list various combinations.

[0140] Furthermore, although the operation of the methods of this disclosure is described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to this disclosure may be embodied in one device. Conversely, the features and functions of one device described above may be further divided and embodied by multiple devices.

[0141] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0142] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data interaction method, characterized in that, The data interaction method is executed by the data interaction system, and the data interaction method includes: During the system initialization phase, a fixed-capacity contiguous physical memory pool is pre-allocated, which contains multiple memory blocks organized in a circular queue. Based on a preset data writing strategy, data is written into the memory blocks of the continuous physical memory pool; Data is read from memory blocks of the contiguous physical memory pool via memory mapping.

2. The data interaction method according to claim 1, characterized in that, The preset data writing strategy includes a memory block circular reuse strategy, and the step of writing data into the memory blocks of the contiguous physical memory pool based on the preset data writing strategy includes: Based on the memory block cyclic reuse strategy, the data is written into the memory blocks of the contiguous physical memory pool according to the organization order among multiple memory blocks.

3. The data interaction method according to claim 2, characterized in that, The circular reuse strategy based on the memory blocks, which writes the data into the memory blocks of the contiguous physical memory pool according to the organizational order among multiple memory blocks, includes: Based on the circular reuse strategy of the memory blocks, determine the status flag of the memory block to be written. When the state flag of the memory block is writable, the data is written to the memory block, and the state flag of the memory block is updated to non-writable through an atomic operation; After the write operation is completed, the status flag of the memory block is updated to a writable state through an atomic operation.

4. The data interaction method according to claim 3, characterized in that, The status flags of the multiple memory blocks are maintained through a shared atomic variable region.

5. The data interaction method according to claim 1, characterized in that, The multiple memory blocks are each indexed by a key value. After writing data into the memory blocks of the contiguous physical memory pool based on a preset data writing strategy, the data interaction method further includes: The key value of the memory block is stored in the shared atomic variable area as the key value of the published memory block.

6. The data interaction method according to claim 5, characterized in that, The step of reading data from memory blocks of the contiguous physical memory pool via memory mapping includes: The key value of the published memory block is obtained from the shared atomic variable region through atomic operations; Data is read from the published memory block using memory mapping based on the Key value of the published memory block.

7. A data interaction system, characterized in that, The data interaction system includes: The memory management unit is used to pre-allocate a fixed-capacity contiguous physical memory pool during the system initialization phase. The contiguous physical memory pool contains multiple memory blocks organized in a circular queue. The data writing unit is used to write data into the memory blocks of the continuous physical memory pool based on a preset data writing strategy. A data reading unit is used to read data from memory blocks of the contiguous physical memory pool via memory mapping.

8. The data interaction system according to claim 7, characterized in that, The data interaction system is equipped with a standardized interface, which is used to embed the data interaction system into an external system framework.

9. An apparatus comprising: processor; And a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform any of the data interaction methods of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the data interaction method according to any one of claims 1 to 7.

Images