Data transmission method and related device

Abstract

This invention discloses a data transmission method and related equipment. The data transmission method includes: acquiring multimodal sensing data from a sensor module; encapsulating the multimodal sensing data to obtain a protocol frame; wherein the protocol frame includes a frame header, metadata area, and payload area with consecutive addresses in the physical memory of the transmitting end; and sending the protocol frame to the receiving end through a physical communication interface. Addressing the timestamp discrepancies and timing inconsistencies caused by the transmission of multimodal sensing data in separate channels in existing technologies, the transmitting end data transmission method provided in this application encapsulates low-bandwidth, structured first-type data and high-bandwidth, unstructured second-type data into the same protocol frame, and implements consecutive address arrangements (frame header, metadata area, payload area) in the transmitting end's memory, effectively solving the timing inconsistency problem caused by differences in bandwidth, protocol stack, and driver between heterogeneous interfaces.

Description

Technical Field

[0001] This invention relates to the field of data transmission technology, and in particular to data transmission methods and related equipment. Background Technology

[0002] With the evolution of artificial intelligence and perception technologies, multimodal data fusion has become a core technology for achieving high-precision perception and decision-making in fields such as biometrics, autonomous driving, and industrial robotics. In these application scenarios, systems typically need to simultaneously acquire and process heterogeneous data from multiple sources, including high-resolution images, Time-of-Flight (ToF) depth information, Inertial Measurement Unit (IMU) data, and temperature information. The high consistency of these data in both time and space dimensions is a crucial prerequisite for achieving functions such as target recognition, behavior analysis, and environmental modeling.

[0003] However, existing technologies typically employ a separate transmission architecture: for example, image data is primarily transmitted via the MIPI CSI-2 interface or a USB interface based on the UVC (USB Video Class) protocol, while auxiliary sensor data is transmitted via interfaces such as I2C, UART, or HID. Due to differences in physical bandwidth, transmission mechanisms, and protocol stack implementations among these interfaces, various types of data are often transmitted in independent channels and processed separately by their respective driver modules. This separate architecture results in a significant timestamp discrepancy between the image data received by the main control unit and the sensor data, making precise alignment on the timeline difficult.

[0004] Due to fundamental differences in physical bandwidth, transmission mechanisms, and protocol stack implementations, heterogeneous interfaces transmit various types of data independently through different physical channels, and are received and processed separately by different driver modules. Under this architecture, the images received by the main control unit and the data from auxiliary sensors often have significant timestamp discrepancies, making it impossible to achieve high-precision alignment of multi-source data on the timeline.

[0005] Therefore, the separation transmission mechanism based on heterogeneous protocols has inherent defects in terms of time synchronization and consistency, making it difficult to meet the application requirements of strict time alignment in high-precision, multi-modal fusion scenarios. Summary of the Invention

[0006] To address the technical problems existing in the prior art, the present invention aims to provide a data transmission method and related equipment that meet the application requirements of strict timing alignment in high-precision, multimodal fusion scenarios.

[0007] The objective of this invention is achieved through the following technical solution:

[0008] In a first aspect, this application provides a data transmission method applied at a sending end, the method comprising:

[0009] Acquire multimodal sensing data from a sensor module. The multimodal sensing data has a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, and the second type of data is unstructured sensor array data or raw binary code stream.

[0010] The multimodal sensing data is encapsulated to obtain a protocol frame; wherein, the protocol frame includes a frame header, a metadata area, and a payload area with contiguous addresses in the physical memory of the transmitting end; the metadata area carries the first type of data, and the payload area carries the second type of data;

[0011] The protocol frame is sent to the receiving end through the physical communication interface.

[0012] This application provides a data transmission method that addresses the issues of timestamp discrepancies and timing asynchrony caused by the transmission of multimodal sensor data in separate channels in the prior art. The data transmission method provided by this application encapsulates low-bandwidth, structured first-type data and high-bandwidth, unstructured second-type data into the same protocol frame and implements continuous address arrangement (frame header, metadata area, payload area) in the sender's memory, effectively solving the timing inconsistency problem caused by differences in bandwidth, protocol stack, and driver between heterogeneous interfaces.

[0013] Specifically, through the logical partitioning design of the frame header, metadata area, and payload area, low-bandwidth sensor data can be stored promptly in the metadata area, while high-bandwidth data is uniformly transmitted through the payload area. This enables continuous and unified layout of multimodal data in the physical memory of the transmitting end, thereby reducing the time delay and uncertainty introduced by separate data transmission. Furthermore, the uniformly encapsulated protocol frames facilitate subsequent transmission through a single physical communication interface, simplifying the transmitting end's driving logic and achieving efficient scheduling and synchronous transmission of heterogeneous data. This improves the timing consistency and fusion reliability of multimodal data at the receiving end, thus meeting the application requirements of high-precision sensing systems for strict time synchronization.

[0014] In one possible implementation, encapsulating the multimodal sensing data to obtain a protocol frame specifically includes:

[0015] The first type of data is converted using a schema-based extensible serialization format to generate the corresponding serialized stream;

[0016] Identify the byte length value of the serialized stream and write the byte length value into the metadata length field of the frame header;

[0017] The starting offset address of the load area in physical memory is determined based on the metadata length field and the preset memory alignment granularity.

[0018] The second type of data is stored directly in the physical memory area corresponding to the starting offset address in the form of raw binary large objects;

[0019] The frame header, serialized stream, and raw binary data are sequentially integrated into a single protocol frame in physical memory.

[0020] In one possible implementation, the data transmission method further includes:

[0021] At the trigger moment when the sensor module acquires multimodal sensing data, the current value of the free-running counter at the transmitting end is read and used as a hardware timestamp;

[0022] The hardware timestamp is treated as the first type of data and encapsulated in the metadata area.

[0023] In one possible implementation, the data transmission method further includes:

[0024] The hardware security chip at the sending end is invoked to encrypt the metadata area based on the negotiated session key and generate a security verification tag.

[0025] Write the security verification tag into the frame header;

[0026] The original binary large object in the payload area is obfuscated or stream-encrypted.

[0027] In one possible implementation, the data transmission method further includes:

[0028] Before encapsulating the protocol frame, obtain the value of the rolling counter at the sending end;

[0029] The hardware security chip is invoked to sign the value of the rolling counter;

[0030] The signed rolling counter value is written as the frame sequence number into the frame header of the protocol frame;

[0031] The security verification tag is calculated based on the frame header and the encrypted metadata area, and the data obfuscation seed is generated based on the session key and the rolling counter value.

[0032] In one possible implementation, the frame header is a fixed-length structure, and the frame header includes at least a synchronization word, a metadata length field, a frame sequence number, and a security check tag;

[0033] The synchronization word is a preset fixed byte sequence used by the receiving end to identify the start boundary of the protocol frame; the security verification tag is an authentication code encrypted in the metadata area, used by the receiving end to verify the integrity of the protocol frame.

[0034] Secondly, this application provides a data transmission method applied at a receiving end, the data transmission method comprising:

[0035] Protocol frames are received from the transmitting end through a physical communication interface. The protocol frames are obtained by encapsulating multimodal sensing data. The protocol frames in the physical memory of the transmitting end include a frame header, a metadata area, and a payload area with contiguous addresses. The frame header includes a metadata length field and a security check tag. The metadata area carries structured sensing data with semantic tags in the multimodal sensing data. The payload area carries unstructured sensor array data or raw binary code streams in the multimodal sensing data.

[0036] Parse the header of the protocol frame and extract the metadata length field and security check tag from the header;

[0037] Based on the metadata length field, locate the boundary between the metadata area and the payload area in the current protocol frame;

[0038] The metadata area is decrypted and its integrity is verified using a security verification tag.

[0039] When the integrity verification passes, the first type of data is parsed from the metadata area, and the payload area is directly mapped to the system memory or video memory of the receiving end through direct memory access (DMA).

[0040] This application provides a data transmission method in the second aspect, addressing the problems of difficulty in accurately aligning multi-source data and the separation and complexity of low-bandwidth and high-bandwidth data processing paths in the prior art. The data transmission method provided in this application obtains the metadata length field and security verification tag by parsing the protocol frame header, thereby achieving accurate partitioning and synchronous processing of low-bandwidth structured data and high-bandwidth unstructured data.

[0041] Specifically, the receiver uses the frame header field to precisely locate the boundary between the metadata area and the payload area. This allows low-bandwidth Class I data to be directly parsed after security verification, while high-bandwidth Class II data can be directly mapped to system memory or video memory via DMA, eliminating the need for additional copy operations. This method ensures the integrity, security, and time consistency of multimodal data at the receiver, while avoiding buffering delays and inter-driver conflicts caused by traditional multi-channel reception. Through a unified protocol frame structure and security verification mechanism, the receiver can efficiently and reliably acquire all sensor data, providing an accurate and low-latency data foundation for subsequent multimodal fusion, environmental modeling, and target recognition, thereby significantly improving the overall perception accuracy and real-time performance of the system.

[0042] In one possible implementation, the frame header further includes a frame sequence number, and the data transmission method further includes:

[0043] Parse the header of the protocol frame and extract the frame sequence number from the header;

[0044] If the current frame number is less than or equal to the historical frame number, it is determined that the frame number is rolled back or repeated.

[0045] If the difference between the current frame number and the historical frame number exceeds the preset sequence number step size, it is determined to be an abnormal jump in frame number.

[0046] If a frame sequence number rollback, duplication, or abnormal jump is detected, the current protocol frame is discarded.

[0047] In one possible implementation, the data transmission method includes:

[0048] Parse the metadata area of ​​the current protocol frame and extract the hardware timestamp from the metadata area;

[0049] Calculate the globally unified time based on the preset linear regression model and the hardware timestamp;

[0050] The global unified time is used as the common timestamp for the first type of data and the second type of data within the current protocol frame, and the first type of data and the second type of data are spatiotemporally aligned.

[0051] In one possible implementation, the preset linear regression model is established by the following method:

[0052] It periodically sends clock synchronization request messages to the sender and records the first local timestamp at the time of transmission;

[0053] Receive the synchronization response message returned by the sender, record the second local timestamp at the time of reception, and extract the hardware timestamp at the time when the sender received the clock synchronization request message from the synchronization response message;

[0054] Calculate the round-trip transmission delay of the message based on the first local timestamp and the second local timestamp;

[0055] Calculate the clock offset based on the first local timestamp, the second local timestamp, and the hardware timestamp;

[0056] Based on multiple consecutive clock offsets, a linear regression model is established between the hardware time of the transmitting end and the system time of the receiving end.

[0057] Thirdly, this application provides a data transmission device for transmitting data, comprising:

[0058] The acquisition module is used to acquire multimodal sensing data from the sensor module. The multimodal sensing data has a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, and the second type of data is unstructured sensor array data or raw binary code stream.

[0059] An encapsulation module is used to encapsulate the multimodal sensing data to obtain a protocol frame; wherein, the protocol frame includes a frame header, a metadata area, and a payload area with contiguous addresses in the physical memory of the transmitting end; the metadata area carries the first type of data, and the payload area carries the second type of data;

[0060] The transmitting module is used to transmit the protocol frame to the receiving end through a physical communication interface.

[0061] Fourthly, this application provides a data transmission device for receiving data, comprising:

[0062] A receiving module is used to receive protocol frames from a transmitting end through a physical communication interface. The protocol frames are obtained by encapsulating multimodal sensing data. The protocol frames in the physical memory of the transmitting end include a frame header, a metadata area, and a payload area with contiguous addresses. The frame header includes a metadata length field and a security check tag. The metadata area carries structured sensing data with semantic tags in the multimodal sensing data. The payload area carries unstructured sensor array data or raw binary code streams in the multimodal sensing data.

[0063] The extraction module is used to parse the frame header of the protocol frame and extract the metadata length field and security check tag of the frame header;

[0064] The positioning module is used to locate the boundary between the metadata area and the payload area in the current protocol frame based on the metadata length field.

[0065] The verification module is used to decrypt and verify the integrity of the metadata area using a security verification tag;

[0066] The parsing module is used to parse the first type of data from the metadata area when the integrity verification passes, and to directly map the payload area to the system memory or video memory of the receiving end through direct memory access (DMA).

[0067] Fifthly, this application provides a computer-readable storage medium having program instructions stored thereon, which, when executed by a computer, cause the computer to perform the data transmission method in any possible implementation of the first or second aspect.

[0068] Sixthly, this application provides a computer program including program instructions that, when executed by a computer, cause the computer to perform the data transmission method in any possible implementation of the first or second aspect. Attached Figure Description

[0069] The following description, with reference to the accompanying drawings, further illustrates the various features of this application and the relationships between them. The drawings are exemplary; some features are not shown to scale, and some drawings may omit conventional features in the field of this application that are not essential to it, or additional features that are not essential to this application may be shown. The combination of features shown in the drawings is not intended to limit this application. Furthermore, throughout this specification, the same reference numerals refer to the same things. Specific descriptions of the drawings are as follows:

[0070] Figure 1 This is a schematic diagram illustrating an exemplary application scenario according to an embodiment of this application;

[0071] Figure 2 This is a schematic flowchart of a data transmission method according to an embodiment of this application;

[0072] Figure 3 This is a schematic diagram of the process of obtaining a protocol frame by encapsulating multimodal sensing data according to an embodiment of this application;

[0073] Figure 4 This is a schematic diagram illustrating the process of recording a hardware timestamp at the trigger moment according to an embodiment of this application;

[0074] Figure 5 This is a flowchart illustrating the security enhancement steps according to an embodiment of this application;

[0075] Figure 6 This is a flowchart illustrating a data transmission method according to another embodiment of this application;

[0076] Figure 7 This is a schematic diagram of a process for calculating a globally unified time based on a hardware timestamp and completing spatiotemporal alignment of multimodal data according to another embodiment of this application;

[0077] Figure 8 This is a schematic diagram of the structure of a transmitting device according to an embodiment of this application;

[0078] Figure 9 This is a schematic diagram of the structure of a receiving device according to an embodiment of this application;

[0079] Figure 10 This is a schematic diagram of the structure of a data transmission system according to an embodiment of this application. Detailed Implementation

[0080] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0081] It should be understood that the terms "first," "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0082] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0083] Multimodal sensing systems are widely used in fields such as biometrics (e.g., palm vein recognition, iris recognition), autonomous driving, and industrial robotics. To achieve high-precision perception and decision-making, these systems typically need to simultaneously acquire high-resolution image data and auxiliary information from different types of sensors, such as time-of-flight (ToF) distance data, inertial measurement unit (IMU) data, and temperature information.

[0084] However, existing technologies have significant shortcomings in multimodal data transmission and processing:

[0085] First, the issue of protocol fragmentation is prominent. Image data is typically transmitted via the MIPI CSI-2 interface or an interface based on the USB Video Class (UVC) protocol, while low-bandwidth auxiliary sensor data is transmitted via I2C, UART, or HID interfaces. Due to the differences in physical bandwidth, transmission mechanisms, and protocol stack implementations among these interfaces, the image data received by the master control unit and the auxiliary sensor data are difficult to align precisely on the timeline, resulting in timestamp inconsistencies. This significantly impacts the sensing accuracy and real-time performance of the multimodal fusion algorithm.

[0086] Secondly, there is insufficient scalability. The existing standard UVC protocol has limited support for custom metadata, and adding new sensor types often requires modifying the driver, which restricts the system's ability to expand and upgrade its functions, making it difficult to flexibly adapt to diverse sensor configurations.

[0087] Secondly, bandwidth utilization is low. Traditional network transmission protocols (such as GigE Vision and RTSP) have significant overhead in terms of header overhead and memory copy counts, making them unsuitable for the stringent low-latency requirements of embedded systems and limiting the system's performance in real-time sensing scenarios.

[0088] Finally, it lacks inherent security. Universal USB data transfer is typically in plaintext, lacking security mechanisms such as tamper-proofing and replay protection for biometric information, making it difficult to meet the stringent data integrity and security requirements of automotive or financial applications.

[0089] Therefore, existing technologies have inherent defects in multimodal data synchronization, protocol scalability, transmission efficiency, and data security, which bring significant performance limitations to the practical application of high-precision, multimodal sensing systems.

[0090] To address this issue, this application provides a data transmission method that solves the problems of timestamp discrepancies and timing asynchrony caused by the transmission of multimodal sensor data in separate channels in the prior art. The data transmission method provided by this application encapsulates low-bandwidth, structured first-type data and high-bandwidth, unstructured second-type data into the same protocol frame and implements continuous address arrangement (frame header, metadata area, payload area) in the sender's memory, effectively solving the timing inconsistency problem caused by differences in bandwidth, protocol stack, and driver between heterogeneous interfaces.

[0091] Figure 1 A schematic diagram illustrating an exemplary application scenario according to an embodiment of this application is shown. Those skilled in the art should understand that... Figure 1 The application scenario shown is only one exemplary scenario to which the solution of this application can be applied. In addition to... Figure 1Beyond the scenarios shown, the solution presented in this application can also be applied to any other suitable application scenarios, such as, but not limited to, biometric terminals, intelligent monitoring equipment, and industrial robot work units.

[0092] Figure 1 The exemplary application scenario shown is an embedded multimodal sensing system, which includes a host device 110 and sensor modules 120. The host device 110 establishes a connection with one or more sensor modules 120 via a high-speed serial bus (such as USB 3.0 or high-speed differential harness HSD) to realize unified acquisition and processing of multimodal data.

[0093] like Figure 1 As shown, the sensor module 120 includes a CMOS image sensor 121, multiple auxiliary sensors 122 (e.g., ToF sensors, IMU), a security chip 123 (SE), a protocol processing unit 124 (e.g., FX3 or FPGA), and an LED driver circuit. The sensor module 120 is responsible for collecting multi-source heterogeneous data and, through the protocol processing unit 124, encapsulating the image data and auxiliary sensor data into a unified protocol frame before sending it to the main control device 110. Auxiliary sensor data may include distance information, inertial measurement information, and temperature information, while image data includes high-resolution color or grayscale images. The security chip 123 provides data encryption and anti-tampering capabilities, ensuring the integrity and security of biometric or industrial sensor data during transmission.

[0094] The main control device 110 includes a USB controller 111, a driver parsing module 112, and an upper-layer application 113. The USB controller 111 manages high-speed serial communication with the sensor module 120. The driver parsing module 112 parses, verifies, and partitions the received protocol frames, extracting low-bandwidth, structured sensor data into system memory, while simultaneously mapping high-bandwidth unstructured data directly to system video memory or a memory buffer via DMA. The upper-layer application 113 uses this multimodal data for subsequent processing, such as biometric recognition, environmental modeling, or robot control.

[0095] In this exemplary application scenario, the master control device 110 acts as the system's host, responsible for managing and scheduling the resources of the sensor modules 120. Each sensor module 120 acts as a system slave node. The sensor module 120 itself acts as the data transmission transmitter. Through unified protocol frame encapsulation, multi-channel synchronous transmission, and security verification mechanisms, the system can achieve high-precision fusion of multi-source data while ensuring low latency and data integrity, providing a reliable data foundation for subsequent sensing and control.

[0096] The method embodiments provided in this application will be described in detail below.

[0097] Figure 2 This is a schematic flowchart of a data transmission method provided in this application. Figure 2 The data transmission method shown is used to send data, and can be performed by a sending end or a sending device (see...). Figure 8 This can be implemented using a sensor module 120 or any other suitable device. For example, the sensor module 120 itself acts as the transmitter, or transmitting device, of the data transmission method. In another example, the transmitter, or transmitting device, includes at least one... Figure 1 The sensor module 120 is shown. Specifically, the data transmission method includes steps S210-S230.

[0098] S210: Acquire multimodal sensing data from the sensor module. The multimodal sensing data has a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, and the second type of data is unstructured sensor array data or raw binary code stream.

[0099] S220: Encapsulate the multimodal sensing data to obtain a protocol frame; wherein, the protocol frame includes a frame header, metadata area and payload area with consecutive addresses in the physical memory of the transmitting end; the metadata area carries the first type of data and the payload area carries the second type of data.

[0100] S230: Send the protocol frame to the receiving end through the physical communication interface.

[0101] Specifically, the steps S210-S230 of the data transmission method are explained in detail below.

[0102] S210: Acquire multimodal sensing data from the sensor module. The multimodal sensing data has a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, and the second type of data is unstructured sensor array data or raw binary code stream.

[0103] In some embodiments, the sensor module acquires multimodal sensing data, including a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, such as IMU acceleration, gyroscope data, ToF distance measurement, ambient light sensor information, temperature data, or physiological parameters (such as pulse wave, skin resistance). The second type of data is unstructured sensor array data or raw binary code streams, such as CMOS RGB images, infrared images, or high-resolution point cloud data.

[0104] For example, the first type of data is typically represented as environmental scalar data or system state data. Examples include, but are not limited to: distance information characterizing target depth (such as single-point or low-resolution floating-point ranging scalars output by a time-of-flight (ToF) sensor), illuminance or brightness data characterizing ambient light characteristics (such as real-time Lux values ​​collected by an ambient light sensor (ALS), temperature data reflecting the physical operating environment (such as temperature status values ​​output by a thermistor), and spatial attitude data characterizing the three-dimensional kinematics of the device or monitored target (such as high-frequency three-axis accelerations and fixed-length arrays of gyroscope quaternions output by an inertial measurement unit (IMU)).

[0105] For example, the second type of data includes, but is not limited to: image pixel matrix output by the visual imaging unit (such as uncompressed RAW format raw data stream continuously output by a CMOS image sensor after the exposure cycle ends, or RGB / YUV format video frames), and laser point cloud data output by the spatial scanning unit (such as large-scale, disordered three-dimensional spatial floating-point coordinate stream generated by a 3D solid-state LiDAR).

[0106] In some examples, sensor modules such as Figure 1 As shown, the sensor module (Node) integrates multiple physical sensing units, and the multimodal sensing data it generates is divided into a first category and a second category based on data characteristics. Specifically, the first category of data is structured sensing data with semantic tags, such as, but not limited to: hardware timestamps, inertial measurement unit (IMU) data, time-of-flight (ToF) distance information, ambient light intensity, device temperature, and sensor configuration parameters. This type of data typically has a fixed data structure and requires extremely high real-time alignment.

[0107] In some implementations, the second type of data is unstructured sensor array data or raw binary streams, such as, but not limited to, raw format pixel data acquired by CMOS image sensors, YUV image streams, JPEG compressed streams, or large-scale point cloud data generated by LiDAR. This type of data is characterized by its massive volume (typically in matrix form) and extremely high requirements for transmission bandwidth and processing performance.

[0108] It is understandable that in practical applications, the specific way to divide the first type of data and the second type of data is not limited to the above example, and can also be flexibly defined according to data bandwidth usage, data structure form or application requirements.

[0109] S220: Encapsulate the multimodal sensing data to obtain a protocol frame; wherein, the protocol frame includes a frame header, metadata area and payload area with consecutive addresses in the physical memory of the transmitting end; the metadata area carries the first type of data and the payload area carries the second type of data.

[0110] To improve the processing efficiency of data within the sending end and the continuity of transmission on the physical link, the embodiments of this application optimize the layout of protocol frames in physical memory.

[0111] In some implementations, the protocol frame includes a header, a metadata region, and a payload region in the physical memory of the sending end.

[0112] Specifically, the protocol processing unit at the transmitting end (such as an FPGA, DSP, or dedicated interface chip) encapsulates the two types of data according to a specific hybrid payload protocol. To achieve extremely high transmission efficiency and reduce system overhead, this application uses memory mapping technology to present the protocol frame in a contiguous "three-segment" layout in the physical memory of the transmitting end, specifically including:

[0113] 1. Frame Header: Located at the beginning of contiguous memory, it has a fixed byte length (e.g., 16 bytes). For example, the frame header includes a synchronization word (Magic Code), a metadata length identifier (MetaLen), a frame sequence number (FrameID), and a security check tag (CRC / Tag) for integrity verification. This fixed-length design facilitates quick identification of data boundaries by the hardware DMA (Direct Memory Access) controller.

[0114] Understandably, the synchronization word is a preset fixed byte sequence used by the receiving end to identify the start boundary of the protocol frame; the security check tag is an authentication code encrypted in the metadata area, used by the receiving end to verify the integrity of the protocol frame.

[0115] 2. Metadata Area: Following the frame header, this area carries the first type of data. In this embodiment, the metadata area is stored using a serialization encoding format (such as Cap'n Proto or Protobuf). Through a structured schema definition, the master control unit can quickly extract sensor attributes and timestamp information without parsing the entire image.

[0116] 3. Payload Area: Located in the address space following the metadata area, it carries the second type of data. The payload area stores the raw binary large objects (BLOBs) of the second type of data, which are not serialized.

[0117] Optionally, a padding area is provided between the metadata area and the payload area to ensure that the starting offset address of the payload area meets the preset byte alignment requirements (such as 8-byte or 16-byte alignment), thereby optimizing the DMA transfer speed and achieving "zero-copy" performance from the module end to the master end memory.

[0118] In this embodiment, the metadata area is used to carry the aforementioned first type of data, and the payload area is used to carry the aforementioned second type of data. By placing the structured sensor data with semantic tags (first type of data) in the metadata area immediately adjacent to the frame header, and placing the unstructured sensor array data or raw binary bitstream (second type of data) in the payload area, efficient classification and indexing of sensor data can be achieved.

[0119] Furthermore, since the frame header, metadata area, and payload area are located contiguously in physical memory, the sending end can move the entire protocol frame from memory to the communication interface buffer in a single bus transaction without performing multiple scatter-gather operations during DMA (Direct Memory Access) operations. This contiguous address encapsulation method significantly reduces processor interrupt frequency and bus overhead, and can effectively reduce data processing latency, especially when processing high-bandwidth multimodal data.

[0120] Specifically, this application also provides the following examples of sensor data attribution comparison, please refer to the table below.

[0121]

[0122] In this embodiment, to fully illustrate the adaptive dynamic adjustment capability of the present invention in complex application scenarios, an example of microscopic data flow across two consecutive frames is provided, combined with... Figure 1 The embedded multimodal perception system shown has two consecutive frames, Frame N and Frame N+1, exemplarily applied to a dual-modal sensor module in medical surgical robots or advanced autonomous driving scenarios. The module uses the Omni-Link protocol for multimodal data transmission, with 60fps real-time scanning as an example.

[0123] Regarding the acquisition and transmission of normal frames (Frame N), in some embodiments, Frame N is in a normal environment, the target distance is within a safe range, and the module outputs a standard RGB image.

[0124] For example, the microstructure of a data flow is as follows:

[0125] Header segment (8 Bytes);

[0126] Magic Code: 0x50414C4D ("PALM");

[0127] Meta_Length: 0x0040 (64 bytes);

[0128] Metadata segment (64 bytes, serialized data);

[0129] Frame_ID: 1000;

[0130] Timestamp: 1683012345000 (microseconds);

[0131] ToF_Distance: 350.5 mm → Safe distance;

[0132] Lux_Light: 450.0 Lux → Normal lighting;

[0133] Image_Format: 0x01 (RGB888);

[0134] Payload_Size: 0x0E1000 (921,600 Bytes, corresponding to 640x480 RGB);

[0135] Payload segment (921,600 Bytes);

[0136] Raw Data: [R, G, B, R, G, B…] → CMOS exposure pixel stream, no protocol header;

[0137] Specifically, the Header provides hardware-level parsing roadmaps, the Metadata carries structured sensor data with semantic tags, and the Payload carries a high-bandwidth image matrix, enabling zero-copy DMA transfer.

[0138] Regarding the acquisition and adaptive adjustment of sudden environmental frames (Frame N+1), in some embodiments, a sudden environmental change occurs in Frame N+1, such as when the ToF sensor detects an obstacle approaching within 50mm, causing a sudden change in ambient light intensity. The module-side MCU immediately triggers an adaptive mechanism: the LED driver circuit switches to the 850nm infrared band, the image format is changed to high-contrast RAW10_IR, and high-priority safety alarm fields are injected into the metadata.

[0139] For example, the microstructure of a data segment is as follows:

[0140] Header segment (8 Bytes);

[0141] Magic Code: 0x50414C4D;

[0142] Meta_Length: 0x0060 (96 bytes) → Dynamically adjusted to match newly added security fields;

[0143] Metadata segment (96 bytes, serialized data);

[0144] Frame_ID: 1001;

[0145] Timestamp: 1683012361600 (microseconds);

[0146] ToF_Distance: 50.2 mm → abrupt change in distance;

[0147] Lux_Light: 450.0 Lux;

[0148] Image_Format: 0x02 (RAW10_IR);

[0149] Payload_Size: 0x04B000 (307,200 Bytes) → Reduced data size to decrease bus load;

[0150] Safety_Flag: 0x01 (WARNING_PROXIMITY) → Directly triggers a main control interrupt;

[0151] Payload segment (307,200 Bytes);

[0152] Raw Data: [IR_Val, IR_Val, IR_Val…] → 640x480 RAW10 infrared pixels;

[0153] Specifically, through the above methods, hardware-level rapid parsing of the header, dynamic scaling of the metadata structure and synchronization of multimodal parameters, and seamless switching of the payload's image format and data volume are achieved. This allows the main control chip to trigger safety responses, such as controlling the emergency stop of a medical robotic arm or triggering an autonomous driving warning, without having to search for warning information in the image data.

[0154] S230: Send the protocol frame to the receiving end through the physical communication interface.

[0155] In some specific implementations, the encapsulated protocol frame is sent to the receiving end (Host) through a high-speed physical communication interface (such as USB 3.0 / 3.1 interface, high-bandwidth vehicle bus HSD, or MIPI interface).

[0156] It should be noted that, since the first type of data and the second type of data reside in the same protocol frame in physical memory and are at contiguous addresses, the receiving end can obtain complete spatiotemporal alignment information based on a single DMA operation when receiving this frame of data. Through this method, this application effectively solves the problem of protocol separation between image streams and sensor streams in traditional technologies, ensuring a high degree of synchronization of multimodal data at the physical transmission layer.

[0157] The data transmission method described in this application breaks away from the traditional "physical channel separation" or "application layer hard synchronization" model for multimodal data. By constructing three-segment protocol frames with contiguous addresses at the physical memory layer, high-frequency small packets (Metadata) and large-volume data streams (Payload) are forcibly bound to the same DMA transmission transaction. This design not only eliminates jitter caused by bus contention but also achieves "protocol-level alignment," enabling the receiving end to simultaneously acquire the sensor's physical state information and its corresponding raw data stream in a single memory access, thus completely solving the problem of timing inaccuracies in heterogeneous data from the ground up.

[0158] Combination Figure 3 As shown, encapsulating the multimodal sensing data to obtain a protocol frame specifically includes the following steps:

[0159] S310: Convert the first type of data using a schema-based extensible serialization format to generate the corresponding serialized stream.

[0160] In this application embodiment, in order to achieve efficient extension of the protocol, this application uses Schema (schema definition) to standardize the first type of data (such as IMU, timestamp, GPIO status, etc.).

[0161] In some embodiments, the first type of structured sensor data with semantic labels is converted using a schema-based extensible serialization format to generate a corresponding serialized stream. The serialization format can be Cap'n Proto or Protobuf to support dynamic expansion and zero-copy access of subsequent protocol frames.

[0162] In other embodiments, the first type of data may include IMU acceleration, gyroscope data, ToF distance, ambient light information, and physiological signals, etc., and hardware timestamps are retained during the serialization process for microsecond-level spatiotemporal alignment.

[0163] Specifically, serialization formats such as Cap'n Proto, Protobuf, or FlatBuffers are used to convert scattered structured data into a continuous binary serialized stream. Through schema definition, the receiving end can parse known fields from the metadata description even without upgrading the driver, thus ensuring backward compatibility of the protocol.

[0164] S320: Identify the byte length value of the serialized stream and write the byte length value into the metadata length field of the frame header.

[0165] After serialization is complete, the system dynamically obtains the actual length of the serialized stream. The length value is then filled into the predefined metadata length field (MetaLen) in the frame header. By explicitly declaring the metadata length in the frame header, the receiving hardware or driver can directly locate the boundary of the subsequent payload area through simple arithmetic operations without parsing the metadata content, greatly improving packet splitting efficiency.

[0166] S330: Determine the starting offset address of the load area in physical memory based on the metadata length field and the preset memory alignment granularity.

[0167] To optimize DMA (Direct Memory Access) transfer performance, the starting position of the payload area in memory must conform to the alignment requirements of the underlying hardware. The specific logic is as follows: the length of the padding bits is calculated based on the metadata length field and the preset alignment granularity (e.g., 4 bytes, 8 bytes, 16 bytes, or 64 bytes).

[0168] For example, if the alignment granularity is set to 8 bytes and the metadata length is N bytes, then the padding length is calculated. Therefore, the starting offset address of the payload area in physical memory is determined as: frame header length + metadata length + padding length. This ensures that subsequent high-bandwidth image or point cloud data (Type II data) is always stored at aligned memory addresses, avoiding performance jitter caused by unaligned access.

[0169] It is understood that alignment granularity refers to the starting base address of a specific data segment in physical memory, which is constrained to an integer multiple of a preset alignment size. For example, the size of the alignment granularity can be flexibly configured to 8 bytes, 16 bytes, 32 bytes, 64 bytes, or 4KB, etc. (for example, but not limited to the above values), depending on the bus width or cache characteristics of the underlying hardware architecture.

[0170] In practice, during the allocation or writing of memory space, the system dynamically evaluates whether the total length of the preceding data segment matches the alignment granularity. If they do not match, the system will not store the subsequent data segment immediately after the preceding data segment. Instead, it will adaptively expand the memory space by injecting padding bits, so that the physical starting address of the subsequent high-bandwidth data segment (such as the load area) is precisely and forcibly aligned to an integer multiple of the alignment granularity boundary.

[0171] S340: Store the second type of data directly into the physical memory area corresponding to the starting offset address in the form of a raw binary large object.

[0172] In this embodiment, unlike the first type of data, the second type of data (such as raw video frames) is not serialized. The system directly calls the DMA controller to move the raw bitstream output by the sensing unit to the starting offset address determined in step S330. This "direct mapping" storage method eliminates the memory copying and encoding overhead of large-scale data, realizing true zero-copy transmission.

[0173] S350: In physical memory, the frame header, serialized stream, and raw binary data are sequentially integrated into a single protocol frame.

[0174] In this embodiment, a single data unit with a completely contiguous address space is finally formed in physical memory. From a low-level perspective, the protocol frame is a pipelined structure consisting of a frame header (fixed length), serialized metadata (dynamic length), padding bits (optional), and the original payload (massive data). This design allows the entire protocol frame to be sent in batches through a single physical interface request, significantly reducing the context switching cost and header overhead of the transmission link.

[0175] Through the above steps, schema-based serialization ensures backward compatibility of the protocol (adding sensors does not require driver modifications), while padding is dynamically calculated using preset memory alignment granularity. This fine-grained address alignment logic ensures that unstructured massive payloads are always at the optimal address for hardware DMA access, achieving "zero-copy" transmission and "dynamic metadata" coexisting, significantly improving the throughput efficiency of embedded systems.

[0176] like Figure 4 As shown, to achieve precise spatiotemporal synchronization of multimodal sensing data, the data transmission method in this embodiment further includes steps S410 and S420, used to record a hardware timestamp at the trigger moment of data acquisition by the sensor module, and encapsulate it as a first type of data into the metadata area. Specifically, the data transmission method further includes:

[0177] S410: At the trigger moment when the sensor module acquires multimodal sensing data, read the current value of the free-running counter at the transmitting end and use it as a hardware timestamp.

[0178] In the embodiments of this application, "trigger time" refers to the critical moment when the sensing unit captures the physical signal. For example, when the sensor module is an image sensor, the trigger time is specifically the start time of the vertical synchronization signal (VSYNC) or the exposure start time (SoF); when the sensor module is an IMU or a ToF sensor, the trigger time is specifically the moment when data acquisition is completed and an interrupt signal is sent to the processor.

[0179] Through logical triggering by the hardware circuit, the system synchronously latches the current value of the free-running counter at the moment of the trigger, using it as the original hardware timestamp of the frame data. Compared to obtaining the system time at the application layer, this method of directly obtaining the hardware counter value at the physical or link layer completely eliminates time jitter caused by operating system scheduling, bus contention, or interrupt delays.

[0180] In some embodiments, when the sensor module triggers the acquisition of multimodal sensing data (e.g., CMOS image exposure start signal, ToF measurement trigger, or IMU sampling trigger), it reads the current value of the free-running counter at the transmitting end and uses this value as a hardware timestamp. The free-running counter is an independent hardware counter capable of continuous counting with microsecond-level precision.

[0181] Specifically, the transmitter internally maintains a high-precision free-running counter, which is driven by a local crystal oscillator within the module and provides a monotonically increasing count value.

[0182] S420: The hardware timestamp is used as the first type of data and encapsulated into the metadata area.

[0183] In this embodiment, after obtaining the hardware timestamp, the sending end uses it as a core field in the metadata. According to the aforementioned step S310, the hardware timestamp is included in the first type of data for serialization processing.

[0184] In this embodiment, the metadata area may also carry information such as sensor type identifier and frame sequence number. This information, together with the hardware timestamp, constitutes the "identity certificate" of the protocol frame. For example, the hardware timestamp obtained in step S410 is used as the first type of data and encapsulated into the metadata area of ​​the protocol frame. The metadata area adopts a schema-based extensible serialization format, which can simultaneously carry multiple low-bandwidth structured sensor data.

[0185] Specifically, since the hardware timestamp is encapsulated in a metadata area with contiguous addresses and sent along with high-bandwidth payload data (such as image pixels), after unpacking, the receiving end can use the hardware timestamp and a preset time mapping model (such as a linear regression model fitted by the least squares method) to map the local time domain of the module to the system time domain of the master control end, thereby achieving multimodal data alignment with microsecond-level precision.

[0186] Through the above steps, the timestamp generation node is moved from the "software layer" to the "physical signal trigger moment." By latching a hardware free-running counter instead of calling system time functions, time noise caused by CPU scheduling delays, bus handshake time, and operating system interrupt responses is avoided. This "hardware-level anchoring" provides the most original and purest data source for subsequent microsecond-level synchronization, which is a technical prerequisite for achieving high-precision SLAM or multi-view vision fusion.

[0187] like Figure 5 As shown, in some embodiments of this application, in order to ensure the security of sensor data during transmission and prevent data from being tampered with or illegally eavesdropped on, the data transmission method further includes security enhancement steps S510 to S530:

[0188] S510: Invoke the hardware security chip of the sending end to encrypt the metadata area based on the negotiated session key and generate a security verification tag.

[0189] Specifically, the sending node integrates a hardware security chip (Secure Element, SE) or a trusted execution environment (TEE). During the communication initialization phase, the sending and receiving ends complete mutual authentication using asymmetric encryption algorithms (such as ECDSA or RSA) and negotiate to generate a symmetric session key.

[0190] When encapsulating protocol frames, the system invokes the hardware acceleration engine to perform strong encryption on the serialized metadata area using the session key. In this embodiment, an authentication encryption algorithm that supports associated data is preferably used, such as ChaCha20-Poly1305 or AES-GCM.

[0191] Since the metadata area contains highly sensitive scalar information such as hardware timestamps and device status, strong encryption ensures its confidentiality and simultaneously generates corresponding security verification tags (Tag / MAC) to verify the integrity and authenticity of the data. The generated security verification tags ensure that the metadata is not tampered with during transmission and also support the receiving end in quickly verifying data integrity.

[0192] S520: Write the security verification tag into the frame header.

[0193] Specifically, the security verification tag (e.g., a 16-byte Poly1305 Tag) generated in step S510 is filled into the reserved verification field (CRC / Tag) in the protocol frame header. Placing the verification information in the frame header allows the receiver to perform security checks in parallel using hardware logic in the early stages of receiving the data packet. If the tag verification fails, the frame data is directly discarded and an early warning is triggered, without having to proceed to the subsequent time-consuming payload decryption process.

[0194] S530: Perform data obfuscation or streaming encryption on the original binary large object in the payload area.

[0195] Given the extremely large volume of Type II data (such as high-resolution raw images) stored in the payload area, full-scale strong encryption could lead to significant transmission delays and increased power consumption. Therefore, this application adopts a differentiated protection strategy for the payload area:

[0196] Optionally, a stream cipher algorithm can be used to process the payload data bit by bit to maintain the continuity of zero-copy transmission.

[0197] Optionally, a lightweight data obfuscation algorithm may be used, such as remapping pixel addresses based on a specific pseudo-random sequence, or sampling and encrypting certain key rows / blocks in the payload area.

[0198] Understandably, the segmented strategy of "strong metadata encryption + lightweight payload protection" ensures the absolute security of core semantic information such as biometrics while maximizing the real-time transmission performance of high-bandwidth sensor data.

[0199] Through the above steps, strong encryption and signing supported by the SE chip are used for metadata to ensure the absolute security of core semantic information; for massive payloads, dynamic obfuscation is achieved using session keys and rolling counter derivation seeds. This design achieves an optimal balance between "protecting biometric privacy" and "maintaining high-bandwidth real-time transmission," and the rolling counter enables the protocol to inherently resist replay attacks, filling a gap in data security for general sensing protocols (such as UVC and HID).

[0200] In some preferred embodiments of this application, to further defend against replay attacks and improve the anti-decryption strength of payload data, the data transmission method further includes the following steps and dynamic verification logic based on a rolling counter. Specifically, the data transmission method further includes:

[0201] Step 10: Before encapsulating the protocol frame, obtain the value of the rolling counter at the sending end.

[0202] In some embodiments, before encapsulating a protocol frame, the sending end reads the current value of a local rolling counter and calls a hardware security chip to digitally sign the rolling counter value. The rolling counter is a freely incrementing counter that can be used to mark the protocol frame sequence, ensure the uniqueness of frame sequence numbers, and provide anti-replay protection.

[0203] Specifically, the sending node maintains a hardware-level rolling counter. Whenever the system is ready to encapsulate and send a new protocol frame, the value of this rolling counter increments by one. Because the counter value is monotonically increasing and does not repeat over a long period, it serves as a unique logical identifier for each frame of data.

[0204] Step 20: Call the hardware security chip to sign the value of the rolling counter.

[0205] In some embodiments, the signed rolling counter value is written into the frame header as the frame sequence number field of the protocol frame, while other header fields remain unchanged (Magic Code, metadata length, security check tag, etc.). The frame header field, used in conjunction with the signature rolling counter, enables the receiving end to perform fast and unique frame identification and verify the validity of the frame sequence number signature when parsing the frame header.

[0206] Understandably, to prevent hackers from intercepting bus data and forging incrementing frame sequence numbers, the system calls the aforementioned hardware security chip (SE) to digitally sign the current rolling counter value using the sender's private key or a preset HMAC key. This process ensures the frame sequence number's unforgeability during transmission.

[0207] Step 30: Write the signed rolling counter value as the frame sequence number into the frame header of the protocol frame.

[0208] In some embodiments, the security check tag (MAC) of the protocol frame is calculated based on the frame header field and the encrypted metadata area, thereby ensuring the consistency and integrity of the frame header, metadata area and frame sequence number.

[0209] Understandably, the signed count value is filled into the FrameID field of the frame header. Upon receiving a protocol frame, the host first verifies the validity of the FrameID signature and checks if the value conforms to the expected incrementing order. If the FrameID is detected to be less than or equal to the maximum value already received, it is determined to be a replay attack, the frame is immediately discarded, and the communication link is interrupted, thus achieving inherent security at the protocol layer.

[0210] Furthermore, embodiments of this application also relate to a multi-dimensional security coupling verification mechanism, including a security verification tag calculated based on the frame header and the encrypted metadata area, and a data obfuscation seed generated based on the session key and the rolling counter value. Specifically, the following description is provided:

[0211] (a) Overall verification of frame header and metadata:

[0212] The security verification tag generated in step S510 above has its calculation input including not only the encrypted metadata area, but also the complete frame header including the FrameID. This "full header association" verification method ensures that any attack targeting the frame header fields (such as tampering with the metadata length or frame sequence number) will cause the tag verification to fail.

[0213] (ii) Count-based dynamic data obfuscation:

[0214] During the data obfuscation process described in step S530, the system does not use a fixed obfuscation mode, but instead performs logical operations (such as XOR or hash operations) based on the session key and the current rolling counter value to generate a unique obfuscation seed (Salt / Seed) for that frame.

[0215] Because the rolling counter changes every frame, even if two frames of sensor data (such as the image payload of a still image) are exactly the same, the binary streams transmitted on the bus after obfuscation will be completely different. This dynamic obfuscation mechanism greatly increases the difficulty of reverse engineering and ensures the privacy of high-bandwidth multimodal data when transmitted at the physical layer.

[0216] Figure 6 This is a schematic flowchart of a data transmission method provided in this application. Figure 6 The data transmission method shown is used to receive data, and can be performed by a receiving end or a receiving device (see...). Figure 8 This can be implemented using a receiver or any other suitable device. For example, the receiver or receiving device includes... Figure 1 The main control device 110 is shown.

[0217] The following describes the data transmission method using the receiving end as an example. Specifically, the data transmission method includes steps S610-S650.

[0218] S610: Receives a protocol frame from the transmitting end via a physical communication interface. The protocol frame is obtained by encapsulating multimodal sensing data. Specifically, the protocol frame in the transmitting end's physical memory includes a frame header, metadata area, and payload area with contiguous addresses. The frame header includes a metadata length field and a security check tag. The metadata area carries structured sensing data with semantic tags from the multimodal sensing data. The payload area carries unstructured sensor array data or raw binary code streams from the multimodal sensing data.

[0219] In this embodiment, the receiving end receives the protocol frames generated by the sending end in real time through a high-speed serial physical interface (such as a USB 3.0 bus, HSD harness, or fiber optic interface).

[0220] As mentioned earlier, this protocol frame is represented at the physical layer as a completely contiguous bit stream in the address space, with the frame header, metadata area, and payload area arranged in sequence. This compact and address-contiguous layout allows the receiving end hardware interface layer to acquire data in a bulk transfer mode, reducing the system load caused by frequent interruptions.

[0221] S620: Parse the frame header of the protocol frame and extract the metadata length field and security verification tag of the frame header.

[0222] In this embodiment, after receiving a protocol frame, the receiving driver first accesses a fixed-length frame header area. Due to the pre-defined frame header structure, the system can directly extract the metadata length field (MetaLen) and the security check tag (Tag / CRC). The metadata length field indicates the precise size of the subsequent serialized data stream, while the security check tag is used for subsequent authentication and tamper-proof checks.

[0223] S630: Locate the boundary between the metadata area and the payload area in the current protocol frame based on the metadata length field.

[0224] In this embodiment, the key step for achieving fast unpacking is that the receiving end does not need to traverse or parse the entire protocol frame, but only needs to extract the data from the frame header. Combined with the preset frame header length By using memory alignment rules, the offset of the payload relative to the frame start address can be accurately calculated. .

[0225] The specific logic is as follows: ,in This refers to the number of padding bytes calculated based on a preset alignment granularity (e.g., 8 bytes). Through this boundary positioning mechanism, the system can logically separate the metadata processing path from the massive payload processing path, ensuring that they do not block each other.

[0226] S640: Decrypt and verify the integrity of the metadata area using a security verification tag.

[0227] In this embodiment, the receiving end invokes the encryption instruction set within a local security chip or general-purpose processor, using a pre-negotiated session key combined with a security verification tag in the frame header to decrypt and hash the metadata area. In this embodiment, by verifying the legitimacy of the tag, it can be ensured that metadata (such as hardware timestamps and key sensor parameters) has not been tampered with by an attacker, and it can be confirmed that the frame data indeed comes from a legitimate sending device.

[0228] S650: When the integrity check passes, the first type of data is parsed from the metadata area, and the payload area is directly mapped to the system memory or video memory of the receiving end through direct memory access DMA.

[0229] Understandably, once the security check passes, the system will execute a dual-track processing strategy:

[0230] Lightweight parsing: Deserializes the metadata area to extract the first type of data (such as hardware timestamps, IMU data, etc.) for time alignment and logic triggering of upper-layer algorithms.

[0231] Zero-copy mapping: For high-bandwidth Type II data (such as raw images or point clouds), the receiver uses a DMA controller to move the data in the payload area directly from the protocol buffer to the pre-allocated system memory or the video memory of the graphics processing unit (GPU).

[0232] It should be noted that this processing method avoids secondary copying and redundant parsing of large-volume unstructured data. Optionally, if the receiving end detects a verification failure in the frame data, it directly discards the payload without performing memory mapping, thereby protecting the system's computing resources from the impact of illegal / invalid data.

[0233] By employing the steps described above, the length field in the frame header is directly used for boundary location, allowing the receiving end to accurately separate metadata from the payload without scanning the entire frame. Metadata is processed through a lightweight parsing path for rapid response, while the bulky payload is directly mapped to GPU memory via DMA. This design significantly reduces the load on the main control unit and effectively addresses performance bottlenecks in high-frame-rate multimodal data transmission scenarios.

[0234] In some embodiments of this application, the frame header of the protocol frame also includes a frame sequence number (FrameID). By monitoring the continuity and validity of the frame sequence number in real time, the receiving end can effectively identify abnormal situations such as replay attacks, network congestion, or packet loss. Specifically, the data transmission method further includes anti-replay and logical verification steps 10 to 40:

[0235] Step 10: Parse the frame header of the protocol frame and extract the frame sequence number from the frame header.

[0236] In this embodiment, after acquiring a protocol frame, the receiving end first extracts the current frame sequence number from a preset field in the frame header. As mentioned earlier, this frame sequence number undergoes signature processing based on a rolling counter at the sending end. After extracting the sequence number, the receiving end compares and analyzes it with the locally stored "historical frame sequence numbers" (i.e., the sequence numbers of the previous frame that were successfully received and verified).

[0237] Step 20: If the current frame number is less than or equal to the historical frame number, it is determined that the frame number is rolled back or repeated.

[0238] Under normal communication conditions, since the rolling counter at the sending end is monotonically increasing, the frame sequence numbers received at the receiving end should exhibit a strictly increasing pattern. If the parsed current frame sequence number is less than or equal to the historical frame sequence number, the system determines that the data packet may be a historical message intercepted and retransmitted by a hacker (replay attack), or a redundant duplicate packet caused by physical link layer retransmission logic. This rollback or duplication determination is an important means of ensuring the uniqueness of the timing of perceived data.

[0239] Step 30: If the difference between the current frame number and the historical frame number exceeds the preset sequence number step size, it is determined that the frame number has changed abnormally.

[0240] In this implementation, to monitor the packet loss rate of the transmission link, the system is configured with a preset sequence number step size (e.g., a step size of 1, which can be set to N to allow for small-range packet loss compensation). If the difference (jump amplitude) between the current frame sequence number and the historical frame sequence number exceeds this preset range, it is determined to be an abnormal jump. This usually means that the physical bus has experienced severe electromagnetic interference, network congestion leading to sudden packet loss, or the sending end logic has experienced an abnormal reset.

[0241] Step 40: When it is determined that there is a frame sequence number rollback, duplication, or abnormal jump, discard the current protocol frame.

[0242] For example, once any of the above determination results are triggered, the receiving end will immediately start the exception handling procedure, directly discard the current protocol frame, and not map its payload area to system memory or video memory.

[0243] Optionally, while discarding protocol frames, the receiving end can also record abnormal event logs and send a warning interrupt to the application layer of the master control end to indicate that there is a security risk or physical link instability in the current multimodal sensing link. Through this mechanism based on sliding window and monotonicity verification, this application not only achieves intrinsic security protection, but also provides real-time link quality feedback for the system.

[0244] Through the above steps, including monotonicity checks (rollback / repetition determination) and transition amplitude checks on frame sequence numbers, the system achieves coordinated early warning at the protocol layer for both hacker attacks and physical link failures (such as severe packet loss caused by electromagnetic interference). This mechanism not only protects data assets but also provides robust protection for multimodal perception in complex electromagnetic environments (such as vehicular systems).

[0245] like Figure 7 As shown, in some embodiments of this application, in order to achieve the fusion processing of multimodal data with different physical interfaces and different sampling frequencies under the same time base, for example, to achieve accurate spatiotemporal synchronization of multimodal sensing data at the receiving end, a globally unified time is calculated based on a hardware timestamp and multimodal data alignment is completed. Specifically, the data transmission method further includes steps S710 to S730.

[0246] S710: Parse the metadata area of ​​the current protocol frame and extract the hardware timestamp of the metadata area.

[0247] In practice, after completing the security verification and deserialization of the protocol frame, the receiving end extracts the hardware timestamp recorded by the sending end at the moment the sensing signal is triggered from the metadata area. As mentioned earlier, this timestamp represents the raw value of the local free-running counter at the transmitting end. Because the transmitting and receiving ends are driven by different crystal oscillators, and there is a physical offset and frequency drift between the crystal oscillators, this hardware timestamp cannot be directly used as the absolute time at the system level.

[0248] S720: Calculate the global unified time based on the preset linear regression model and the hardware timestamp.

[0249] In this embodiment of the application, the receiving end maintains a preset linear regression model. This maps the module's local time domain to the receiver's global unified time domain.

[0250] Here, the slope 'a' reflects the frequency ratio between the crystal oscillators at the transmitting and receiving ends, and the intercept 'b' reflects the initial phase difference between them.

[0251] In this embodiment, the linear regression model is dynamically updated. Specifically, the master control end and the sending end periodically perform a "ping-pong" time synchronization handshake (e.g., sending a SYNC request and recording the round-trip time (RTT)) to obtain multiple sets of... Observation samples. The receiver uses the most recent N sets of observation samples and algorithms such as Ordinary Least Squares to continuously correct the model parameters a and b. This is achieved by using real-time extracted hardware timestamps. By substituting this high-precision mathematical model, the corresponding global unified time can be calculated. The synchronization accuracy at this time can reach the microsecond level.

[0252] In one specific implementation, to accurately calculate the global uniform time of the protocol frames, the receiving end (Host) dynamically establishes and updates the linear regression model through periodic "ping-pong" synchronization logic. The specific steps are as follows:

[0253] S721: Periodically sends clock synchronization request messages to the sender and records the first local timestamp at the time of transmission.

[0254] In some embodiments, the receiver's main control processor or dedicated protocol module periodically (e.g., every 10ms to 100ms) generates a synchronization request message (SYNC_REQ). The moment the message leaves the receiver's physical layer interface, a hardware stamping module records the current receiver system timestamp. (i.e., the first local timestamp).

[0255] S722: Receives the synchronization response message returned by the sender, records the second local timestamp at the time of reception, and extracts the hardware timestamp at the time the sender received the clock synchronization request message from the synchronization response message.

[0256] In some embodiments, upon receiving the request message, the sending node immediately latches the current value of its internal free-running counter. (i.e., the sender's hardware timestamp), and sends it back to the receiver along with a synchronization acknowledgment message (SYNC_ACK). The receiver records its current system timestamp the instant it fully receives the acknowledgment message. (i.e., the second local timestamp).

[0257] S723: Calculate the round-trip transmission delay of the message based on the first local timestamp and the second local timestamp.

[0258] In some implementations, the receiving end calculates... and The difference between these values ​​yields the total round-trip time (RTT) of the message on the physical link. Considering the symmetry of the physical link, the one-way transmission delay can be estimated as... .

[0259] S724: Calculate the clock offset based on the first local timestamp, the second local timestamp, and the hardware timestamp.

[0260] In some implementations, the system further calculates the time offset between the transmitter and receiver. Specifically, the "equivalent receiver system time" corresponding to the receiving time at the transmitter should be... Therefore, the current instantaneous clock deviation can be defined as follows: This deviation reflects the static shift between the transmitter's counter value and the receiver's global time axis at a specific moment.

[0261] S725: Based on multiple consecutive clock offsets, a linear regression model is established between the hardware time of the transmitting end and the system time of the receiving end.

[0262] In some implementations, due to noise caused by bus scheduling jitter in a single measurement, the receiver maintains a sliding observation window to collect N consecutive sets of synchronization samples. Utilizing the multiple correspondences within the sliding window (transmitter hardware time, receiver system time), the system employs Ordinary Least Squares (OLS) for linear fitting.

[0263] Specifically, establish a linear model The slope 'a' is used to correct the frequency drift of the crystal oscillators at both ends, and the intercept 'b' is used to correct the phase offset of the time base. Through this continuous dynamic feedback modeling, the system can track and compensate for the time drift of the sensing link in real time, ensuring that even after long-term operation, the first type of multimodal sensing data (such as IMU data) and the second type of data (such as images) can still maintain extremely high spatiotemporal alignment accuracy.

[0264] S730: Use the global unified time as the common timestamp of the first type of data and the second type of data in the current protocol frame, and perform spatiotemporal alignment on the first type of data and the second type of data.

[0265] In practical implementation, the globally unified time is calculated. Then, the receiving end assigns it all the sensing data within the protocol frame. This means that the first type of data (such as IMU pose and ambient temperature) and the second type of data (such as raw image frames and point clouds) parsed at that moment are logically bound to the same precise time anchor point.

[0266] Through the aforementioned spatiotemporal alignment mechanism, upper-layer applications (such as SLAM algorithms and multi-sensor fusion recognition algorithms) can eliminate asynchronous deviations caused by transmission delays, system scheduling, or crystal oscillator drift. For example, in autonomous driving or robot vision scenarios, the algorithm can accurately know the precise three-axis acceleration and gyroscope state of the sensor at the instant of image acquisition for a certain frame, thereby significantly improving the temporal consistency and algorithm accuracy of multimodal perception.

[0267] Through the above steps, the phase shift and frequency drift of the two crystal oscillators are captured in real time by periodic "ping-pong" measurements, and the regression coefficients are continuously corrected using the least squares method. This method not only achieves microsecond-level global unified time alignment but also has the ability to "adapt to environmental changes." Its innovative contribution lies in simplifying the originally complex network synchronization scheme (such as PTP) that requires dedicated hardware support into a low-overhead, high-precision protocol-built-in algorithm, completely eliminating fusion errors between asynchronous sensors.

[0268] To more clearly illustrate the contiguous "three-segment" physical memory layout described in this application, the following detailed explanation is provided in conjunction with the hardware and software collaborative transfer mechanism between the sending and receiving ends.

[0269] At the transmitting end, this application maximizes bus transmission throughput and significantly reduces configuration latency by constructing a completely contiguous frame memory space. Specifically, since the frame header, metadata area, and payload area exhibit an absolutely linear topology and contiguous address distribution in physical memory, the Direct Memory Access (DMA) controller within the main control chip does not need to perform complex address traversal searches when initiating a transmission transaction. The DMA controller only needs to obtain a single physical starting address of this contiguous memory space and the total length of the entire frame data to initiate a single, continuous, and uninterrupted burst transmission on the system bus, sending the entire frame of multimodal mixed data to the transmission link at once.

[0270] In contrast, if a traditional non-contiguous memory architecture is used (such as using a linked list structure or distributed memory blocks to store heterogeneous sensor data separately), the DMA controller must configure an extremely complex hash-convergence table at the hardware level when transferring data. This not only increases the reading overhead of the DMA descriptor and the switching loss of the hardware state machine, but also introduces a non-negligible bus configuration delay, resulting in timing jitter in high-frequency burst environments.

[0271] At the receiving end, the contiguous address layout enables true hardware-level zero-copy and nanosecond-level data stripping. When the data from a single burst transmission arrives at the receiving end, the hardware directly maps and writes the complete data stream of that frame into a pre-allocated, contiguous-address receive buffer. Based on this physical layer state, the receiving end software protocol stack does not need to perform any memory copy operations when parsing multimodal sensing data, thus completely eliminating the CPU's computational overhead in data handling.

[0272] Specifically, the receiving processor only needs to perform pointer type casting and address offset calculation to directly locate the boundaries and semantically associate each data segment within the original receive buffer. For example, its control logic is implemented as follows:

[0273] ; ;

[0274] Here, `buffer` is the starting address of the contiguous buffer, and `offset` is the physical offset of the payload area calculated based on the metadata length declared in the frame header and the preset alignment granularity. Since pointer casting and address addition are register-level instantaneous instructions in computer architecture, their core parsing time is theoretically 0 nanoseconds (ns), thus ensuring the real-time response boundary in high-throughput, high-concurrency multimodal scenarios (such as high-throughput palm vein recognition, millisecond-level in-cabin radar alarms, etc.) at the underlying level.

[0275] This embodiment compares the application of the "palm vein / medical dual-modal sensing module" in scenarios such as high-throughput access gates in rail transit or high-precision detection during medical surgery under high-frequency real-time scanning at 60fps (frames per second), where the system's frame interval is only about 16.6 milliseconds (ms).

[0276] In traditional technical solutions, multimodal sensing systems adopt a driver architecture with separate physical channels and independent protocols. The infrared pixel image data, which is the second type of data, is huge and is transmitted through high-speed video channels such as UVC or MIPI and managed by the video driver stack. Meanwhile, the ToF distance anti-counterfeiting data, ambient temperature, and LED status, which are the first type of data, are transmitted asynchronously through low-speed serial ports such as I2C or UART.

[0277] At 60fps high-frequency scanning, due to the nondeterminism of the operating system kernel scheduling and the jitter of the interrupt response of various drivers, when the host receives the image and distance data corresponding to the same physical event, millisecond-level clock drift often occurs. This causes the algorithm layer to mismatch the "distance of the previous frame" and the "image of the current frame" when the user waves quickly or the probe moves, resulting in recognition failure. Furthermore, the main control CPU is forced to consume a lot of computing power for software timestamp alignment.

[0278] In contrast, using the data transmission method provided in this application, at the microsecond-level instant triggered by the exposure start signal of the visual sensor, the hardware logic circuit directly binds and packages the infrared pixel matrix (load area), ToF ranging value and LED status (metadata area) corresponding to the current exposure frame at the hardware physical layer through the underlying clock latch mechanism, and adaptively fills them according to the preset alignment granularity to construct a single protocol frame with absolutely continuous addresses.

[0279] Thanks to this special physical layer structure, the multi-source heterogeneous data acquired by the receiving end has microsecond-level spatiotemporal conjugate alignment characteristics during encapsulation, which fundamentally eliminates the clock drift caused by asynchronous transmission of the operating system bus and multi-channel drive jitter, completely smooths out the risk of identification gaps and security misjudgments caused by the rapid movement of the target, and ensures the absolute accuracy of liveness detection and high-precision detection.

[0280] Meanwhile, the receiver's protocol stack software only needs to use register-level pointer type casting to directly separate the high-frequency, high-bandwidth data stream and metadata within 0 nanoseconds and map it to the inference core with zero copy. This keeps the end-to-end data transmission and parsing latency in the sub-millisecond range and consumes almost no CPU computing power, perfectly meeting the stringent requirements of high-throughput seamless access and medical-grade precise detection.

[0281] In one embodiment, this application also provides a data transmission device, which can be implemented by hardware circuitry, software functional units, or a combination of both. The data transmission device is used to send data to execute and implement... Figures 2-5 The data transmission method is illustrated. Specifically, the data transmission device includes an acquisition module, an encapsulation module, and a transmission module. The execution logic of each module is described in detail below:

[0282] The acquisition module is used to acquire multimodal sensing data from the sensor module. The multimodal sensing data has a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, and the second type of data is unstructured sensor array data or raw binary code stream.

[0283] An encapsulation module is used to encapsulate the multimodal sensing data to obtain a protocol frame; wherein, the protocol frame includes a frame header, a metadata area, and a payload area with contiguous addresses in the physical memory of the transmitting end; the metadata area carries the first type of data, and the payload area carries the second type of data;

[0284] The transmitting module is used to transmit the protocol frame to the receiving end through a physical communication interface.

[0285] In one embodiment, this application also provides a data transmission device, which can be implemented by hardware circuitry, software functional units, or a combination of both. The data transmission device is used to receive data to execute and implement... Figures 6-7 The data transmission method is illustrated. The data transmission device includes a receiving module, an extraction module, a positioning module, a verification module, and a parsing module. The collaborative logic of each module is as follows:

[0286] A receiving module is used to receive protocol frames from a transmitting end through a physical communication interface. The protocol frames are obtained by encapsulating multimodal sensing data. The protocol frames in the physical memory of the transmitting end include a frame header, a metadata area, and a payload area with contiguous addresses. The frame header includes a metadata length field and a security check tag. The metadata area carries structured sensing data with semantic tags in the multimodal sensing data. The payload area carries unstructured sensor array data or raw binary code streams in the multimodal sensing data.

[0287] The extraction module is used to parse the frame header of the protocol frame and extract the metadata length field and security check tag of the frame header;

[0288] The positioning module is used to locate the boundary between the metadata area and the payload area in the current protocol frame based on the metadata length field.

[0289] The verification module is used to decrypt and verify the integrity of the metadata area using a security verification tag;

[0290] The parsing module is used to parse the first type of data from the metadata area when the integrity verification passes, and to directly map the payload area to the system memory or video memory of the receiving end through direct memory access (DMA).

[0291] Figure 8 A schematic diagram of a transmitting device 8 according to an embodiment of this application is shown. Figure 8 As shown, the transmitting device 8 (specifically corresponding to a sensor module Node in this application) includes a first communication interface 81, at least one first processor 82, and at least one first memory 83. The first memory 83 stores program instructions, which, when executed by the first processor 82, cause the first processor 82 to perform the following actions: Figures 2-5 The data transmission method shown includes the acquisition of multimodal sensor data, encapsulation based on a hybrid load protocol, and transmission logic.

[0292] It should be understood that the first communication interface 81 is used for communication with other devices. In this embodiment, it is specifically a physical interface that supports high-speed data transmission, such as a USB 3.0 controller, a MIPI interface circuit, or an HSD physical transceiver. The first processor 82 can be an FPGA, DSP, microprocessor (such as the Cortex-M series), or application-specific integrated circuit (ASIC). The first processor 82 is responsible for controlling the synchronous triggering of the sensor unit, reading the hardware timestamp latched by the free-running counter, and performing schema-based metadata serialization.

[0293] In another implementation, the transmitting device 8 further includes a hardware security chip and a sensing unit. The sensing unit is used to collect multimodal sensing data, and may include an image sensor (such as CMOS), an inertial measurement unit (IMU), a time-of-flight (ToF) sensor, a structured light module, an ambient light sensor, or a thermometer, etc. The first processor 82 is responsible for encapsulating the heterogeneous data collected by the sensing unit into address-contiguous protocol frames and transmitting them using the first communication interface 81.

[0294] It should be noted that the transmitting device 8 described in this application embodiment can also be referred to as a sensor module, a smart camera, or an edge sensing terminal. The transmitting device 8 can be integrated into other devices as a functional module or chip device. For example, it can be integrated into the surround-view perception system of an autonomous vehicle, or into the biometric module of a financial payment terminal.

[0295] Figure 9 A schematic diagram of a receiving device 9 according to an embodiment of this application is shown. Figure 9 As shown, the receiving device 9 (specifically corresponding to the host in this application) includes: a second communication interface 91, at least one second processor 92, and at least one second memory 93. The second memory 93 stores program instructions, which, when executed by the second processor 92, cause the second processor 92 to perform... Figures 6-7 The data transmission method shown includes protocol frame parsing, boundary location, security verification, and clock synchronization logic based on a linear regression model.

[0296] In this embodiment, the receiving device 9 can be a standalone device or integrated as a functional module into a complex computing platform. In one possible implementation, the receiving device 9 can be a cockpit domain controller (CDC), an autonomous driving computing platform (MDC), or an industrial robot controller. The second processor 92 executes program instructions to perform zero-copy mapping of the protocol frame payload area, directly mapping it to the system memory or GPU memory of the receiving end.

[0297] In a vehicle application scenario, the transmitting device 8 can be a multimodal sensor module integrated into the vehicle, which includes at least one image sensor and a set of auxiliary sensors (such as an IMU and a thermometer). The receiving device 9 can be an on-board cockpit domain controller.

[0298] It should be noted that the transmitting device 8 and the receiving device 9 can exist as independent entities, or they can together form a multimodal sensing system. In some integrated scenarios, the transmitting device 8 and the receiving device 9 can also be integrated into the same physical enclosure, achieving efficient data interaction based on a hybrid load protocol through an internal bus.

[0299] Furthermore, such as Figure 10 As shown, in one embodiment, the transmitting device 8 and the receiving device 9 are integrated into a data transmission system 1. This data transmission system 1 constitutes a complete "Omni-Link" sensor interface link. The receiving device and the transmitting device communicate via a high-speed serial bus. Based on the above embodiments, and in conjunction with an in-vehicle scenario, this application also proposes a vehicle including the data transmission system 1.

[0300] The data transmission method, device, system, and related application technologies provided in this application, through a "three-segment" hybrid payload protocol structure, continuously encapsulate low-dimensional metadata (first type of data) and high-dimensional raw payload (second type of data) in physical memory, ensuring physical synchronization of data of different natures at the transmission layer. Combined with a hardware-software co-operational linear regression clock synchronization model, the system can compensate for crystal oscillator drift between the transmitting and receiving ends, achieving globally unified alignment.

[0301] Furthermore, by leveraging memory mapping technology and hardware DMA acceleration, massive amounts of unstructured sensor data (such as image raw streams) can be processed without complex serialization / deserialization. The receiving end can directly map the payload data to system memory or video memory by parsing a lightweight metadata length field. This "zero-copy" mechanism significantly reduces CPU load and data transfer latency.

[0302] The terms "an embodiment" or "an embodiment" as used in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in at least one embodiment of this application. Therefore, the terms "in an embodiment" or "in an embodiment" appearing throughout this specification do not necessarily refer to the same embodiment, but may refer to the same embodiment. Furthermore, in one or more embodiments, the particular features, structures, or characteristics can be combined in any suitable manner, as will be apparent to those skilled in the art from this disclosure.

[0303] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0304] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0305] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0306] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0307] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0308] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0309] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, performs a data transmission method, including at least one of the schemes described in the above embodiments.

[0310] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0311] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0312] The program code contained on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0313] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0314] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this application, all of which fall within the scope of protection of this application.

Claims

1. A data transmission method, characterized in that, Applied to the sending end, the method includes: Acquire multimodal sensing data from a sensor module. The multimodal sensing data has a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, and the second type of data is unstructured sensor array data or raw binary code stream. The multimodal sensing data is encapsulated to obtain a protocol frame; wherein, the protocol frame includes a frame header, a metadata area, and a payload area with contiguous addresses in the physical memory of the transmitting end; the metadata area carries the first type of data, and the payload area carries the second type of data; The protocol frame is sent to the receiving end through the physical communication interface.

2. The data transmission method according to claim 1, characterized in that, Encapsulating the multimodal sensing data to obtain a protocol frame specifically includes: The first type of data is converted using a schema-based extensible serialization format to generate the corresponding serialized stream; Identify the byte length value of the serialized stream and write the byte length value into the metadata length field of the frame header; The starting offset address of the load area in physical memory is determined based on the metadata length field and the preset memory alignment granularity. The second type of data is stored directly in the physical memory area corresponding to the starting offset address in the form of raw binary large objects; The frame header, serialized stream, and raw binary data are sequentially integrated into a single protocol frame in physical memory.

3. The data transmission method according to claim 2, characterized in that, The data transmission method further includes: At the trigger moment when the sensor module acquires multimodal sensing data, the current value of the free-running counter at the transmitting end is read and used as a hardware timestamp; The hardware timestamp is treated as the first type of data and encapsulated in the metadata area.

4. The data transmission method according to claim 2, characterized in that, The data transmission method further includes: The hardware security chip at the sending end is invoked to encrypt the metadata area based on the negotiated session key and generate a security verification tag. Write the security verification tag into the frame header; The original binary large object in the payload area is obfuscated or stream-encrypted.

5. The data transmission method according to claim 4, characterized in that, The data transmission method further includes: Before encapsulating the protocol frame, obtain the value of the rolling counter at the sending end; The hardware security chip is invoked to sign the value of the rolling counter; The signed rolling counter value is written as the frame sequence number into the frame header of the protocol frame; The security verification tag is calculated based on the frame header and the encrypted metadata area, and the data obfuscation seed is generated based on the session key and the rolling counter value.

6. The data transmission method according to any one of claims 1 to 5, characterized in that: The frame header is a fixed-length structure, and the frame header includes at least a synchronization word, a metadata length field, a frame sequence number, and a security check tag. The synchronization word is a preset fixed byte sequence used by the receiving end to identify the start boundary of the protocol frame; the security verification tag is an authentication code encrypted in the metadata area, used by the receiving end to verify the integrity of the protocol frame.

7. A data transmission method, characterized in that, Applied to the receiving end, the data transmission method includes: Protocol frames are received from the transmitting end through a physical communication interface. The protocol frames are obtained by encapsulating multimodal sensing data. The protocol frames in the physical memory of the transmitting end include a frame header, a metadata area, and a payload area with contiguous addresses. The frame header includes a metadata length field and a security check tag. The metadata area carries structured sensing data with semantic tags in the multimodal sensing data. The payload area carries unstructured sensor array data or raw binary code streams in the multimodal sensing data. Parse the header of the protocol frame and extract the metadata length field and security check tag from the header; Based on the metadata length field, locate the boundary between the metadata area and the payload area in the current protocol frame; The metadata area is decrypted and its integrity is verified using a security verification tag. When the integrity verification passes, the first type of data is parsed from the metadata area, and the payload area is directly mapped to the system memory or video memory of the receiving end through direct memory access (DMA).

8. The data transmission method according to claim 7, characterized in that, The frame header also includes a frame sequence number, and the data transmission method further includes: Parse the header of the protocol frame and extract the frame sequence number from the header; If the current frame number is less than or equal to the historical frame number, it is determined that the frame number is rolled back or repeated. If the difference between the current frame number and the historical frame number exceeds the preset sequence number step size, it is determined to be an abnormal jump in frame number. If a frame sequence number rollback, duplication, or abnormal jump is detected, the current protocol frame is discarded.

9. The data transmission method according to claim 7, characterized in that, The data transmission method includes: Parse the metadata area of ​​the current protocol frame and extract the hardware timestamp from the metadata area; Calculate the globally unified time based on the preset linear regression model and the hardware timestamp; The global unified time is used as the common timestamp for the first type of data and the second type of data within the current protocol frame, and the first type of data and the second type of data are spatiotemporally aligned.

10. The data transmission method according to claim 9, characterized in that, The preset linear regression model is established using the following method: It periodically sends clock synchronization request messages to the sender and records the first local timestamp at the time of transmission; Receive the synchronization response message returned by the sender, record the second local timestamp at the time of reception, and extract the hardware timestamp at the time when the sender received the clock synchronization request message from the synchronization response message; Calculate the round-trip transmission delay of the message based on the first local timestamp and the second local timestamp; Calculate the clock offset based on the first local timestamp, the second local timestamp, and the hardware timestamp; Based on multiple consecutive clock offsets, a linear regression model is established between the hardware time of the transmitting end and the system time of the receiving end.

11. A data transmission device, characterized in that, The data transmission device is used to transmit data, including: The acquisition module is used to acquire multimodal sensing data from the sensor module. The multimodal sensing data has a first type of data and a second type of data. The first type of data is structured sensing data with semantic tags, and the second type of data is unstructured sensor array data or raw binary code stream. An encapsulation module is used to encapsulate the multimodal sensing data to obtain a protocol frame; wherein, the protocol frame includes a frame header, a metadata area, and a payload area with contiguous addresses in the physical memory of the transmitting end; the metadata area carries the first type of data, and the payload area carries the second type of data; The transmitting module is used to transmit the protocol frame to the receiving end through a physical communication interface.

12. A data transmission device, characterized in that, The data transmission device is used to receive data, including: A receiving module is used to receive protocol frames from a transmitting end through a physical communication interface. The protocol frames are obtained by encapsulating multimodal sensing data. The protocol frames in the physical memory of the transmitting end include a frame header, a metadata area, and a payload area with contiguous addresses. The frame header includes a metadata length field and a security check tag. The metadata area carries structured sensing data with semantic tags from the multimodal sensing data. The payload area carries unstructured sensor array data or raw binary code streams from the multimodal sensing data. The extraction module is used to parse the frame header of the protocol frame and extract the metadata length field and security check tag of the frame header; The positioning module is used to locate the boundary between the metadata area and the payload area in the current protocol frame based on the metadata length field. The verification module is used to decrypt and verify the integrity of the metadata area using a security verification tag; The parsing module is used to parse the first type of data from the metadata area when the integrity verification passes, and to directly map the payload area to the system memory or video memory of the receiving end through direct memory access (DMA).

13. A computer-readable storage medium having program instructions stored thereon, characterized in that, When the program instructions are executed by a computer, the computer performs the data transmission method according to any one of claims 1-10.

14. A computer program, characterized in that, It includes program instructions that, when executed by a computer, cause the computer to perform the data transmission method according to any one of claims 1-10.

Images