Multi-modal data recording method and device

Abstract

The invention relates to the technical field of robot middleware and distributed system architecture, in particular to a multi-modal data recording method and device. The method comprises the following steps: receiving a message of at least one data stream from robot middleware; aiming at each received message, executing byte-based admission control operation which comprises the following steps: trying to apply for a corresponding quota permission according to the actual memory occupancy of the message, discarding the message in response to application failure, and associating the quota permission with the message in response to application success; respectively recombining the load data and metadata of the plurality of successfully applied messages to form a batch data unit conforming to a column type storage format; and writing the batch data units into a storage file. According to the method, recording and analysis are realized, zero-copy efficient reading is supported, and the stability, efficiency and integrity of multi-modal data recording are greatly improved.

Description

Technical Field

[0001] This application relates to the field of robot middleware and distributed system architecture technology, and in particular to a multimodal data recording method and apparatus. Background Technology

[0002] In real-time computing fields such as robotics and autonomous driving, efficiently and reliably recording multimodal data streams (such as camera images, LiDAR point clouds, and inertial measurement data) generated during system operation is a crucial foundation for algorithm training, functional verification, and fault diagnosis. Currently, the industry commonly uses the recording tools (such as rosbag) built into robot middleware (such as ROS / ROS 2) or subsequent general-purpose container formats (such as MCAP) as the mainstream solution. While these solutions can meet basic data recording needs, their underlying architecture has inherent and insurmountable flaws when dealing with the stringent requirements of high throughput, strong real-time performance, and heterogeneous data in robot systems. Furthermore, these flaws are interconnected and have a cumulative effect.

[0003] These shortcomings are mainly reflected in three aspects: First, insufficient system stability. Existing tools adopt a passive blocking write model, lacking a fine-grained resource management mechanism. When storage I / O performance fluctuates, the blocking in the recording process will quickly create backpressure and propagate to upstream core business modules such as sensing, positioning, and control, which may lead to real-time system failure or even overall system crash in severe cases. Second, low data processing efficiency. The files generated by the existing solution adopt a row-oriented storage structure, storing raw data line by line according to message sequence. Although suitable for conventional recording scenarios, it is difficult to support column-oriented batch computing and analysis tasks such as AI training. Data must undergo a large amount of deserialization and row-to-column format conversion before use, significantly increasing overhead and forming a pipeline bottleneck from data acquisition to model training. Finally, multi-data stream recording is prone to mutual interference. Existing architectures usually use a shared processing queue and I / O channel to record various types of data uniformly. The instantaneous pressure of high-bandwidth data streams (such as image streams) will crowd out system resources, causing frequent loss of high-frequency, lightweight critical data streams (such as control commands and IMU data), and the recording results cannot truly and completely reflect the system's operating status.

[0004] In summary, existing recording technologies in the robotics field have long faced three intertwined challenges: system instability caused by backpressure propagation, inefficient data processing due to row-based storage, and loss of critical data due to resource contention. To address these issues, this application proposes a method and apparatus for efficient multimodal data recording, aiming to systematically resolve the aforementioned shortcomings at the architectural level. Summary of the Invention

[0005] The multimodal data high-efficiency recording solution provided in this application addresses three core defects in the existing technology: recording backpressure propagation and system real-time failure due to lack of precise resource management; low data processing efficiency and high preprocessing overhead due to the use of row-based storage format; and loss of key data and insufficient recording integrity due to competition for shared resources among multiple data streams.

[0006] The first aspect of this application provides a multimodal data recording method, including: Receive messages from at least one data stream from the robot middleware; For each received message, a byte-based admission control operation is performed, wherein the admission control operation includes: attempting to request a corresponding quota license based on the actual memory usage of the message; discarding the message in response to the request failure; and associating the quota license with the message in response to the request success. The payload data and metadata of multiple successful application messages are reassembled to form a batch data unit that conforms to the columnar storage format. Write the batch data units into a storage file.

[0007] In some embodiments of this application, associating the quota license with the message includes: Transfer ownership of the successfully applied quota license to the message object; The quota permission is released when the message object is written to a storage file or destroyed.

[0008] In some embodiments of this application, the reorganization of the payload data and metadata of multiple successful application messages to form a batch data unit conforming to a columnar storage format includes: The payload data and metadata of the same fields of the multiple successful application messages are stored in the corresponding memory arrays respectively; Based on the memory array, construct a ListArray data structure conforming to the Apache Arrow format; The ListArray data structure is encapsulated into RecordBatch to form the batch data unit.

[0009] In some embodiments of this application, the reassembly of the payload data and metadata of multiple successful application messages, and the writing of the batch data units to a storage file, are performed in an asynchronous task independent of receiving messages; and, The step of writing the batch data units to the storage file includes: A single batch of data units is written to the storage file through a single, continuous I / O operation.

[0010] In some embodiments of this application, before reassembling the payload data and metadata of the multiple successful application messages, the method further includes: An asynchronous transmission channel, controlled by memory quotas, is established for the asynchronous task as its independent data channel; The asynchronous task reads the successful application message from the asynchronous transmission channel and processes it.

[0011] In some embodiments of this application, the method further includes: When a message of a new data type is received for the first time, a dedicated asynchronous processing task isolated at the physical thread level and an independent asynchronous data channel controlled by memory quota are dynamically created to handle that data type. For subsequent messages of the same type, the reassembly and writing are performed through the corresponding independent asynchronous data channel.

[0012] In some embodiments of this application, the method further includes: In response to the stop recording command, stop receiving new messages; After all ongoing reorganizations and writes are complete, close the storage file.

[0013] In some embodiments of this application, the robot middleware is dora-rs; and / or, The at least one data stream includes at least two of the following: lidar, camera, inertial measurement unit, map data, and control commands.

[0014] In some embodiments of this application, the stored file is an ApacheArrow format file containing a self-describing file footer; The data in the storage file is organized in a columnar storage format to support zero-copy reading by Pandas and PyTorch data processing frameworks.

[0015] A second aspect of this application provides a multimodal data recording apparatus, comprising: A receiving module is used to receive messages from at least one data stream from the robot middleware; An admission control module is used to perform a byte-based admission control operation for each received message, wherein the admission control operation includes: attempting to request a corresponding quota license based on the actual memory usage of the message; discarding the message in response to the request failure; and associating the quota license with the message in response to the request success. The reassembly module is used to reassemble the payload data and metadata of multiple successful application messages to form a batch data unit that conforms to the columnar storage format. The writing module is used to write the batch data units into a storage file.

[0016] In summary, the multimodal data recording method and apparatus proposed in the embodiments of this application firstly, by introducing byte-based admission control operations, immediately discards the current message when memory quota allocation fails. This non-blocking lossy strategy actively cuts off the propagation chain of backpressure to upstream core business modules, thereby ensuring the deterministic execution of key tasks such as robot perception and control, fundamentally solving the system instability problem caused by backpressure propagation. Secondly, by reorganizing messages into batch data units conforming to columnar storage format, the conversion from streaming data to columnar structure (such as Apache Arrow RecordBatch) is completed in real time within the recording process. This makes the data organization of the output file natively match analysis tasks such as AI training, realizing recording and analysis simultaneously, and enabling zero-copy reading downstream, completely eliminating the deserialization and conversion overhead required by traditional row-based storage formats, overcoming the bottleneck of low data processing efficiency. In addition, by dynamically creating parallel processing channels with physical thread-level isolation for different data types, the isolation of heterogeneous data streams at the resource and I / O levels is achieved. This approach ensures that high-bandwidth and high-frequency lightweight streams do not block each other during recording, significantly reducing packet loss rates for critical control commands, IMU data, and other information. It effectively solves the problem of critical data loss caused by resource contention, guaranteeing the integrity of the recording. These technical features are designed collaboratively from the perspectives of stability, efficiency, and integrity, providing robotic systems with a high-performance, highly reliable data recording solution native to AI training. Attached Figure Description

[0017] The features and advantages of this application will become clearer with reference to the accompanying drawings, which are illustrative and should not be construed as limiting the application in any way. In the drawings: Figure 1 This is a schematic diagram of the system architecture of a multimodal data recording method according to some embodiments of this application; Figure 2 This is a flowchart of the synchronous main loop processing according to some embodiments of this application; Figure 3 This is a schematic diagram of data reorganization and writing in an asynchronous write task according to some embodiments of this application; Figure 4 This is a flowchart illustrating a multimodal data recording method according to some embodiments of this application; Figure 5This is a schematic diagram of a multimodal data recording apparatus according to some embodiments of this application. Detailed Implementation

[0018] In the following detailed description, numerous specific details of this application are illustrated by example to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this application can be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this application are one way of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced with other expressions if other expressions can achieve the same purpose.

[0019] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to or communicate with other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used herein includes any one and all combinations of one or more of the relevant listed items.

[0020] The terminology used in this application is for the purpose of describing specific embodiments only and is not intended to limit the scope of this application. As shown in the specification and claims of this application, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may also include the plural. Generally speaking, the terms "comprising" and "including" only indicate that explicitly identified features, integrals, steps, operations, elements, and / or components are included, and such expressions do not constitute an exclusive list, and other features, integrals, steps, operations, elements, and / or components may also be included.

[0021] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this application can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. It is understood that the drawings are not drawn to scale.

[0022] Various structural diagrams are used in this application to illustrate various variations of the embodiments according to this application. It should be understood that the preceding or following structures are not intended to limit this application. The scope of protection of this application is determined by the claims.

[0023] In the research and deployment of real-time systems such as robotics and autonomous driving, the complete and reliable recording of multimodal data streams (such as images, LiDAR point clouds, inertial measurement unit data, and control commands) generated during system operation is the core foundation for algorithm iteration, functional verification, and post-fault diagnosis. This recording process is not a simple data dump, but a professional operation for the efficient capture, real-time processing, and persistent storage of high-throughput, high-real-time data streams. Currently, the field generally relies on robot middleware ecosystems such as ROS / ROS 2 and uses recording tools like rosbag, which are based on row-oriented storage and blocking I / O, to record data. However, the underlying architecture of such solutions is difficult to adapt to the stringent data processing requirements of modern robot systems, and has the following interrelated fundamental defects: First, the system lacks stability. Due to the lack of a fine-grained memory and I / O resource management mechanism, recording blockage caused by disk write latency will create back pressure and spread to upstream core business modules such as perception and control, which can easily lead to real-time system failure. Second, the data processing efficiency is low. The row-oriented storage file structure, which stores messages sequentially line by line, is inherently incompatible with the columnar computing requirements of downstream tasks such as AI training. The consumer must perform high-overhead deserialization and row-to-column format conversion operations. Third, the recording integrity is difficult to guarantee. Heterogeneous data streams share processing resources and I / O channels. The instantaneous pressure of high-bandwidth data streams (such as image streams) can easily cause channel blockage, which in turn leads to the loss of high-frequency lightweight critical data streams such as control commands and IMU data.

[0024] To systematically address the aforementioned technical deficiencies, this application proposes a method and apparatus for efficient multimodal data recording. The core of this method is a two-layer processing architecture that deeply decouples a synchronous master control loop from an asynchronous task pool (e.g., ...). Figure 1 As shown): Synchronous master control loop (its logic flow is shown in...) Figure 2 As the system's data entry point, it performs byte-level quota-based access control on incoming messages, requests quota permissions as needed and performs discarding or association processing, and routes messages to corresponding independent data channels according to data type. It also cuts off backpressure propagation at the source through a lossy recording strategy. The asynchronous task pool consists of dynamically derived asynchronous processing task units that are physically thread-isolated (e.g., ...). Figure 3 As shown, each unit reads messages from a dedicated channel and reassembles identical fields from multiple messages into columnar RecordBatch batch data units conforming to the Apache Arrow standard. This application deeply utilizes the core features of the Apache Arrow columnar memory model—contiguous column storage, native compatibility with computing frameworks, and self-describing files—to achieve the technical goal of "recording as analysis." The generated Apache Arrow format (.arrow) files support zero-copy direct reading and include a self-describing file footer. The final generated RecordBatch is written to disk in batches through continuous I / O operations.

[0025] This application optimizes multimodal data recording capabilities from three dimensions: system stability, data processing efficiency, and recording integrity, through the collaborative optimization of three core mechanisms: byte-based admission control, dynamic isolation of heterogeneous data streams, and real-time columnar reorganization based on Apache Arrow. It provides a native, high-performance, and highly reliable recording solution for AI training in high real-time fields such as robotics and autonomous driving.

[0026] The following detailed description of the efficient multimodal data recording scheme of this application is provided in conjunction with the accompanying drawings.

[0027] Figure 4 This is a flowchart illustrating a multimodal data recording method according to some embodiments of this application. For example... Figure 4 As shown, the method includes: S410. Receive a message from at least one data stream from the robot middleware.

[0028] like Figure 1 The "Synchronization Main Loop" section and Figure 2 As shown, the system's main loop (event_loop) runs continuously, receiving and processing various events distributed by the robot middleware in real time. Input events represent standard message transmission events within the middleware and are the primary focus of the recording object. The main loop employs differentiated processing logic for different events: Input event: The system determines it as data to be recorded, immediately extracts the message identifier, timestamp and data payload, and calls the data interface to obtain the actual byte size of the message in memory, providing a basis for subsequent access control.

[0029] Non-input events: If an error event is detected, the system performs error interception and logs the event to monitor the transmission link status. If an InputClosed signal or a manual stop signal is detected, the system will stop receiving new data and enter a blocking waiting state until all existing asynchronous reassembly tasks are completed, ensuring the integrity of data termination.

[0030] S420. For each of the received messages, perform a byte-based admission control operation, wherein the admission control operation includes: attempting to request a corresponding quota license based on the actual memory usage of the message; discarding the message in response to a failed request; and associating the quota license with the message in response to a successful request.

[0031] After extracting the fields of the message, the system performs access control.

[0032] Traditional recording tools often use fixed-length FIFO queues limited by the number of messages as their buffers. This has significant drawbacks in multimodal data streams from robots: the size of data from different sensors can vary by six orders of magnitude (from bytes to MB). Limiting by the number of messages cannot accurately reflect the actual memory usage. This can lead to situations where, when recording large messages (such as image frames), the number of messages in the recording node buffer does not exceed the limit, but the total memory consumed exceeds the physical memory threshold, thus triggering the operating system's memory protection mechanism (such as OOM-Killer) and terminating the process, resulting in system crashes.

[0033] To address this deficiency, this application introduces a byte-quota-based semaphore mechanism. The system pre-allocates a shared memory semaphore for each data type. The semaphore's counter is measured in bytes; attempting to acquire a corresponding number of permissions involves atomically decreasing the counter by a value equal to the number of bytes occupied by the message memory. The initial counter is set to a preset buffer capacity (e.g., 16MB). When the main loop receives a message: 1. Real-time measurement: Obtain the exact physical memory usage of this message.

[0034] 2. Atomic access: Attempt to acquire the corresponding number of licenses.

[0035] 3. Deterministic Discard: If the request fails (the buffer is full), the operation of discarding the message is performed. In this scenario, the system immediately discards the current frame.

[0036] This "lossy recording" strategy sacrifices the integrity of non-critical frames in exchange for the absolute real-time performance of the system's core logic (such as obstacle avoidance and planning), thus eliminating backpressure propagation caused by disk I / O fluctuations at the source.

[0037] 4. Lifecycle Binding: A successfully requested permission is encapsulated in a message object. The permission flows between asynchronous tasks along with the message and is automatically released only when the data is successfully written to disk or destroyed due to an exception, thus achieving extremely precise memory control.

[0038] S430. The payload data and metadata of the multiple successful application messages are recombined to form a batch data unit conforming to the columnar storage format.

[0039] After successful admission control, the system needs to route messages based on their data type. For example... Figure 1 and Figure 2 As shown, the system "finds the corresponding pipe based on data_type".

[0040] In some embodiments of this application, the system internally maintains an active type mapping table. This active type mapping table is a dataset maintained internally by the system to store the mapping relationship between data types and corresponding asynchronous processing resources (pipelines, tasks, writers), supporting dynamic addition and deletion of data types and fast retrieval. Its processing logic is as follows: Type recognition and environment setup: Whenever a data stream with a new type (new schema) is detected, the system automatically triggers an asynchronous environment initialization process to create: A dedicated producer-consumer asynchronous pipeline; A semaphore initialized to a preset buffer size (e.g., 16MB); A standalone Arrow file writer; An asynchronous processing task responsible for reassembling and writing data to disk.

[0041] Existing type index: If the received message belongs to a known type that already exists in the mapping table, the system will not create the environment again. Instead, it will directly retrieve the communication pipe handle corresponding to the type according to the type mapping table and prepare to make the next step of semaphore admission request, and attempt to send the message into the asynchronous processing pipeline corresponding to the pipe.

[0042] This design ensures that sensor data of different frequencies and bandwidths do not interfere with each other in the processing pipeline. Even if the video stream is briefly blocked due to encoding delays, the high-frequency IMU channel can continue high-frequency recording in an independent thread context through the established mapping relationship.

[0043] like Figure 1 "Asynchronous write task" and Figure 3 As shown, each data type's corresponding asynchronous write task continuously reads messages from its dedicated, memory-quota-controlled mpsc channel (multi-producer, single-consumer channel) to achieve efficient memory reuse and lock-free or low-lock concurrency. This is a key step in realizing "record-as-analysis".

[0044] The system abandons the traditional row-based binary storage scheme and switches to columnar transformation based on the Apache Arrow standard. After receiving messages from the pipeline, the task does not immediately write them to disk, but instead parses out fields such as message payload, timestamp, and source topic, and temporarily stores them in memory arrays.

[0045] When the accumulated data volume reaches a threshold, several buffer arrays of the same length will be generated (such as message payload arrays, timestamp arrays, source topic arrays, etc.). At this point, the asynchronous task performs a reorganization operation: Offset calculation: The system will iterate through all temporarily stored message carriers in the message payload array and calculate the offset.

[0046] Memory splicing: Multiple non-contiguous small arrays are concatenated into a large array in memory at once.

[0047] Building ListArray and RecordBatch: By combining the concatenated arrays with offset information, a ListArray conforming to the Arrow standard is constructed. Each message has its own fields, and since each field is stored separately in an array buffer, these arrays are guaranteed to be of equal length. They are ultimately assembled into a RecordBatch.

[0048] S440. Write the batch data units into a storage file.

[0049] The RecordBatch generated by the S430 is written to disk using file operators. This method transforms numerous small I / O operations into large, sequential I / O writes, greatly improving the sequential write performance of the disk.

[0050] After the write is complete, the worker task clears the memory vector. At this time, the memory quota permission that flowed with the message is automatically released, and the corresponding memory quota is returned to the semaphore for subsequent message requests.

[0051] The system achieves graceful shutdown by listening to middleware lifecycle events. When the recording end signal is triggered: Explicitly close: The main loop actively releases the sender handles of all work tasks, triggering the closing logic of the asynchronous channel.

[0052] Force clear: Upon receiving the channel closure signal, each task will force the remaining data in memory, which is less than a batch size, to undergo a final reassembly and disk write-to-disk process.

[0053] Write the following in the file footer: Finally, the `finish` method of the Arrow writer is called to supplement the metadata index information in the file footer, ensuring that the generated `.arrow` file is self-descriptive. Because the Apache Arrow format uses a columnar memory layout and is compatible with the memory model of the data processing framework, the stored file can be efficiently read by frameworks such as Pandas and PyTorch in a zero-copy or near-zero-copy manner.

[0054] Ultimately, the system generates multiple .arrow files categorized by data type. The data within each file is arranged in a columnar format, containing a list of original carriers, a unified timestamp, and topic tags.

[0055] Figure 5 This is a schematic diagram of a multimodal data recording apparatus according to some embodiments of this application. Figure 5 As shown, the multimodal data recording device 500 includes a receiving module 510, an admission control module 520, a reconstruction module 530, and a writing module 540. Wherein: The receiving module 510 is used to receive messages from at least one data stream from the robot middleware; The admission control module 520 is used to perform a byte-based admission control operation for each received message, wherein the admission control operation includes: attempting to apply for a corresponding quota license based on the actual memory usage of the message; discarding the message in response to application failure; and associating the quota license with the message in response to application success. The reorganization module 530 is used to reorganize the payload data and metadata of multiple successful application messages to form a batch data unit that conforms to the columnar storage format. The writing module 540 is used to write the batch data units into a storage file.

[0056] In summary, the multimodal data recording method and apparatus proposed in the embodiments of this application firstly, by introducing byte-based admission control operations, immediately discards the current message when memory quota allocation fails. This non-blocking lossy strategy actively cuts off the propagation chain of backpressure to upstream core business modules, thereby ensuring the deterministic execution of key tasks such as robot perception and control, fundamentally solving the system instability problem caused by backpressure propagation. Secondly, by reorganizing messages into batch data units conforming to columnar storage format, the conversion from streaming data to columnar structure (such as Apache Arrow RecordBatch) is completed in real time within the recording process. This makes the data organization of the output file natively match analysis tasks such as AI training, realizing recording and analysis simultaneously, and enabling zero-copy reading downstream, completely eliminating the deserialization and conversion overhead required by traditional row-based storage formats, overcoming the bottleneck of low data processing efficiency. In addition, by dynamically creating parallel processing channels with physical thread-level isolation for different data types, the isolation of heterogeneous data streams at the resource and I / O levels is achieved. This approach ensures that high-bandwidth and high-frequency lightweight streams do not block each other during recording, significantly reducing packet loss rates for critical control commands, IMU data, and other information. It effectively solves the problem of critical data loss caused by resource contention, guaranteeing the integrity of the recording. These technical features are designed collaboratively from the perspectives of stability, efficiency, and integrity, providing robotic systems with a high-performance, highly reliable data recording solution native to AI training.

[0057] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0058] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0059] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0060] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this application and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this application should be included within the protection scope of this application. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A multimodal data recording method, characterized in that, include: Receive messages from at least one data stream from the robot middleware; For each received message, a byte-based admission control operation is performed, wherein the admission control operation includes: attempting to request a corresponding quota license based on the actual memory usage of the message; discarding the message in response to the request failure; and associating the quota license with the message in response to the request success. The payload data and metadata of multiple successful application messages are reassembled to form a batch data unit that conforms to the columnar storage format. Write the batch data units into a storage file.

2. The method according to claim 1, characterized in that, Associating the quota license with the message includes: Transfer ownership of the successfully applied quota license to the message object; The quota permission is released when the message object is written to a storage file or destroyed.

3. The method according to claim 1, characterized in that, The step of reassembling the payload data and metadata of multiple successful application messages to form a batch data unit conforming to a columnar storage format includes: The payload data and metadata of the same fields of the multiple successful application messages are stored in the corresponding memory arrays respectively; Based on the memory array, construct a ListArray data structure conforming to the Apache Arrow format; The ListArray data structure is encapsulated into RecordBatch to form the batch data unit.

4. The method according to claim 1, characterized in that: The reassembly of the payload data and metadata of the multiple successful application messages and the writing of the batch data units to the storage file are performed in an asynchronous task independent of receiving messages. and, The step of writing the batch data units to the storage file includes: A single batch of data units is written to the storage file through a single, continuous I / O operation.

5. The method according to claim 4, characterized in that, Before reassembling the payload data and metadata of the multiple successful application messages, the method further includes: An asynchronous transmission channel, controlled by memory quotas, is established for the asynchronous task as its independent data channel; The asynchronous task reads the successful application message from the asynchronous transmission channel and processes it.

6. The method according to claim 1, characterized in that, The method further includes: When a message of a new data type is received for the first time, a dedicated asynchronous processing task isolated at the physical thread level and an independent asynchronous data channel controlled by memory quota are dynamically created to handle that data type. For subsequent messages of the same type, the reassembly and writing are performed through the corresponding independent asynchronous data channel.

7. The method according to claim 1 or 4, characterized in that, The method further includes: In response to the stop recording command, stop receiving new messages; After all ongoing reorganizations and writes are complete, close the storage file.

8. The method according to claim 1, characterized in that, The robot middleware is dora-rs; and / or, The at least one data stream includes at least two of the following: lidar, camera, inertial measurement unit, map data, and control commands.

9. The method according to claim 1, characterized in that: The stored file is an Apache Arrow format file containing a self-describing file footer; The data in the storage file is organized in a columnar storage format to support zero-copy reading by Pandas and PyTorch data processing frameworks.

10. A multimodal data recording device, characterized in that, include: A receiving module is used to receive messages from at least one data stream from the robot middleware; An admission control module is used to perform a byte-based admission control operation for each received message, wherein the admission control operation includes: attempting to request a corresponding quota license based on the actual memory usage of the message; discarding the message in response to the request failure; and associating the quota license with the message in response to the request success. The reassembly module is used to reassemble the payload data and metadata of multiple successful application messages to form a batch data unit that conforms to the columnar storage format. The writing module is used to write the batch data units into a storage file.

Images