Reading method and device of embodied intelligence training data, medium and equipment

Abstract

The present disclosure relates to the technical field of data processing, and provides a reading method and device for embodied intelligent training data, a medium and equipment, comprising: loading a global trajectory index in response to a data loading request, which records the physical address and boundary position of each training trajectory; in response to a target sample reading request, locating a target data block according to the physical address obtained by querying the index according to the trajectory to which the target sample belongs; the data block contains a column storage area (storing metadata at each time step) and a tensor storage area (storing images organized in micro-batches); reading a block header offset index, querying according to the sample time step, judging the type of the storage area, if it is column type, reading metadata, if it is tensor type, obtaining the position according to the offset index to read images; judging whether it is cross-trajectory according to the boundary position of the global index, and generating a boundary mask; assembling and returning the data and the mask. The embodiment realizes fast reading through global index and partition storage, guarantees correctness in combination with the boundary mask, and improves reading efficiency and resource utilization.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more specifically, to a method, apparatus, medium, and device for reading embodied intelligence training data. Background Technology

[0002] With the rapid development of embodied intelligence model training, the scale of training data has increased dramatically and its modalities are highly heterogeneous. This data includes both high-frequency structured metadata such as action commands and massive amounts of unstructured visual tensors such as images and depth maps, exhibiting strong temporal dependencies. Robot training tasks typically require organizing data at the millisecond time step level; a single trajectory can contain tens of thousands of time steps and corresponding multi-view image sequences, placing extremely high demands on the read / write bandwidth, retrieval latency, and cross-modal alignment capabilities of storage systems. Currently, the industry has adopted various data formats such as HDF5, Parquet, Zarr, MCAP, and ROS bag to address this challenge. However, these formats, stemming from different design motives, have yet to provide a unified and efficient storage and retrieval solution specifically for the unique workload of robot data.

[0003] While various data formats exist in related technologies, their fundamentally different designs often fall short in meeting the core requirements of embodied training. The fragmented storage of large visual objects and their metadata leads to lengthy random I / O paths. For example, Parquet externalizes images as video files, requiring repeated decoding and localization during training. HDF5 and Zarr, on the other hand, lack columnar joint retrieval capabilities, making it difficult to support the high-frequency operation of simultaneously querying action tags and corresponding images at time steps. Furthermore, existing trajectory data formats are often scattered across different files or large contiguous blocks. Loading a single sample fails to detect trajectory boundaries, necessitating the maintenance of complex mapping tables for cross-trajectory access and easily introducing invalid padding. This restricts the sampling efficiency and throughput of the training framework, making data loading a significant bottleneck in model iteration. Summary of the Invention

[0004] This disclosure provides at least one method, apparatus, medium, and device for reading embodied intelligent training data. By constructing a global trajectory index and using columnar and tensor region separation storage and intra-block offset index, it achieves rapid positioning and on-demand reading of training samples. At the same time, it uses boundary masks to ensure the temporal correctness of cross-trajectory samples, thereby improving data reading efficiency and system resource utilization.

[0005] This disclosure provides a method for reading embodied intelligence training data, including: In response to a data loading request, a global trajectory index is loaded. The global trajectory index records the physical address information of each training trajectory in the data file and the boundary position information of each training trajectory. In response to a read request for a target training sample, the global trajectory index is queried based on the target training trajectory to which the target training sample belongs, to obtain the physical address information of the target training trajectory; Based on the physical address information, the target data block where the target training sample is located is located in the data file; wherein, the target data block includes a columnar storage area and a tensor storage area, the columnar storage area is used to store metadata corresponding to the time step, and the tensor storage area is used to store image data organized in micro-batches, with the starting frame of each micro-batch being an independent decoding frame; Read the offset index of the header of the target data block, and query the offset index according to the time step position of the target training sample to determine the storage area type of the target data of the target training sample in the target data block; If the storage area type is the columnar storage area, the target metadata of the target training sample is read from the columnar storage area of ​​the target data block; if the storage area type is the tensor storage area, the storage location of the target training sample in the tensor storage area is obtained according to the offset index, and the target image data of the target training sample is read from the tensor storage area. Based on the boundary position information in the global trajectory index, determine whether the target training sample involves crossing trajectory boundaries and generate corresponding boundary mask information; Based on the data of the target training sample read from the target data block and the boundary mask information, the target training sample is assembled and returned.

[0006] This disclosure provides an embodiment of a device for reading embodied intelligence training data, including: The index loading module is used to load the global trajectory index in response to the data loading request. The global trajectory index records the physical address information of each training trajectory in the data file and the boundary position information of each training trajectory. The address determination module is used to respond to a read request for a target training sample by querying the global trajectory index based on the target training trajectory to which the target training sample belongs, and obtaining the physical address information of the target training trajectory. A data block determination module is used to locate the target data block where the target training sample is located in the data file according to the physical address information; wherein, the target data block includes a columnar storage area and a tensor storage area, the columnar storage area is used to store metadata corresponding to the time step, and the tensor storage area is used to store image data organized in micro-batches, and the starting frame of each micro-batch is an independent decoding frame; The region determination module is used to read the offset index of the header of the target data block, and query the offset index according to the time step position of the target training sample to determine the storage region type of the target data of the target training sample in the target data block; The data reading module is used to read the target metadata of the target training sample from the columnar storage area of ​​the target data block if the storage area type is the columnar storage area; and to obtain the storage location of the target training sample in the tensor storage area according to the offset index if the storage area type is the tensor storage area, and to read the target image data of the target training sample from the tensor storage area. The boundary determination module is used to determine whether the target training sample involves crossing the trajectory boundary based on the boundary position information in the global trajectory index, and to generate corresponding boundary mask information. The data assembly module is used to assemble the target training sample based on the data of the target training sample read from the target data block and the boundary mask information, and then return the data.

[0007] This disclosure provides a computer device, including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the method for reading embodied intelligence training data as described in any of the above possible embodiments is executed.

[0008] This disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for reading embodied intelligence training data as described in any of the possible embodiments above.

[0009] The embodied intelligence training data reading method, apparatus, medium, and device provided in this disclosure construct a global trajectory index that records the physical address and boundary position of each training trajectory. During data loading, only this index needs to be loaded to respond to sample reading requests, avoiding traversing the entire data file and reducing reading latency. Furthermore, by dividing the data file into columnar storage areas and tensor storage areas to store metadata and image data organized in micro-batches with independent decoding frames as the starting frame, respectively, and using a data block header offset index to locate the storage area type and tensor storage position, separate storage and retrieval of metadata and image data, and independent decoding and reading of image data are achieved. This reduces the amount of irrelevant data loaded and improves the positioning accuracy and reading efficiency of target samples. Simultaneously, a boundary mask is generated based on the boundary position information in the global trajectory index. When assembling samples, the handling method across trajectory boundaries is clearly defined, ensuring the temporal continuity of training samples and accurate distinction of trajectory boundaries, avoiding data confusion during cross-trajectory sample assembly. Therefore, while ensuring data integrity and correctness, the overall reading speed and system resource utilization of embodied intelligence training data are improved.

[0010] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings referenced in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0012] Figure 1 A flowchart illustrating a method for reading embodied intelligence training data provided in an embodiment of this disclosure is shown; Figure 2 A flowchart illustrating a method for constructing a global trajectory index provided in an embodiment of this disclosure is shown; Figure 3 A flowchart illustrating a method for reading target image data from a tensor storage region according to an embodiment of this disclosure is shown; Figure 4 A flowchart illustrating a method for aligning and reading multi-frequency data streams according to an embodiment of this disclosure is shown. Figure 5 A schematic diagram of the structure of a device for reading embodied intelligent training data provided in an embodiment of this disclosure is shown; Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0014] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0015] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0016] In the field of embodied intelligence model training, the choice of data format directly affects training efficiency and scalability. Currently, HDF5 and LeRobotDataset (based on Parquet + MP4 / images) are the most widely used data formats, while TFRecord, Zarr, MCAP, ROS bag, and LMDB are also used in different scenarios. These formats originate from different design backgrounds such as scientific computing, tabular data processing, streaming message recording, or key-value storage, and each has its own advantages. For example, HDF5 and Zarr are good at storing large multidimensional arrays, LeRobot / Parquet performs well in structured table queries, and MCAP / ROSbag provides efficient timestamp indexes for high-frequency message streams. However, robot data has unique characteristics such as high frequency heterogeneity, dense visual information, and strong temporal dependence, which puts stringent requirements on random access, multimodal joint retrieval, variable-length sequence sampling, and high-concurrency reading of storage systems. The above formats have yet to form a unified and efficient solution.

[0017] Research revealed that while each of the aforementioned data formats has its advantages, none can simultaneously meet the rigid requirements of dense visual storage, temporal sequence sampling, and high-concurrency reading in embodied robot training scenarios. This manifests in three interconnected structural defects. First, there is a separation between the storage of visual data and metadata. LeRobot / Parquet externalizes images as video files, necessitating repeated random I / O for localization, frame extraction, and decoding during training. While HDF5 and Zarr can store arrays inline, their array layer and table metadata are independent, preventing joint retrieval using columnar filtering. Operations such as filtering trajectories by attribute must be implemented additionally at a higher layer. Second, while message stream formats like MCAP / ROS bag have temporal indexes, they lack multimodal alignment support for training samples, resulting in a heavy preprocessing burden. This fragmentation directly exacerbates the second problem: the temporal correlation representation is too basic. Existing formats are only statically arranged by time step or frame number, lacking native indexes designed for variable-length trajectories and sliding window sampling. This results in long sequence construction requiring additional unpacking and rearrangement, and during sliding window sampling, irrelevant context is easily mixed in across episode boundaries, compromising data independence and task consistency. The shortcomings of the above two aspects ultimately converge into concurrency and scalability bottlenecks—HDF5 is prone to blocking under multiple workers due to file locks and object serialization issues, TFRecord lacks random access and is constrained by Protobuf, Zarr attribute filtering is cumbersome, and LMDB's key-value semantics lack native awareness of high-dimensional arrays. Each format requires a heavy adaptation layer to be integrated into the training pipeline, significantly reducing the efficiency of data supply for large-scale distributed training.

[0018] Based on the above research, this disclosure provides a method, apparatus, medium, and device for reading embodied intelligent training data. Specifically, in response to a data loading request, a global trajectory index is loaded, which records the physical address and boundary position of each training trajectory in the data file. In response to a reading request for a target sample, the physical address is obtained by querying the index based on the trajectory to which the target sample belongs, thereby locating the target data block in the data file. The data block includes a columnar storage area (used to store metadata corresponding to each time step) and a tensor storage area (used to store image data organized in micro-batches, with the starting frame of each micro-batch being an independent decoding frame). The offset index at the header of the data block is read, and the offset index is queried based on the time step position of the sample to determine whether the target data is stored in the columnar area or the tensor area. If it is columnar, the metadata is read directly; if it is tensor, the storage location is obtained based on the offset index, and the image data is read. At the same time, the boundary position information in the global index is used to determine whether the sample crosses the trajectory boundary, and a corresponding boundary mask is generated. Finally, the read data and the mask are assembled to return a complete training sample.

[0019] In this embodiment, a global trajectory index is constructed to record the physical address and boundary position of each training trajectory. During data loading, only this index is loaded to respond to sample reading requests, avoiding traversing the entire data file and reducing reading latency. Furthermore, by dividing the data file into columnar storage areas and tensor storage areas to store metadata and image data organized in micro-batches with independent decoding frames as the starting frame, respectively, and using data block header offset indexes to locate the storage area type and tensor storage position, separate access to metadata and image data and independent decoding and reading of image data are achieved. This reduces the amount of irrelevant data loaded and improves the positioning accuracy and reading efficiency of target samples. Simultaneously, a boundary mask is generated based on the boundary position information in the global trajectory index. When assembling samples, the handling method across trajectory boundaries is clearly defined, ensuring the temporal continuity of training samples and accurate distinction of trajectory boundaries. This avoids data confusion during cross-trajectory sample assembly, thereby improving the overall reading speed and system resource utilization of embodied intelligent training data while ensuring data integrity and correctness.

[0020] To facilitate understanding of this embodiment, the executing entity of the embodied intelligence training data reading method provided in this disclosure will first be described in detail. The executing entity of the embodied intelligence training data reading method provided in this disclosure is a computer device. This computer device can be a terminal device or a server. The terminal device can also be a mobile device, a user terminal, a terminal, a handheld device, a computing device, an in-vehicle device, a wearable device, etc. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. Optionally, this method can also be applied to an implementation environment composed of computer devices and servers.

[0021] The method for reading embodied intelligence training data provided in the embodiments of this application will be described in detail below with reference to the accompanying drawings. See also Figure 1 The diagram shown is a flowchart of a method for reading embodied intelligence training data according to an embodiment of this disclosure. The method for reading embodied intelligence training data includes the following steps S101 to S107: S101, in response to the data loading request, loads the global trajectory index.

[0022] Understandably, a data load request is a read instruction generated when starting an embodied intelligent model training task. This read instruction triggers the loading of the dataset metadata required for training from the storage system. In engineering practice of embodied intelligent model training, data load requests are typically issued by the dataset initialization module in the training program. This module initiates a data read call to the operating system when building the data loader. The data load request carries identification information of the target dataset, which specifies the path to the data file to be read and the range of training trajectories contained in the dataset. Here, in response to the data load request, a global trajectory index loading operation is performed. This allows the global trajectory index to be read from the storage medium into memory space, which is used in subsequent steps to quickly locate the storage location of each training sample in the data file.

[0023] Specifically, the global trajectory index is a mapping data used to describe the storage distribution of each training trajectory in the data file. This index records the physical address information of each training trajectory in the data file and the boundary position information of each training trajectory. The global trajectory index is pre-built during the dataset preparation stage and stored in association with the data file. During the training stage, it is loaded into memory by the data loader for querying.

[0024] In this context, a training trajectory refers to the basic data organization unit during the training process of an embodied intelligence model. Each training trajectory corresponds to the complete data sequence collected by the robot from the initial state to the final state during a single task execution. For example, a training trajectory may include the data recorded during the entire process of the robot moving from the starting position to the target position and completing the grasping operation in one operation round. The training trajectories are statistically independent, and different training trajectories correspond to different task execution processes. Their data content is discontinuous in time and semantically isolated.

[0025] Specifically, physical address information is spatial description information used to locate the storage position of training trajectories in the data file. This physical address information may include a data block identifier, the starting line offset within the data block, and the trajectory length. The data block identifier indicates which data block in the data file the training trajectory is stored in, the starting line offset indicates the starting line position of the training trajectory within that data block, and the trajectory length indicates the total number of time steps contained in the training trajectory. Boundary position information of the training trajectory is used to mark the start and end positions of each training trajectory. This boundary position information includes a start boundary position and an end boundary position. The start boundary position indicates the line position of the first time step of the training trajectory, and the end boundary position indicates the line position of the last time step of the training trajectory.

[0026] For example, the global trajectory index is an indexing mechanism proposed based on the storage distribution structure of each training trajectory in the data file. This indexing mechanism can provide a fast physical address location service for sample queries during the training phase, referring to... Figure 2 As shown, its specific construction steps may include the following steps S201~S204: S201, acquire data from multiple training trajectories.

[0027] Here, the acquired data refers to the raw data obtained from the robot's sensors and control system through the data acquisition system. This acquired data is used as the raw material for constructing the training dataset. The acquired data for each training trajectory includes a metadata sequence and an image data sequence arranged by time steps. A time step is the basic time unit for data sampling, and each time step corresponds to a sampling moment. The time steps are arranged in chronological order of acquisition time to form a temporally continuous data sequence.

[0028] The metadata sequence refers to the set of structured data, excluding image data, recorded at each time step. This metadata sequence is arranged sequentially according to the time steps, with the metadata at each time step temporally aligned with the image data at the same time step. Specifically, the metadata may include the robot's joint angle values, joint angular velocity values, joint torque values, spatial pose information of the end effector, robot base position information, control command signals, and task status marker information, etc. The image data sequence refers to the set of image data acquired by the robot's vision sensors at each time step. This image data sequence is arranged sequentially according to the time steps, with the image data at each time step temporally aligned with the metadata at the same time step. Specifically, the image data may include RGB color images and depth images. For robot systems equipped with multiple vision sensors, the image data sequence at each time step also includes multi-channel image data from multiple camera views.

[0029] S202, the collected data of each training trajectory is divided into multiple data blocks according to a preset data block size, and a data file is generated.

[0030] Furthermore, after acquiring the collected data from multiple training trajectories, the collected data from all training trajectories can be logically organized and divided according to a preset data block size to generate a data file. The preset data block size controls the batch granularity of each write operation, and its value can be configured according to the transmission characteristics of the storage medium and the read mode of the training platform; for example, it can be set to 256 megabytes. A data block is the basic unit of a data file, and each data block contains a continuous data segment, which may contain partial or complete data spanning multiple training trajectories. The data file is a binary file used to persistently store all the collected training trajectory data. This data file can be constructed by appending data, with newly generated training trajectory data sequentially appended to the end of the data file in units of data blocks. The multiple data blocks contained in the data file are arranged contiguously in physical storage, and each data block is organized according to a dual-area structure of columnar storage and tensor storage. The data file is stored in association with the global trajectory index constructed in subsequent steps.

[0031] S203, for each data block, the metadata of each time step in the data block is compressed column-wise and written into the columnar storage area of ​​the data block, and the image data of each time step in the data block is organized into micro-batches and written into the tensor storage area of ​​the data block, and the starting frame of each micro-batch is encoded into an independent decoding frame.

[0032] Here, after dividing the data blocks, the internal storage area can be divided for each data block, namely, the data block can be divided into two independent data storage areas: a columnar storage area and a tensor storage area. The columnar storage area is dedicated to storing metadata, and the tensor storage area is dedicated to storing image data. Specifically, the metadata of each time step within the data block can be compressed column-wise and written to the columnar storage area, while the image data of each time step within the data block can be organized into micro-batches and written to the tensor storage area.

[0033] Specifically, the columnar storage area is a data storage area organized according to a columnar storage structure. Within this area, the values ​​of the same type of metadata at each time step are stored consecutively. For example, joint angle data for all time steps are stored in one column, and end effector pose data for all time steps are stored in another column. Columnar storage combined with compression algorithms can reduce the storage space occupied by data. In scenarios where only a portion of the metadata needs to be read, only the required columns are read, rather than all the metadata. The tensor storage area is a data storage area organized according to tensor blocks. Within this area, image data for each time step is stored consecutively in micro-batches. The starting frame of each micro-batch is encoded as an independent decoding frame. When reading image frames within any micro-batch, decoding can begin independently from the starting frame of that micro-batch, without relying on data from other micro-batches before or after it.

[0034] For example, when compressing by column, the Zstd compression algorithm or the LZ4 compression algorithm can be used to compress and encode each column of data to reduce storage volume and improve read and write efficiency.

[0035] Specifically, in the process of organizing image data into micro-batches and writing it into the tensor storage area, the following steps (1) to (4) can be performed: (1) Obtain image data of multiple time steps arranged consecutively within a data block; (2) Divide the image data of multiple time steps into at least one micro-batch according to the preset micro-batch length. Each micro-batch includes consecutive image frames with a number equal to the preset micro-batch length. (3) For each micro-batch, the starting frame of the micro-batch is encoded as an intra-coded frame, and the remaining frames in the micro-batch except for the starting frame are encoded as inter-predictive coded frames. (4) Write each encoded micro-batch into the tensor storage area sequentially according to the time step, and record the starting storage address and preset micro-batch length of each micro-batch in the offset index of the data block header.

[0036] Understandably, image data from multiple consecutive time steps within a data block can be acquired first, and then these image data can be grouped and divided according to a preset micro-batch length, dividing the image data from consecutive time steps into at least one micro-batch. Here, a micro-batch refers to a data group that organizes image data from multiple consecutive time steps into a single encoding / decoding processing unit. Each micro-batch contains a sequence of consecutive image frames equal to the preset micro-batch length. The preset micro-batch length is a pre-configured integer value used to control the number of image frames contained in each micro-batch. This length is set according to the balance requirements of compression efficiency and random access granularity. For example, the preset micro-batch length can be set to 30 frames, 50 frames, or 100 frames, and can be configured according to the characteristics of the dataset and the needs of the training task during specific implementation.

[0037] Furthermore, after dividing the data into micro-batches, each micro-batch can be encoded. Specifically, the starting frame of the micro-batch is encoded as an intra-coded frame, and the remaining frames within the micro-batch (excluding the starting frame) are encoded as inter-predictive coded frames. Here, an intra-coded frame is an image frame compressed using only the spatial redundancy information of the current frame itself; this frame can be independently decoded and reconstructed into a complete image without relying on data from other frames. An inter-predictive coded frame is an image frame compressed using the temporal redundancy information between the current frame and other encoded frames within the same micro-batch; this frame requires reference to a reference frame within the same micro-batch during decoding. The encoding methods for these intra-coded and inter-predictive coded frames are consistent with the encoding principles of I-frames and P / B frames in common video compression standards, and will not be elaborated further in this embodiment.

[0038] In this way, after encoding each micro-batch is completed, the encoded micro-batch can be continuously written into the tensor storage area in the order of time steps. During the writing process, each micro-batch is arranged sequentially in the tensor storage area, and the internal frame order of each micro-batch remains unchanged. The starting storage address and preset micro-batch length of each micro-batch are recorded in the offset index of the data block header, so as to quickly locate the storage address of the micro-batch to which the time step belongs in the tensor storage area according to the time step position when reading.

[0039] In this way, by organizing image data in micro-batches and encoding the starting frame of each micro-batch as an independent decoding frame, subsequent random access can be performed using each micro-batch as an independent decoding unit without relying on data from other micro-batches. This achieves fast random access to image data at any time step within the tensor storage area while ensuring compression efficiency.

[0040] S204, construct a global trajectory index and associate the global trajectory index with the data file for storage.

[0041] Specifically, after the data file is constructed, a global trajectory index can be built. This global trajectory index is used to quickly locate the physical storage location of each training trajectory in the data file and the boundary position of each training trajectory during sample querying during the training phase. By traversing each data block in the data file, the storage distribution information of each training trajectory can be obtained, and this information is summarized into the global trajectory index. The global trajectory index is then associated with and stored in the data file.

[0042] Here, the global trajectory index records the data block identifier, the starting line offset within the data block, and the trajectory length corresponding to each training trajectory, and uses these as physical address information. The data block identifier indicates which data block in the data file the training trajectory is stored in, the starting line offset indicates the starting line position of the training trajectory within that data block, and the trajectory length indicates the total number of time steps contained in the training trajectory starting from the starting line. The global trajectory index also marks the start and end boundary positions of each training trajectory, and uses these as boundary position information. The global trajectory index is stored in association with the data file, specifically in a preset storage area of ​​the data file or in an index file independent of the data file. When the global trajectory index is stored in the preset storage area of ​​the data file, this preset storage area can be configured to support a copy-on-write mechanism for updates to avoid modifying appended data blocks and maintain the immutability of the main body of the data file.

[0043] In this way, by constructing a global trajectory index and storing it in association with the data file, when loading data during the training phase, the global trajectory index can be loaded into memory first, and then the specific data blocks in the data file can be read as needed according to the physical address information in the index, thus realizing the rapid positioning and loading of training samples.

[0044] In some possible embodiments, the data file can also be constructed using an append-only write method. During data acquisition, the acquired data continuously accumulates in memory. When the amount of data accumulated in memory reaches a preset data block size (which can be set to 256 megabytes), the data block is written to the end of the data file all at once. Simultaneously with appending the data block to the end of the data file, index records of each training trajectory contained in the data block can be appended to the global trajectory index. Because the main body of the data file uses an append-only write method, newly acquired data is sequentially appended to the end of the data file in blocks. The data file does not need to reserve storage space or modify the already written data. During large-scale data expansion, only append-only write operations are performed, and no storage fragmentation occurs within the data file. To reduce the risk of data loss due to abnormal interruptions during memory buffering, a low-water mark flushing strategy can also be configured. For example, it can be set to force the data cached in memory to be written to the data file every 5 seconds or every 1000 frames. In practical applications, this can be configured according to the balance requirements of throughput and data persistence.

[0045] In some possible embodiments, since there are also scenarios where multiple processes concurrently read training data, in which multiple data loading processes need to read different samples from the same data file at the same time, when the data loading request includes multiple data loading processes, a global trajectory index can be provided for each data loading process, so that each data loading process can independently determine the distribution of the samples it is responsible for reading in the data file based on the global trajectory index.

[0046] Furthermore, based on the physical address information of each training trajectory in the global trajectory index—which records the data block identifier of each training trajectory in the data file, the starting line offset within the data block, and the trajectory length—sample subsets can be allocated to each data loading process according to the number of samples. Each data loading process is assigned to one sample subset, and each sample subset contains the same number of samples. Here, the number of samples refers to the total number of training samples contained in the data file, which is the basis for task partitioning in the data loading process to ensure load balancing among the data loading processes.

[0047] For example, taking the concurrent reading of training samples from a data file by four data loading processes as an example, the total number of samples can be set to 10,000. The 10,000 training samples are divided into four sample subsets according to the sample arrangement order. The first sample subset contains the 1st to the 2500th training samples, the second sample subset contains the 2501st to the 5000th training samples, the third sample subset contains the 5001st to the 7500th training samples, and the fourth sample subset contains the 7501st to the 10000th training samples. The four sample subsets are assigned to the four data loading processes respectively, and each data loading process reads the corresponding training sample data from the data file according to its assigned sample subset.

[0048] Specifically, the global trajectory index provides precise location information for each training sample within the data file. Therefore, after allocating sample subsets, the location of the corresponding data block within the data file can be determined based on the subset each data loading process is responsible for. Then, memory mapping is used to map the corresponding data block from the data file to the virtual address space of each data loading process. Each data loading process reads its assigned sample data from its own virtual address space, achieving lock-free concurrent reading of the target data block. Memory mapping is a technique that maps file content to the process's virtual address space. This allows the data loading process to directly access file data via memory addresses, eliminating the need for data copying through standard file read / write interfaces.

[0049] In this way, by using sample-level load partitioning and memory mapping for concurrent reading, load imbalance caused by uneven data density of different data blocks can be avoided in multi-process concurrent scenarios. At the same time, the file locking mechanism required when multiple processes read the same file is eliminated, thus improving the concurrent efficiency of data loading.

[0050] In some possible embodiments, for cross-platform compatibility, the metadata description information of the data file can be serialized using a structured data format defined by an interface description language, supporting parsing of the data file by multiple programming languages. The underlying storage of the data file is in standard binary format, supporting storage on local file systems as well as distributed file systems compatible with portable operating system interfaces. For object storage systems, partial data from the data file is downloaded on demand through a local caching layer, or data is retrieved from the object storage system using a block prefetching strategy to avoid the high-latency random access problem caused by directly performing memory mapping operations on remote storage.

[0051] S102, in response to a read request for a target training sample, the global trajectory index is queried based on the target training trajectory to which the target training sample belongs, to obtain the physical address information of the target training trajectory.

[0052] Here, a target training sample refers to a basic data unit that needs to be read from the data file during training. This basic data unit corresponds to a complete data record at a time step, including metadata and / or image data for that time step. A read request for a target training sample refers to a single-sample read instruction initiated by the training program when constructing training batches. This read instruction is generated by the data loader during iterative training and can specifically include the sample index number of the target training sample, the training trajectory identifier to which the target training sample belongs, and the time step position of the target training sample within its respective training trajectory.

[0053] Specifically, in response to a read request for a target training sample, the global trajectory index can be queried based on the trajectory identifier of the target training trajectory to which the target training sample belongs, carried in the read request. The index entry corresponding to the trajectory identifier of the target training trajectory is then located within the global trajectory index. Further, the data block identifier, the starting row offset within the data block, and the trajectory length of the target training trajectory can be extracted from this index entry. This is used as the physical address information of the target training trajectory, which can be used to locate the target data block where the target training sample is located in subsequent steps.

[0054] S103, based on the physical address information, locate the target data block where the target training sample is located in the data file.

[0055] Furthermore, after obtaining the physical address information of the target training trajectory, the data block indicated by the data block identifier in the physical address information can be located in the data file, and this data block can be identified as the target data block containing the target training sample. The data block identifier is a component of the physical address information and is used to uniquely indicate a data block among multiple data blocks in the data file.

[0056] Here, the target data block is the physical storage unit that carries the target training samples. This target data block is divided into columnar storage areas and tensor storage areas. The columnar storage area stores the metadata of the time step where the target training sample is located, while the tensor storage area stores the image data of the time step where the target training sample is located. The image data within the tensor storage area is organized in micro-batches. The image data of the target training sample belongs to one of these micro-batches. The starting frame of this micro-batch is an independent decoding frame, used as a reference frame within that micro-batch when decoding the image data of the target training sample.

[0057] S104, read the offset index of the header of the target data block, and query the offset index according to the time step position of the target training sample to determine the storage area type of the target data of the target training sample in the target data block.

[0058] Specifically, the offset index refers to the index structure stored in the header of the target data block. This index structure records the distribution information of each storage area within the target data block and the positional mapping relationship of each data unit within the storage area. By reading the offset index in the header of the target data block, the boundary information between the columnar storage area and the tensor storage area within the data block can be obtained, as well as the starting storage address of each micro-batch within the tensor storage area and the corresponding time step range of each micro-batch. By querying the offset index based on the time step position of the target training sample, the storage area type of the target data corresponding to that time step position within the target data block can be determined, i.e., whether the target data is stored in the columnar storage area or the tensor storage area.

[0059] Here, storage region types are represented as columnar storage regions and tensor storage regions. These are classification information indicating the storage location of target data within the target data block. Columnar storage regions are used to store metadata, while tensor storage regions are used to store image data. Depending on the actual reading requirements, a target training sample may involve one or both columnar and tensor storage regions. When the training task only requires metadata, the target data only involves columnar storage regions; when the training task only requires image data, the target data only involves tensor storage regions; when the training task requires both metadata and image data, the target data involves both storage region types.

[0060] S105, if the storage area type is the columnar storage area, read the target metadata of the target training sample from the columnar storage area of ​​the target data block; if the storage area type is the tensor storage area, obtain the storage location of the target training sample in the tensor storage area according to the offset index, and read the target image data of the target training sample from the tensor storage area.

[0061] Here, after determining the storage area type, the corresponding read operation can be performed according to the storage area type to read the data required for the target training sample from the corresponding storage area.

[0062] Specifically, if the storage area type is a columnar storage area, it means that the target data of the target training sample is metadata. In this case, the target metadata of the target training sample can be read from the columnar storage area of ​​the target data block. When reading metadata, since metadata of the same type is arranged continuously in each time step within the columnar storage area, it is only necessary to read the metadata value of that time step from the corresponding data column according to the time step position of the target training sample.

[0063] Specifically, if the storage area type is tensor storage area, it means that the target data of the target training sample is image data. At this time, the storage location of the target training sample in the tensor storage area can be obtained according to the offset index, and the target image data of the target training sample can be read from the tensor storage area.

[0064] In some possible embodiments, since the image data stored in the tensor storage region is in a compressed encoding format, a decoding operation can be performed to recover the original image from the compressed data, referring to... Figure 3 As shown, when reading target image data of target training samples from the tensor storage area, the following steps S301~S303 may be included: S301, Based on the offset index, determine the starting storage address of the target micro-batch to which the time step position of the target training sample belongs in the tensor storage area.

[0065] Here, the time step position of the target training sample refers to the time step number of the target training sample within its respective training trajectory. This time step number can be used to locate the target micro-batch to which the image data of the target training sample belongs in the offset index. Based on the start and end time steps corresponding to each micro-batch recorded in the offset index, the target micro-batch to which the time step position of the target training sample belongs can be determined. Then, the starting storage address of the target micro-batch in the tensor storage area is read from the offset index.

[0066] S302, based on the starting storage address of the target micro-batch and the frame offset of the target training sample within the target micro-batch, locate the storage position of the compressed image data of the target training sample in the tensor storage area.

[0067] Furthermore, after obtaining the starting storage address of the target micro-batch, the storage location of the compressed image data of the target training sample in the tensor storage area can be determined based on the starting storage address of the target micro-batch and the frame offset of the target training sample within the target micro-batch. The starting storage address of the target micro-batch indicates the offset position of the first byte of the target micro-batch within the data block, and the frame offset indicates the relative position of the image frame of the target training sample within the micro-batch. Combining these two factors allows the calculation of the specific storage location of the compressed image data of the target training sample in the tensor storage area.

[0068] S303, read the compressed image data of the target training sample from the storage location, call the decoder to perform inter-frame prediction decoding starting from the starting frame of the target micro-batch, and obtain the target image data of the target training sample.

[0069] Specifically, after locating the storage location of the compressed image data, the compressed image data stored at that location is read. Since the compressed image data is in an encoded format, it cannot be directly used as input for model training. At this point, the decoder can be invoked to perform inter-frame prediction decoding starting from the beginning frame of the target micro-batch, reconstructing the compressed image data into the original image data. During the decoding process, the decoder only refers to the decoded frames within the target micro-batch and does not refer to frames from other micro-batches outside the target micro-batch.

[0070] Here, the decoder is a program module or hardware unit used to decode and reconstruct compressed image data into original image data. The decoder supports decoding operations of intra-coded frames and inter-predictive coded frames, and can recover all image frames in a micro-batch frame by frame with intra-coded frames as reference frames.

[0071] In some possible embodiments, if the same target training sample requires both metadata and image data, the steps of reading target metadata from the columnar storage area and reading target image data from the tensor storage area can be performed separately, and the read target metadata and target image data can be combined into the complete data of the target training sample.

[0072] S106, Based on the boundary position information in the global trajectory index, determine whether the target training sample involves crossing the trajectory boundary, and generate the corresponding boundary mask information.

[0073] Here, in the sliding window sampling scenario, the target training sample corresponds to a sampling window containing multiple consecutive time steps. The length of this sampling window is determined by the number of observation steps and prediction steps in the training task. Considering that the sliding window sampling process may cross the boundary of the training trajectory, causing frames from different trajectories to be mixed into the same training sample, thus disrupting the causal consistency of the training samples, in order to identify the validity of each position in the sampling window during training, we can determine whether the sampling window of the target training sample crosses the end boundary of the target training trajectory based on the boundary position information of the target training trajectory in the global trajectory index. In the case of crossing the boundary, the part of the sampling window that crosses the boundary is marked, and then the corresponding boundary mask information is generated based on the judgment result. This boundary mask information is provided to the model training process along with the data of the target training sample, which can instruct the model to ignore the data in the part that crosses the boundary when calculating the loss.

[0074] For example, generating boundary mask information may specifically include the following steps (a) to (c): (a) Based on the time step position of the target training sample and the boundary position information of the target training trajectory in the global trajectory index, determine whether the sampling window of the target training sample crosses the end boundary of the target training trajectory; (b) If the sampling window crosses the end boundary, a boundary mask with a first identifier is generated in the crossed portion of the sampling window; (c) If the sampling window does not cross the end boundary, a boundary mask with a second identifier is generated.

[0075] Understandably, the starting position of the sampling window can be determined based on the time step position of the target training sample, and the ending position of the target training trajectory can be determined based on the boundary position information of the target training trajectory. By comparing whether the ending position of the sampling window exceeds the ending position of the target training trajectory, it can be determined whether the sampling window crosses the ending boundary of the target training trajectory. If the ending position of the sampling window exceeds the ending position of the target training trajectory, it means that the sampling window has crossed the trajectory boundary. In this case, a boundary mask with a first identifier can be generated for the crossed portion of the sampling window. The first identifier is used to indicate that the data corresponding to the crossed portion is ignored in model training, preventing the model from learning non-existent causal relationships between trajectories. If the ending position of the sampling window does not exceed the ending position of the target training trajectory, it means that the sampling window has not crossed the trajectory boundary. In this case, a boundary mask with a second identifier is generated. The second identifier is used to indicate that the data corresponding to the sampling window is retained in model training.

[0076] In this way, by determining whether the sampling window crosses the trajectory boundary through the boundary position information and generating the corresponding boundary mask, the validity of each position in the training sample can be correctly marked, thus improving the causal consistency of the training sample.

[0077] S107, Based on the data of the target training sample read from the target data block and the boundary mask information, assemble the target training sample and return it.

[0078] Here, the data of the target training samples read from the target data block provides the training input content for the target training samples. This training input content includes at least one of target metadata and target image data. The boundary mask information provides validity markers for each location in the target training samples, indicating the degree of attention the model pays to each location during training. After completing the reading of the target training sample data and the generation of the boundary mask information, the target training sample data and the boundary mask information are assembled together into a complete data package of the target training samples and returned to the initiator of the data loading request for batch data stitching and loss calculation during model training.

[0079] Here, when the target training sample data contains only target metadata or only target image data, this single type of data can be assembled with boundary mask information into a target training sample and returned; when the target training sample data contains both target metadata and target image data, both can be assembled with boundary mask information into a target training sample and returned.

[0080] In some possible implementations, different sensors may collect data at different frequencies during the data acquisition process. For example, an image sensor might collect image data at 30 Hz, while a joint status sensor might collect joint status data at 500 Hz. In such cases, timestamp alignment is required between data streams of different frequencies at the same time step to ensure temporal matching of the multimodal data in the training samples. Therefore, a local time alignment table mechanism can be introduced to solidify the timestamp correspondence between data streams of different frequencies in the header of the data block, thus solving the preprocessing problem of timestamp synchronization and resampling of data streams of different frequencies during the training phase. Specifically, a local time alignment table can be set in the target data block. This local time alignment table records the timestamp correspondence between data streams of different frequencies within the target data block. Through this local time alignment table, fast matching between data streams of different frequencies can be achieved within the data block without the need for additional offline alignment preprocessing during the training phase.

[0081] At this point, after locating the target data block containing the target training sample in the data file based on the physical address information, when the target training sample involves reading data from the same time step from both the first and second frequency data streams, a local time alignment table can be used for aligned reading of multi-frequency data, as described above. Figure 4 As shown, the steps S401~S403 may be included: S401, Read the local time alignment table in the header of the target data block, and according to the target timestamp of the target training sample, query the local time alignment table for the first storage location of the first frequency data stream and the second storage location of the second frequency data stream that match the target timestamp.

[0082] Here, the local time alignment table of the target data block header can be read, and according to the target timestamp of the target training sample, the storage location of the nearest neighbor timestamp in the first frequency data stream and the storage location of the nearest neighbor timestamp in the second frequency data stream corresponding to the target timestamp can be matched in the local time alignment table to obtain the first storage location and the second storage location.

[0083] S402, read the first data of the target training sample under the first frequency data stream from the columnar storage area and / or the tensor storage area according to the first storage location, and read the second data of the target training sample under the second frequency data stream from the columnar storage area and / or the tensor storage area according to the second storage location.

[0084] Furthermore, based on the first storage location, it can be determined whether the target data of the first frequency data stream belongs to a columnar storage area or a tensor storage area in the target data block. Then, the first data is read according to the reading method corresponding to that storage area. For details, refer to the reading process in step S105 above. Similarly, based on the second storage location, it can be determined whether the target data of the second frequency data stream belongs to a columnar storage area or a tensor storage area in the target data block. The second data is then read according to the corresponding reading method, thereby obtaining the data corresponding to the two frequency data streams at the same time step.

[0085] S403, the first data and the second data are aligned and assembled into multimodal training data at the same time step, which is used as the data of the target training sample.

[0086] Here, after obtaining the first data and the second data, the first data and the second data can be aligned and assembled according to the same target time step. The assembly result is used as the multimodal training data of the target training sample. The subsequent steps assemble the multimodal training data and boundary mask information together into the target training sample and return it.

[0087] In this way, by recording the timestamp correspondence of different frequency data streams through a local time alignment table, the storage location of each frequency data can be obtained by directly querying the table during reading. This avoids the preprocessing overhead of offline timestamp synchronization and resampling of different frequency data streams during the training phase, and improves the data loading efficiency in multi-frequency data scenarios.

[0088] In some other embodiments, if there is a need for alignment of more frequency data streams, the timestamp correspondence of more frequency data streams can be recorded through a local time alignment table for adaptive expansion to accommodate the alignment needs of more sensor data streams. No specific limitation is made here.

[0089] The embodied intelligence training data reading method, apparatus, medium, and device provided in this disclosure construct a global trajectory index that records the physical address and boundary position of each training trajectory. During data loading, only this index needs to be loaded to respond to sample reading requests, avoiding traversing the entire data file and reducing reading latency. Furthermore, by dividing the data file into columnar storage areas and tensor storage areas to store metadata and image data organized in micro-batches with independent decoding frames as the starting frame, respectively, and using a data block header offset index to locate the storage area type and tensor storage position, separate access to metadata and image data and independent decoding and reading of image data are achieved. This reduces the amount of irrelevant data loaded and improves the positioning accuracy and reading efficiency of target samples. Simultaneously, a boundary mask is generated based on the boundary position information in the global trajectory index. When assembling samples, the handling method across trajectory boundaries is clearly defined, ensuring the temporal continuity of training samples and accurate distinction of trajectory boundaries, avoiding data confusion during cross-trajectory sample assembly. Therefore, while ensuring data integrity and correctness, the overall reading speed and system resource utilization of embodied intelligence training data are improved.

[0090] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0091] Based on the same inventive concept, this disclosure also provides a device for reading embodied intelligent training data corresponding to the method for reading embodied intelligent training data. Since the principle of the device in this disclosure for solving the problem is similar to the above-mentioned method for reading embodied intelligent training data in this disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0092] Reference Figure 5 The diagram shown is a schematic of a device 500 for reading embodied intelligent training data according to an embodiment of this disclosure. The device includes: The index loading module 501 is used to load a global trajectory index in response to a data loading request. The global trajectory index records the physical address information of each training trajectory in the data file and the boundary position information of each training trajectory. Address determination module 502 is used to respond to a read request for a target training sample by querying the global trajectory index according to the target training trajectory to which the target training sample belongs, and obtaining the physical address information of the target training trajectory. The data block determination module 503 is used to locate the target data block where the target training sample is located in the data file according to the physical address information; wherein, the target data block includes a columnar storage area and a tensor storage area, the columnar storage area is used to store metadata corresponding to the time step, and the tensor storage area is used to store image data organized in micro-batches, and the starting frame of each micro-batch is an independent decoding frame; The region determination module 504 is used to read the offset index of the header of the target data block and query the offset index according to the time step position of the target training sample to determine the storage region type of the target data of the target training sample in the target data block. The data reading module 505 is used to read the target metadata of the target training sample from the columnar storage area of ​​the target data block if the storage area type is the columnar storage area; and to obtain the storage location of the target training sample in the tensor storage area according to the offset index if the storage area type is the tensor storage area, and to read the target image data of the target training sample from the tensor storage area. The boundary judgment module 506 is used to determine whether the target training sample involves crossing the trajectory boundary based on the boundary position information in the global trajectory index, and generate corresponding boundary mask information. The data assembly module 507 is used to assemble the target training sample based on the data of the target training sample read from the target data block and the boundary mask information, and then return the data.

[0093] In some possible embodiments, the index loading module 501 is further configured to: Acquire data from multiple training trajectories; wherein the data from each training trajectory includes a metadata sequence and an image data sequence arranged by time step; The collected data of each training trajectory is divided into multiple data blocks according to a preset data block size to generate a data file; wherein, the data file includes multiple data blocks corresponding to each training trajectory; For each data block, the metadata of each time step within the data block is compressed column-wise and written into the columnar storage area of ​​the data block, and the image data of each time step within the data block is organized into micro-batches and written into the tensor storage area of ​​the data block. The starting frame of each micro-batch is encoded into an independent decoding frame. A global trajectory index is constructed and stored in association with the data file; wherein, the global trajectory index records the data block identifier, the starting line offset and trajectory length corresponding to each training trajectory as the physical address information, and marks the starting boundary position and ending boundary position of each training trajectory as the boundary position information.

[0094] In some possible embodiments, the index loading module 501 is further configured to: Acquire image data from multiple consecutive time steps within the data block; According to a preset micro-batch length, the image data of the multiple time steps are divided into at least one micro-batch, and each micro-batch includes consecutive image frames with a number equal to the preset micro-batch length. For each micro-batch, the starting frame of the micro-batch is encoded as an intra-coded frame, and the remaining frames in the micro-batch, excluding the starting frame, are encoded as inter-predictive coded frames. The encoded micro-batches are written sequentially into the tensor storage area according to the time step order, and the starting storage address and the preset micro-batch length of each micro-batch are recorded in the offset index of the data block header.

[0095] In some possible embodiments, the data reading module 505 is specifically used for: Based on the offset index, determine the starting storage address of the target micro-batch to which the time step position of the target training sample belongs in the tensor storage area; Based on the starting storage address of the target micro-batch and the frame offset of the target training sample within the target micro-batch, locate the storage position of the compressed image data of the target training sample in the tensor storage area. The compressed image data of the target training sample is read from the storage location, and the decoder is called to perform inter-frame prediction decoding starting from the starting frame of the target micro-batch to obtain the target image data of the target training sample; wherein, during the decoding process, only the decoded frames within the target micro-batch are referenced, and frames of other micro-batches outside the target micro-batch are not referenced.

[0096] In some possible embodiments, the boundary determination module 506 is specifically used for: Based on the time step position of the target training sample and the boundary position information of the target training trajectory in the global trajectory index, determine whether the sampling window of the target training sample crosses the end boundary of the target training trajectory; If the sampling window crosses the end boundary, a boundary mask with a first identifier is generated at the crossed portion of the sampling window; wherein the first identifier is used to indicate that the data corresponding to the crossed portion is ignored in model training; If the sampling window does not cross the end boundary, a boundary mask with a second identifier is generated; wherein the second identifier is used to indicate that the data corresponding to the sampling window is retained in model training.

[0097] In some possible embodiments, the index loading module 501 is further configured to: In the case where the data loading request includes multiple data loading processes, the global trajectory index is provided for each of the data loading processes. Based on the physical address information of each training trajectory in the global trajectory index, a subset of samples is allocated to each data loading process according to the number of samples. Based on the sample subsets each data loading process is responsible for, the corresponding data blocks in the data file are mapped to the virtual address space of each data loading process through memory mapping, so as to achieve concurrent reading without file locks when reading the target data block.

[0098] In some possible embodiments, the target data block further includes a local time alignment table, which records the timestamp correspondence between data streams of different frequencies within the target data block; When the target training samples involve reading data from the same time step from a first frequency data stream and a second frequency data stream, the data block determination module 503 is further configured to: Read the local time alignment table in the header of the target data block, and according to the target timestamp of the target training sample, query the first storage location of the first frequency data stream and the second storage location of the second frequency data stream that match the target timestamp in the local time alignment table; The first data of the target training sample under the first frequency data stream is read from the columnar storage area and / or the tensor storage area according to the first storage location, and the second data of the target training sample under the second frequency data stream is read from the columnar storage area and / or the tensor storage area according to the second storage location; The first data and the second data are aligned and assembled into multimodal training data at the same time step, which is used as the data for the target training sample.

[0099] Based on the same technical concept, this disclosure also provides a computer device. (See also...) Figure 6The diagram shows the structure of a computer device 600 provided in this embodiment of the present disclosure, including a processor 601, a memory 602, and a bus 603. The memory 602 stores execution instructions and includes a main memory 6021 and an external memory 6022. The main memory 6021, also called internal memory, is used to temporarily store computational data in the processor 601 and data exchanged with external memory 6022 such as a hard disk. The processor 601 exchanges data with the external memory 6022 through the main memory 6021.

[0100] In this embodiment, the memory 602 is specifically used to store application code that executes the solution of this application, and its execution is controlled by the processor 601. That is, when the computer device 600 is running, the processor 601 communicates with the memory 602 through the bus 603, so that the processor 601 executes the application code stored in the memory 602, and then executes the method described in any of the foregoing embodiments.

[0101] The memory 602 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0102] Processor 601 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0103] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 600. In other embodiments of this application, the computer device 600 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0104] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the embodied intelligence training data reading method described in the above-described method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0105] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the embodied intelligence training data reading method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0106] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0107] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0110] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0111] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit them. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the scope of the technology disclosed in this disclosure, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure.

Claims

1. A method for reading embodied intelligence training data, characterized in that, include: In response to a data loading request, a global trajectory index is loaded. The global trajectory index records the physical address information of each training trajectory in the data file and the boundary position information of each training trajectory. In response to a read request for a target training sample, the global trajectory index is queried based on the target training trajectory to which the target training sample belongs, to obtain the physical address information of the target training trajectory; Based on the physical address information, the target data block where the target training sample is located is located in the data file; wherein, the target data block includes a columnar storage area and a tensor storage area, the columnar storage area is used to store metadata corresponding to the time step, and the tensor storage area is used to store image data organized in micro-batches, with the starting frame of each micro-batch being an independent decoding frame; Read the offset index of the header of the target data block, and query the offset index according to the time step position of the target training sample to determine the storage area type of the target data of the target training sample in the target data block; If the storage area type is the columnar storage area, the target metadata of the target training sample is read from the columnar storage area of ​​the target data block; if the storage area type is the tensor storage area, the storage location of the target training sample in the tensor storage area is obtained according to the offset index, and the target image data of the target training sample is read from the tensor storage area. Based on the boundary position information in the global trajectory index, determine whether the target training sample involves crossing trajectory boundaries and generate corresponding boundary mask information; Based on the data of the target training sample read from the target data block and the boundary mask information, the target training sample is assembled and returned.

2. The method according to claim 1, characterized in that, Before loading the global trajectory index in response to the data loading request, the following steps are included: Acquire data from multiple training trajectories; wherein the data from each training trajectory includes a metadata sequence and an image data sequence arranged by time step; The collected data of each training trajectory is divided into multiple data blocks according to a preset data block size to generate a data file; wherein, the data file includes multiple data blocks corresponding to each training trajectory; For each data block, the metadata of each time step within the data block is compressed column-wise and written into the columnar storage area of ​​the data block, and the image data of each time step within the data block is organized into micro-batches and written into the tensor storage area of ​​the data block. The starting frame of each micro-batch is encoded into an independent decoding frame. A global trajectory index is constructed and stored in association with the data file; wherein, the global trajectory index records the data block identifier, the starting line offset and trajectory length corresponding to each training trajectory as the physical address information, and marks the starting boundary position and ending boundary position of each training trajectory as the boundary position information.

3. The method according to claim 2, characterized in that, The step of organizing the image data at each time step within the data block into micro-batches and writing it into the tensor storage area of ​​the data block includes: Acquire image data from multiple consecutive time steps within the data block; According to a preset micro-batch length, the image data of the multiple time steps are divided into at least one micro-batch, and each micro-batch includes consecutive image frames with a number equal to the preset micro-batch length. For each micro-batch, the starting frame of the micro-batch is encoded as an intra-coded frame, and the remaining frames in the micro-batch, excluding the starting frame, are encoded as inter-predictive coded frames. The encoded micro-batches are written sequentially into the tensor storage area according to the time step order, and the starting storage address and the preset micro-batch length of each micro-batch are recorded in the offset index of the data block header.

4. The method according to claim 3, characterized in that, The step of obtaining the storage location of the target training sample in the tensor storage region according to the offset index, and reading the target image data of the target training sample from the tensor storage region includes: Based on the offset index, determine the starting storage address of the target micro-batch to which the time step position of the target training sample belongs in the tensor storage area; Based on the starting storage address of the target micro-batch and the frame offset of the target training sample within the target micro-batch, locate the storage position of the compressed image data of the target training sample in the tensor storage area. The compressed image data of the target training sample is read from the storage location, and the decoder is called to perform inter-frame prediction decoding starting from the starting frame of the target micro-batch to obtain the target image data of the target training sample; wherein, during the decoding process, only the decoded frames within the target micro-batch are referenced, and frames of other micro-batches outside the target micro-batch are not referenced.

5. The method according to claim 1, characterized in that, The step of determining whether the target training sample involves crossing trajectory boundaries based on the boundary position information in the global trajectory index, and generating corresponding boundary mask information, includes: Based on the time step position of the target training sample and the boundary position information of the target training trajectory in the global trajectory index, determine whether the sampling window of the target training sample crosses the end boundary of the target training trajectory; If the sampling window crosses the end boundary, a boundary mask with a first identifier is generated at the crossed portion of the sampling window; wherein the first identifier is used to indicate that the data corresponding to the crossed portion is ignored in model training; If the sampling window does not cross the end boundary, a boundary mask with a second identifier is generated; wherein the second identifier is used to indicate that the data corresponding to the sampling window is retained in model training.

6. The method according to claim 1, characterized in that, After loading the global trajectory index in response to the data loading request, the process includes: In the case where the data loading request includes multiple data loading processes, the global trajectory index is provided for each of the data loading processes. Based on the physical address information of each training trajectory in the global trajectory index, a subset of samples is allocated to each data loading process according to the number of samples. Based on the sample subsets each data loading process is responsible for, the corresponding data blocks in the data file are mapped to the virtual address space of each data loading process through memory mapping, so as to achieve concurrent reading without file locks when reading the target data block.

7. The method according to claim 1, characterized in that, The target data block also includes a local time alignment table, which records the timestamp correspondence between data streams of different frequencies within the target data block. After locating the target data block where the target training sample is located in the data file based on the physical address information, when the target training sample involves reading data from the same time step from the first frequency data stream and the second frequency data stream, the method further includes: Read the local time alignment table in the header of the target data block, and according to the target timestamp of the target training sample, query the first storage location of the first frequency data stream and the second storage location of the second frequency data stream that match the target timestamp in the local time alignment table; The first data of the target training sample under the first frequency data stream is read from the columnar storage area and / or the tensor storage area according to the first storage location, and the second data of the target training sample under the second frequency data stream is read from the columnar storage area and / or the tensor storage area according to the second storage location; The first data and the second data are aligned and assembled into multimodal training data at the same time step, which is used as the data for the target training sample.

8. A device for reading embodied intelligent training data, characterized in that, include: The index loading module is used to load the global trajectory index in response to the data loading request. The global trajectory index records the physical address information of each training trajectory in the data file and the boundary position information of each training trajectory. The address determination module is used to respond to a read request for a target training sample by querying the global trajectory index based on the target training trajectory to which the target training sample belongs, and obtaining the physical address information of the target training trajectory. A data block determination module is used to locate the target data block where the target training sample is located in the data file according to the physical address information; wherein, the target data block includes a columnar storage area and a tensor storage area, the columnar storage area is used to store metadata corresponding to the time step, and the tensor storage area is used to store image data organized in micro-batches, and the starting frame of each micro-batch is an independent decoding frame; The region determination module is used to read the offset index of the header of the target data block, and query the offset index according to the time step position of the target training sample to determine the storage region type of the target data of the target training sample in the target data block; The data reading module is used to read the target metadata of the target training sample from the columnar storage area of ​​the target data block if the storage area type is the columnar storage area; and to obtain the storage location of the target training sample in the tensor storage area according to the offset index if the storage area type is the tensor storage area, and to read the target image data of the target training sample from the tensor storage area. The boundary determination module is used to determine whether the target training sample involves crossing the trajectory boundary based on the boundary position information in the global trajectory index, and to generate corresponding boundary mask information. The data assembly module is used to assemble the target training sample based on the data of the target training sample read from the target data block and the boundary mask information, and then return the data.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images