Time sequence model training data processing method, training method and electronic device

Abstract

The application discloses a time sequence model training data processing method, a training method and an electronic device, relates to the technical field of machine learning training data preprocessing, and the training data processing method comprises the following steps: acquiring a time sequence frame file sequence output by a simulation rendering pipeline, the sequence containing a placeholder frame inserted by the simulation rendering pipeline when data is missing; comparing the binary features of each frame file in the time sequence frame file sequence with preset binary features frame by frame to determine the placeholder frame; cutting the time sequence frame file sequence into a plurality of subsegments with the placeholder frame as a separation boundary; regarding each subsegment as an independent training unit, splicing the subsegments into a flat continuous numbered frame index sequence in chronological order, and storing the frame index sequence into a training data set; and the binary features comprise the byte length of the frame file and / or the hash value of the frame file content. The application accurately identifies the placeholder frame by using the constant feature of the binary features of the placeholder frame, avoids generating pseudo-adjacent frames due to the elimination of the placeholder frame, and thus improves the model training effect.

Description

Technical Field

[0001] This application relates to the field of machine learning training data preprocessing technology, and in particular to a training data processing method, training method and electronic device for a time series model. Background Technology

[0002] End-to-end navigation models, especially temporal world models based on the Cyclic State-Space Model (RSSM), rely on learning the dynamic evolution of the environment from continuous time-series observations, i.e., predicting the state at the next moment using adjacent frames. Therefore, the temporal accuracy of the training data is fundamental to the effective learning of the model.

[0003] Currently, simulation is one of the mainstream methods for obtaining training data. A typical data pipeline includes: recording the robot's motion trajectory in a simulation environment, then generating multimodal perception data (such as RGB images, depth maps, etc.) through an offline rendering pipeline, followed by format conversion to form a standard dataset (such as the LeRobot format), and finally providing it for model sampling and training. However, in practice, the applicant has discovered two types of "pseudo-adjacent frames" that cause temporal distortion of training data in this pipeline, resulting in hidden contamination of model training.

[0004] Pseudo-adjacent frames are defined as two frames that are not physically adjacent in time, but are considered adjacent in the data structure due to data defects and both enter the same training sequence window. The harm lies in causing the model to learn anti-physics priors such as "teleportation" that do not exist in reality.

[0005] Specifically, pseudo-neighboring frames are typically generated in the following two ways: Method 1: Insert placeholder frames in the rendering pipeline.

[0006] During batch rendering, when the rendering pipeline fails to return data normally, the system silently writes a placeholder frame with a fixed pattern (such as a completely black image) to maintain the integrity of the file sequence. These placeholder frames are difficult to identify through manual sampling or threshold judgment based on image content (such as blackness detection). If the placeholder frame is simply deleted and the remaining frames are directly spliced ​​together, the frames before and after the original placeholder frame will become pseudo-adjacent frames.

[0007] Method 2: After flattening and stitching multiple trajectory segments, the sampling window crosses the boundary.

[0008] To facilitate storage and loading, multiple independent episodes are typically flattened and stitched together into a continuous sequence of frame indices. Existing dataset loading modules may, during sampling, allow the sampling window to cross the stitching boundaries of different episodes, feeding the last frame of the previous episode and the first frame of the next episode as adjacent frames into the training window. This boundary-crossing behavior also generates pseudo-adjacent frames, and current mainstream open-source implementations (such as LeRobot and X-Mobility) do not address this issue.

[0009] Both types of pseudo-neighboring frames inject invalid or discontinuous "motion" information into the temporal world model, causing the model to learn incorrect temporal priors and thus contaminating downstream decision-making strategies. More seriously, this contamination is usually not visible on the training loss curve, and the model will converge to a suboptimal solution, making it difficult to detect through conventional training metrics.

[0010] In summary, existing technologies lack a method to systematically eliminate the two types of pseudo-adjacent frames mentioned above, so as to ensure the true continuity of training data for time series models. Summary of the Invention

[0011] This application aims to at least partially address one of the technical problems in the related art. To this end, one objective of this application is to propose a training data processing method, training method, and electronic device for a time-series model, to systematically eliminate pseudo-adjacent frames, ensure the true continuity of the training data for the time-series model, and thereby improve the training effect of the time-series model.

[0012] To achieve the above objectives, a first aspect of this application proposes a training data processing method for training time-series models. The method includes: acquiring a sequence of time-series frame files output by a simulation rendering pipeline, wherein the sequence of time-series frame files is generated by continuous rendering by a camera in the simulation rendering pipeline, arranged in chronological order, and includes placeholder frames inserted by the simulation rendering pipeline when data is missing; comparing the binary features of each frame file in the time-series frame file sequence with preset binary features frame by frame, and determining the matching frame files as placeholder frames; using the placeholder frames as dividing boundaries, dividing the time-series frame file sequence into several sub-segments; and concatenating each sub-segment as an independent training unit in chronological order into a flat, consecutively numbered frame index sequence, and storing it in a training dataset; wherein the binary features include the byte length of the frame file and / or the hash value of the frame file content.

[0013] In some embodiments, before storing each of the sub-segments as an independent training unit in the training dataset, the method further includes: discarding sub-segments with a frame count less than a first segment length threshold; wherein the first segment length threshold is not less than the length of the input sequence of the time series model.

[0014] In some embodiments, comparing the binary features of a frame file with preset binary features and determining a matching frame file as a placeholder frame includes: comparing the byte length of the frame file with the byte length in the preset binary features; if the two byte lengths are the same, calculating the hash value of the frame file content; comparing the hash value of the frame file content with the hash value in the preset binary features; if the two hash values ​​are the same, determining that the frame file is a matching frame file.

[0015] In some embodiments, the simulation rendering pipeline is provided with multiple cameras, and the simulation rendering pipeline outputs multiple time-series frame file sequences, which correspond one-to-one with the multiple cameras and are time-aligned; wherein, when any frame file of any camera at a certain moment is determined to be a placeholder frame, the frame files of all cameras at that moment are determined to be placeholder frames.

[0016] In some embodiments, the preset binary features are obtained by performing small-batch trial rendering sampling on the simulation rendering pipeline with a frame count less than a preset threshold, identifying placeholder frames inserted due to data loss from the sampled output frame files, and extracting the binary features of the identified placeholder frames.

[0017] In some embodiments, the method further includes: calculating the proportion of frames determined to be placeholder frames to the total number of frames, using this as the placeholder frame drop rate, and outputting it; and / or calculating the number of segments retained after segmentation, and outputting it.

[0018] In some embodiments, the method further includes: in response to the initialization of the training dataset, obtaining the start and end positions of each training unit in the training dataset in the frame index sequence; for each training unit, determining all starting positions that allow a sampling window of length equal to the length of the input sequence to fall completely within the training unit, thereby obtaining a set of legal sampling start positions corresponding to the training unit; in response to a sampling request for a training sample, selecting a starting position from the set of legal sampling start positions as the window start point, and extracting consecutive input sequence length frames from the frame index sequence starting from the starting point to form a training sample.

[0019] To achieve the above objectives, a second aspect of this application proposes a training method for a time series model, the method comprising: obtaining training samples using a training data processing method for time series model training as described in the first aspect embodiment; and training a preset time series model using the training samples.

[0020] To achieve the above objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the training data processing method for time-series model training as described in the first aspect embodiment, or implements the time-series model training method as described in the second aspect embodiment.

[0021] To achieve the above objectives, a fourth aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. When the computer program is executed by the processor, it implements the training data processing method for time-series model training as described in the first aspect embodiment, or implements the time-series model training method as described in the second aspect embodiment.

[0022] According to the training data processing method, training method, and electronic device of the time series model according to the embodiments of this application, this application utilizes the constant binary features of placeholder frames for accurate identification without decoding images, achieving placeholder frame detection with zero false positives and constant overhead. At the same time, it segments the data into sub-segments with the placeholder frames as boundaries, treating each sub-segment as an independent training unit. Based on the start and end positions of the training unit and the length of the sampling window, it performs effective sampling on the training unit, fundamentally avoiding the problem of pseudo-adjacent frames caused by splicing after removing placeholder frames and the sampling window crossing the boundary. This eliminates the data pollution source for the time series model to learn anti-physical teleportation priors and improves the model training effect.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating the training data processing method for a time-series model according to an embodiment of this application. Figure 2 This is a schematic diagram of the process for constructing training samples according to an embodiment of this application; Figure 3 This is a structural block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0025] The following description, with reference to the accompanying drawings, describes a training data processing method, a training method, and an electronic device for a time-series model according to embodiments of this application.

[0026] Figure 1 This is a flowchart illustrating the training data processing method for time-series model training according to an embodiment of this application.

[0027] In some embodiments of this application, such as Figure 1As shown, the training data processing method for time series model training includes: S1, obtain the sequence of timing frame files output by the simulation rendering pipeline.

[0028] Specifically, the time-series frame file sequence is continuously rendered and generated by one or more cameras in a simulation environment (such as NVIDIA Isaac Sim and its Replicator rendering framework) as the virtual robot moves according to a set strategy. Each frame corresponds to a discrete moment and contains multimodal perception data, such as RGB images, depth maps, or semantic segmentation maps, and is stored in the file system in a conventional image format (such as JPG, PNG). Because the image annotator of the rendering pipeline may intermittently return empty data during batch, long-term rendering, the rendering data writer module does not throw an exception or interrupt the rendering process in this case, but silently writes a placeholder frame (e.g., a completely black JPG image) to maintain the apparent continuity and integrity of the output file sequence in terms of file number and file naming. Therefore, these placeholder frames will inevitably be mixed into the time-series frame file sequence. This application obtains the time-series frame file sequence completely and in its original chronological order, providing accurate and complete input for subsequent precise identification of placeholder frames based on binary features.

[0029] S2 compares the binary features of each frame file in the time-series frame file sequence with the preset binary features, and determines the matching frame file as a placeholder frame.

[0030] It should be noted that, through practical experience, the applicant discovered that the placeholder frames written by the simulation rendering pipeline when data is missing are not random bad frames with content close to black, but rather the result of the same binary template being repeatedly written. In other words, all placeholder frame files have completely constant binary characteristics, including identical file byte lengths and identical file content hash values. Based on this discovery, this application utilizes the binary characteristics of frame files for deterministic identification of placeholder frames, rather than relying on statistical judgments based on image content.

[0031] Specifically, in some embodiments of this application, the binary characteristics include the byte length of the frame file (i.e., the file size, which can be directly read through file attributes without reading the file content) and / or the hash value of the frame file content (e.g., the digest value output by cryptographic hash functions such as MD5, SHA-1, and SHA-256). The preset binary characteristics refer to the constant byte length and / or constant content hash value of the placeholder frames generated by the rendering pipeline.

[0032] More specifically, in some embodiments of this application, the preset binary features are obtained as follows: small-batch trial rendering sampling with fewer than a preset threshold frames is performed on the simulation rendering pipeline. From the sampled output frame files, placeholder frames inserted due to missing data are manually verified and identified (for example, in a composite rendered video, placeholder frames appear as completely black images and can be marked frame by frame). These placeholder frame samples are collected, and their constant file byte length and / or content hash value are statistically analyzed as the placeholder frame signature for this rendering pipeline version. When the version of the simulation rendering pipeline changes, if the placeholder frame template changes, the placeholder frame signature can be updated by performing small-batch trial rendering sampling again. Preferably, the placeholder frame signature is stored in the program configuration file or environment variables in the form of configurable parameters, so that only the configuration needs to be modified when the renderer version changes, without changing the code logic.

[0033] Preferably, in some embodiments of this application, a two-level screening and determination process is adopted for each frame file in the time-series frame file sequence. The first level of determination compares the byte length of the frame file with the byte length in a preset binary feature. If the two byte lengths are the same, the second level of determination is performed: the hash value of the frame file content is calculated and compared with the hash value in the preset binary feature. If the two hash values ​​are also the same, the frame file is determined to be a placeholder frame. If the first-level byte length does not match, it is directly determined to be a valid frame, and the second-level hash calculation is not performed.

[0034] Therefore, this application, through a two-stage screening and judgment process, can not only accurately identify placeholder frames, but also eliminates the need for image decoding, decompression, or pixel-by-pixel statistical analysis. Thus, its time complexity for single-frame judgment is constant. Compared to methods based on image content threshold judgment (such as blackness detection), this application exhibits significant accuracy and speed advantages on large-scale datasets.

[0035] Further preferably, in some embodiments of this application, the simulation rendering pipeline is equipped with multiple cameras (e.g., a robot is equipped with both a front-view camera and a rear-view camera), and the simulation rendering pipeline outputs multiple time-series frame file sequences, each corresponding one-to-one with the multiple cameras, and each camera sequence is strictly aligned in time (i.e., each camera produces one frame at the same time). In this case, to ensure that the data from multiple cameras are strictly aligned in the time dimension and to avoid the loss of modal perception data caused by only some cameras having valid data while others are placeholder frames at a certain time, in some embodiments of this application, when the frame file of any camera at a certain time is determined to be a placeholder frame, the frame files of all cameras at that time are also determined to be placeholder frames.

[0036] To facilitate an accurate understanding of the technical solution of this application, a mathematical formalization of the process for determining the placeholder frame is given below.

[0037] Suppose that the time-series frame file sequence contains several frames in chronological order, and C is the set of cameras in the simulation rendering pipeline. The placeholder frame indicator B(i) of the i-th frame is defined as: if the length of the image file of any camera c in that frame is equal to the length of the placeholder frame S in bytes. * If the frame is a placeholder frame, as shown in equation (1), then the frame is determined to be a placeholder frame. The i-th frame is a valid frame.

[0038]

[0039] in, Let be the image file corresponding to camera c in the i-th frame; size() is the file length in bytes; 1[] is an indicator function, which takes 1 if the condition in parentheses is true, and 0 otherwise.

[0040] when When the initial byte length screening is successful, a two-level determination is made using byte length and content hash value. The placeholder frame indicator is further expressed as equation (2): First, the byte length is compared, then the content hash value is compared. The i-th frame is a placeholder frame.

[0041]

[0042] in, Let be the image file corresponding to camera c in the i-th frame; size() is the file length in bytes, hash() is the hash value of the file content; 1[] is an indicator function, which takes 1 if the condition in parentheses is true, and 0 otherwise.

[0043] S3 uses placeholder frames as dividing boundaries to segment the time frame file sequence into several sub-segments.

[0044] Specifically, after dividing the time-series frame file sequence into several sub-segments using placeholder frames as dividing boundaries, each sub-segment corresponds to a closed interval [a, b] in the original sequence, and its length is... It has the following characteristics: First, all frames within the interval are valid frames, without any placeholder frames, that is, for any... All have Second, all adjacent frames within an interval are physically adjacent in time, and the effective frames within a sub-segment maintain the continuous temporal relationship of the original sequence.

[0045] Therefore, this application uses placeholder frames as dividing boundaries to divide the sequence of time frame files into several sub-segments. Instead of deleting and splicing, the segmentation is used to ensure that adjacent frames within each sub-segment are physically adjacent in time, thus avoiding the problem of pseudo-adjacent frames caused by deleting placeholder frames and splicing.

[0046] S4. Each segment is treated as an independent training unit and concatenated into a flat, continuously numbered frame index sequence according to time order, and stored in the training dataset.

[0047] Specifically, this application will concatenate each segment as an independent training unit into a flat, consecutively numbered frame index sequence according to the time order of each segment in the original sequence, and store it in a standard format training dataset (such as the LeRobot format dataset) to facilitate the storage and loading of the training dataset.

[0048] More specifically, this application treats each sub-segment as an independent training unit. This is because after the original sequence is interrupted by the placeholder frame, the sequence segments on either side of the placeholder frame are no longer physically sequential. Therefore, treating them as independent training units aligns with physical laws. In subsequent sampling steps, these independent training units are protected by their respective boundaries, preventing training sample sampling at their intersections. This fundamentally avoids the problem of two frames before and after the placeholder frame becoming pseudo-adjacent frames after deletion.

[0049] Preferably, in some embodiments of this application, before storing each sub-segment as an independent training unit in the training dataset, the training data processing method for training the time series model further includes: discarding sub-segments with a frame count less than a first segment length threshold, wherein the first segment length threshold is not less than the length of the input sequence of the time series model.

[0050] Specifically, during the training process of the downstream time series model, in order to ensure that the sampled training samples do not contain pseudo-adjacent frames, the sampling window can only sample on a single training unit. However, if the number of frames in a segment is too small, it is impossible to sample enough continuous training samples in that segment, and the segment is equivalent to invalid data, so the segment can be discarded.

[0051] More specifically, for a downstream time series model with an input sequence length of T, a frame number is... The number of training samples that a sub-segment can produce ,in, The maximum value function is defined as the number of sub-frames. When the length of the input sequence is greater than or equal to the length T, at least one valid training sample can be produced.

[0052] Furthermore, in some embodiments of this application, the training data processing method for training time-series models further includes: calculating the proportion of frames determined to be placeholder frames to the total number of frames, using this proportion as the placeholder frame discard rate, and outputting it; and / or calculating the number of segments retained after segmentation, and outputting it.

[0053] Specifically, after each segment of the original trajectory is divided into several retained sub-segments, the occupant frame drop rate can be calculated using the following formula:

[0054] in, Here, 'e' represents the placeholder frame drop rate, 'e' represents the trajectory index, and 'k' represents the retained segment index. Let be the frame number of the k-th retained sub-segment of the e-th trajectory. Let be the frame number of the e-th segment of the trajectory.

[0055] More specifically, a higher placeholder frame drop rate indicates more severe data loss in the rendering pipeline. The statistical output of this metric helps users assess the health of the rendering pipeline and decide whether to investigate rendering anomalies. The statistical output of the number of retained segments helps users determine the degree of fragmentation in the dataset and whether rendering parameters need to be adjusted to reduce the frequency of placeholder frames.

[0056] Furthermore, in some embodiments of this application, such as Figure 2 As shown, the training data processing method for time series model training also includes: S5, in response to the initialization of the training dataset, obtains the start and end positions of each training unit in the training dataset in the frame index sequence.

[0057] Specifically, in this embodiment of the application, the training dataset, in addition to storing the flattened frame index sequence, also stores a trajectory boundary index to record the start and end positions of each training unit in the flattened frame index. For example, for the e-th training unit, its trajectory boundary index is stored as a tuple. This indicates that it occupies a range in the flat frame index. ,in, The starting frame number, The end frame number indicates the number of frames contained in this training unit. .

[0058] S6. For each training unit, determine the starting positions of all sampling windows with a length equal to the length of the input sequence that fall completely within the training unit, and obtain the set of legal sampling starting positions corresponding to the training unit.

[0059] Specifically, for the e-th training unit, considering that a sampling window of length T will cover T frames, the starting position s of the window must satisfy the following: after taking T consecutive frames starting from s, the last frame of the window (numbered as...) It must still be within the training unit, i.e., satisfy... , equivalent to Therefore, for the e-th training unit, its valid sampling start position is: Perform the above operation on all M training units respectively, and gather all valid starting positions into a total set. .

[0060] Preferably, the set of legal sampling starting positions S can be constructed and cached in memory at once during the initialization of the training dataset. Subsequently, each sampling only needs to look up the window starting point from the cached list in constant time, thereby effectively reducing the per-sample computation overhead during the training process.

[0061] S7, in response to the sampling request of the training sample, select a starting position from the set of legal sampling starting positions as the starting point of the window, and extract a continuous input sequence of length _ frames from the frame index sequence starting from the starting point to form a training sample.

[0062] Specifically, as can be seen from the aforementioned method of constructing the set of legal sampling starting positions, for any legal starting position s∈S in this set S, a sampling window with that position as the starting point and a length of T is formed. Each training sample must fall entirely within a specific training unit and will not cross its boundaries. Therefore, any training sample formed by selecting starting positions from this set contains T frames that all originate from the same training unit, thus eliminating pseudo-adjacent frames that cross boundaries. Optionally, the total number of samples in the training dataset is taken as the number of elements in the set S of valid sampling starting positions, which can be calculated using the following formula:

[0063] in, The total number of samples in the training dataset. The total number of training units. For the index of the training unit, Let e ​​be the number of frames in the training unit of segment e. This represents the length of the sampling window.

[0064] It should be noted that each starting position in set S corresponds to a real, valid frame, and the window is kept within bounds by constraining the starting point. Therefore, this method results in zero loss of valid data.

[0065] In summary, the training data processing method for the temporal model according to the embodiments of this application utilizes the constant binary features of placeholder frames for accurate identification, eliminating the need for image decoding and achieving placeholder frame detection with zero false positives and constant overhead. Simultaneously, it segments the data using placeholder frames as boundaries, treating each segment as an independent training unit. Effective sampling is then performed on the training units based on their start and end positions and the length of the sampling window. This fundamentally avoids the problem of pseudo-adjacent frames arising from splicing after removing placeholder frames and from sampling windows crossing boundaries. This eliminates data pollution sources that hinder the temporal model's learning of anti-physical teleportation priors, thereby improving the model training effect.

[0066] Specifically, the training data processing method for the time series model in this application has the following beneficial effects: First, the system systematically eliminates two types of pseudo-adjacent frames, covering the entire data link. Under the unified understanding of pseudo-adjacent frames, this application eliminates pseudo-adjacent frames caused by the insertion of placeholder frames at the rendering end through placeholder frame identification and segmentation, and eliminates pseudo-adjacent frames caused by the sampling window boundary crossing the boundary through sampling window constraints. The two complement each other, covering the entire data link from simulation rendering to training sampling.

[0067] Second, placeholder frame recognition boasts zero false positives and low overhead. This application performs precise matching based on the deterministic fact that the binary characteristics of placeholder frames are constant, rather than relying on statistical judgments of image content blackness thresholds. Therefore, it will not misclassify legitimate dark frames in real-world scenes (such as darkened rooms, shadow areas, and nighttime scenes) as placeholder frames. The false positive probability can be considered zero when using two-stage hash verification. Furthermore, recognition only requires reading file attributes such as file byte length, without decoding the image or performing pixel-by-pixel statistics, offering an order-of-magnitude speed advantage compared to image content-based detection methods.

[0068] Third, segmentation instead of splicing ensures true temporal continuity within each segment. This application uses placeholder frames as dividing boundaries to segment the trajectory into segments, rather than simply deleting placeholder frames and splicing the remaining frames. Therefore, while removing invalid placeholder frames, no new pseudo-adjacent frames are generated, and adjacent frames within each output segment are physically adjacent. This mechanism avoids the problem of pseudo-adjacent frames generated by splicing after deleting placeholder frames.

[0069] Fourth, zero effective data loss and no increase in training overhead. All real frames can still be covered by the training window, with boundary constraints only imposed on the starting position of the sampling window. The set of valid sampling starting positions is constructed and cached once during dataset initialization, and subsequent sampling only requires constant time to look up the table to obtain the window starting point, without increasing the per-sample computation overhead during training.

[0070] Fifth, the two methods of eliminating pseudo-adjacent frames through placeholder frame identification and segmentation, and sampling window constraints, are complementary and can be seamlessly linked in engineering. The sub-segment boundaries generated by placeholder frame identification and segmentation happen to become the trajectory boundaries of the constrained sampling window, and the two methods form a complete pseudo-adjacent frame elimination closed loop from rendering output to training sampling.

[0071] Sixth, this application eradicates hidden contamination from the data mechanism. The contamination of the temporal world model by pseudo-neighboring frames is not visible on the training loss curve, making it difficult to detect through conventional training monitoring. This application identifies and eliminates pseudo-neighboring frames from the data mechanism perspective, preventing the temporal world model from learning anti-physics teleportation priors and fundamentally ensuring the training effect of the temporal model.

[0072] Based on the training data processing method for the time series model in the foregoing embodiments of this application, this application also proposes a training method for the time series model.

[0073] In some embodiments of this application, the training method for a time series model includes: obtaining training samples using the training data processing method for time series model training as described in the foregoing embodiments of this application, and training a preset time series model using the training samples.

[0074] Specifically, the preset temporal model can be a temporal world model based on the Recurrent State-Space Model (RSSM), or any other model that takes consecutive temporal frames as input sequences, such as a Transformer-based temporal prediction model, a sequence model based on a Recurrent Neural Network (RNN / LSTM / GRU), or a video generation model based on a diffusion model. The training process can employ commonly used optimization algorithms (such as Adam, AdamW), loss functions (such as mean squared error (MSE), negative log-likelihood (NLL), variational lower bound (ELBO), and training strategies.

[0075] Since the training samples have been processed by the training data processing method of this application embodiment, each training sample is physically continuous and does not contain any pseudo-adjacent frames. Therefore, the trained temporal model will not learn anti-physical teleportation priors, and the environmental dynamic evolution law it learns is closer to the physical reality, thereby improving the model's performance in downstream tasks (such as end-to-end navigation, motion prediction, and environmental state estimation).

[0076] Based on the training data processing method or training method of the time series model in the foregoing embodiments of this application, this application also proposes a computer-readable storage medium storing a training data processing program or a training program of the time series model thereon. When the program is executed by a processor, it implements the training data processing method or training method of the time series model in the foregoing embodiments of this application.

[0077] It should be understood that the specific implementation of the computer-readable storage medium of this application can be found in the specific implementation of the training data processing method or the time series model training method of the foregoing embodiments of this application. To reduce redundancy, it will not be described again here.

[0078] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of this application.

[0079] like Figure 3 As shown, the electronic device 500 includes a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, for example, via a bus 502. Optionally, the electronic device 500 may also include a transceiver 504. It should be noted that in practical applications, the transceiver 504 is not limited to one type, and the structure of this electronic device 500 does not constitute a limitation on the embodiments of this application.

[0080] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 501 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0081] Bus 502 may include a pathway for transmitting information between the aforementioned components. Bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 502 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0082] The memory 503 is used to store computer programs corresponding to the training data processing method or training method of the timing model in the above embodiments of this application. These computer programs are controlled and executed by the processor 501. The processor 501 executes the computer programs stored in the memory 503 to implement the content shown in the aforementioned method embodiments.

[0083] Figure 3 The electronic device 500 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0084] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0085] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0086] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0087] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential", etc., indicating the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0088] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0089] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "joining," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0090] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "on top of," and "over" the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0091] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A training data processing method for time-series model training, characterized in that, The method includes: Obtain the sequence of time frame files output by the simulation rendering pipeline, wherein the sequence of time frame files is generated by continuous rendering by the camera in the simulation rendering pipeline, arranged in chronological order, and includes placeholder frames inserted by the simulation rendering pipeline when data is missing. The binary features of each frame file in the time-series frame file sequence are compared with preset binary features, and the matching frame file is determined as a placeholder frame. Using the placeholder frame as a dividing boundary, the sequence of time frame files is divided into several sub-segments; Each of the aforementioned segments is treated as an independent training unit, and they are concatenated in chronological order to form a flat, consecutively numbered frame index sequence, which is then stored in the training dataset. The binary features include the byte length of the frame file and the hash value of the frame file content; The preset binary feature is obtained in the following way: The simulation rendering pipeline is sampled in small batches with a frame rate less than a preset threshold. Placeholder frames inserted due to missing data are identified from the sampled output frame files, and the binary features of the identified placeholder frames are extracted.

2. The training data processing method for time-series model training according to claim 1, characterized in that, Before storing each of the sub-segments as an independent training unit in the training dataset, the method further includes: Discard segments whose frame count is less than the first segment length threshold; Wherein, the first segment length threshold is not less than the length of the input sequence of the time series model.

3. The training data processing method for time-series model training according to claim 1, characterized in that, The binary characteristics of the frame file are compared with preset binary characteristics, and the matching frame file is determined to be a placeholder frame, including: The byte length of the frame file is compared with the byte length in the preset binary feature; If the two bytes are the same length, then calculate the hash value of the frame file content; The hash value of the frame file content is compared with the hash value in the preset binary feature; If the two hash values ​​are the same, then the frame file is determined to be a matching frame file.

4. The training data processing method for time-series model training according to claim 1, characterized in that, The simulation rendering pipeline is equipped with multiple cameras, and the simulation rendering pipeline outputs multiple time frame file sequences, which correspond one-to-one with the multiple cameras and are aligned in time. Specifically, if any frame file from any camera at a certain moment is determined to be a placeholder frame, then all frame files from all cameras at that moment will be determined to be placeholder frames.

5. The training data processing method for time-series model training according to claim 1, characterized in that, The method further includes: The percentage of frames identified as placeholder frames out of the total number of frames is calculated as the placeholder frame drop rate and output; and / or Count the number of segments retained after segmentation and output the result.

6. The training data processing method for time-series model training according to any one of claims 1-5, characterized in that, The method further includes: In response to the initialization of the training dataset, the start and end positions of each training unit in the training dataset in the frame index sequence are obtained; For each training unit, determine the starting positions of all sampling windows with a length equal to the length of the input sequence that fall completely within the training unit, thus obtaining the set of valid sampling starting positions corresponding to the training unit; In response to the sampling request of the training sample, a starting position is selected from the set of legal sampling starting positions as the starting point of the window, and a series of consecutive input sequence lengths of frame files starting from the starting point are extracted from the frame index sequence to form a training sample.

7. A training method for a time series model, characterized in that, The method includes: Training samples are obtained using the training data processing method for time-series model training as described in claim 6; The preset time series model is trained using the training samples.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-7.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory, which, when executed by the processor, implements the method as described in any one of claims 1-7.

Images