Data augmentation method and device, computer device, and storage medium

Abstract

The application relates to the technical field of automatic driving and discloses a data expansion method and device, computer equipment and a storage medium, which comprise obtaining a first scene description; determining a matched target map region from a preset map data set based on the first scene description, and generating three-dimensional trajectory information of a dynamic target based on the first scene description, the target map region and a preset traffic flow generation model; performing coordinate conversion based on the three-dimensional trajectory information and a target vehicle to generate three-dimensional bounding box structured information and lane line structured information corresponding to the target vehicle; and processing the three-dimensional bounding box structured information and the lane line structured information to generate expansion data. The application matches a target map region through a first scene description in a natural language form and generates three-dimensional trajectory of a dynamic target, obtains structured information through coordinate conversion, generates scene data with multiple perspectives and continuous frames, reduces the acquisition cost of automatic driving data, and further improves the generalization ability of a model.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically to data expansion methods, devices, computer equipment, and storage media. Background Technology

[0002] With the rapid development of autonomous driving technology, deep learning models play a crucial role in core aspects such as environmental perception and decision-making. The performance of data-driven models relies on large-scale, high-quality training data. Especially in environmental perception tasks, the raw data needs to be labeled with bounding boxes, lane lines, etc., which is labor-intensive and time-consuming, making it difficult to meet the needs of large-scale model training. To alleviate the data shortage problem, related technologies typically employ data augmentation methods, specifically by performing geometric transformations and pixel-level operations on existing labeled image data. However, these methods lack sufficient diversity in the expanded data samples and struggle to cope with complex and ever-changing real-world traffic scenarios, such as extreme weather and complex traffic flows. Summary of the Invention

[0003] In view of this, the present invention provides a data expansion method, apparatus, computer equipment, and storage medium to solve the problem of limited data expansion.

[0004] In a first aspect, the present invention provides a data augmentation method, the method comprising:

[0005] Obtain a first-level scene description in natural language form;

[0006] Based on the first scene description, a matching target map area is determined from a preset map dataset. Based on the first scene description, the target map area, and a preset traffic flow generation model, three-dimensional trajectory information of the dynamic target is generated. The three-dimensional trajectory information includes the three-dimensional trajectory information of each dynamic target in a future continuous time period.

[0007] Based on the 3D trajectory information and the target vehicle, coordinate transformation is performed to generate 3D bounding box structured information and lane line structured information corresponding to the target vehicle.

[0008] The structured information of the three-dimensional bounding boxes and the structured information of the lane lines are processed to generate augmented data. The augmented data includes scene data from multiple perspectives and continuous frames, and the scene data includes three-dimensional annotation information for each frame of the image.

[0009] In one optional implementation, determining the matching target map region from a preset map dataset based on the first scene description includes:

[0010] The preset map dataset is divided into multiple candidate map regions, and the first scene description is converted into target map structured data, which includes the state encoding of dynamic targets and map information encoding.

[0011] Calculate the matching degree between each candidate map region and the target map structured data;

[0012] The candidate map region with the highest matching degree is determined as the target map region.

[0013] In one optional implementation, generating the three-dimensional trajectory information of the dynamic target based on the first scene description, the target map region, and a preset traffic flow generation model includes:

[0014] The target map area and structured scene data are input into the preset traffic flow generation model to extract map features based on the encoder module in the traffic flow generation model and generate an initial query vector for each dynamic target.

[0015] The initial query vectors of all dynamic targets are calculated based on the self-attention mechanism to fuse the dynamic targets and generate the first intermediate data.

[0016] The first intermediate data and the map features are calculated based on the cross-attention mechanism to fuse the dynamic targets and the map, and generate the second intermediate data.

[0017] The second intermediate data is decoded to generate the three-dimensional trajectory information of the dynamic target.

[0018] In one optional implementation, the step of inputting the target map region and structured scene data into the preset traffic flow generation model, so as to extract map features based on the encoder module in the traffic flow generation model and generate an initial query vector for each dynamic target, includes:

[0019] The target map region is input into a map encoder based on a convolutional neural network to extract map features;

[0020] The state encoding of each dynamic target in the structured scene data is input into a multilayer perceptron-based generator to generate the initial query vector for each dynamic target.

[0021] In one optional implementation, the step of performing coordinate transformation based on 3D trajectory information and the target vehicle to generate 3D bounding box structured information and lane line structured information corresponding to the target vehicle includes:

[0022] The three-dimensional trajectory information of other dynamic targets is transformed from the global coordinate system to a vehicle coordinate system centered on the target vehicle. The other dynamic targets do not include the target vehicle.

[0023] Based on preset camera parameters, the three-dimensional trajectory information of other dynamic targets in the vehicle coordinate system is projected onto the image coordinate system to generate three-dimensional bounding box structured information;

[0024] The lane line information in the target map area is transformed from the global coordinate system to the vehicle coordinate system, and projected onto the image coordinate system based on preset camera parameters to generate the lane line structured information.

[0025] In one optional implementation, the processing of the 3D bounding box structured information and lane line structured information to generate augmented data includes:

[0026] The structured information of the three-dimensional bounding box and the structured information of the lane lines are encoded to obtain spatial condition features;

[0027] The text environment description is encoded to obtain semantic condition features;

[0028] Expanded data is generated based on the spatial condition features and the semantic condition features.

[0029] In one optional implementation, generating augmented data based on the spatial condition features and the semantic condition features includes:

[0030] Based on the self-attention mechanism, random noise is interactively computed in the spatial and temporal dimensions to generate intermediate features;

[0031] The intermediate features, spatial condition features, and semantic condition features are fused together;

[0032] The fused features are decoded, and the multi-view and continuous frame autonomous driving scene video data is used as the augmented data.

[0033] The data augmentation method provided in this embodiment includes: acquiring a first scene description in natural language form; determining a matching target map region from a preset map dataset based on the first scene description; generating three-dimensional trajectory information of a dynamic target based on the first scene description, the target map region, and a preset traffic flow generation model; performing coordinate transformation based on the three-dimensional trajectory information and the target vehicle to generate three-dimensional bounding box structured information and lane line structured information corresponding to the target vehicle; and processing the three-dimensional bounding box structured information and lane line structured information to generate augmented data. This invention matches a target map region with a first scene description in natural language form and generates a dynamic target's three-dimensional trajectory. After obtaining structured information through coordinate transformation, it generates multi-view and continuous frame scene data. This eliminates the need for real data collection and complex manual annotation, reducing the cost of acquiring 3D autonomous driving data, providing more complex scene data for autonomous driving models, and thus improving the model's generalization ability.

[0034] In a second aspect, the present invention provides a data expansion device, the device comprising:

[0035] The data acquisition module is used to acquire a first scene description in natural language form;

[0036] The trajectory information generation module is used to determine the matching target map area from the preset map dataset based on the first scene description, and generate three-dimensional trajectory information of the dynamic target based on the first scene description, the target map area and the preset traffic flow generation model. The three-dimensional trajectory information includes the three-dimensional trajectory information of each dynamic target in a continuous time period in the future.

[0037] The information generation module is used to perform coordinate transformation based on the three-dimensional trajectory information and the target vehicle to generate three-dimensional bounding box structured information and lane line structured information corresponding to the target vehicle.

[0038] The data generation module is used to process the structured information of the three-dimensional bounding box and the structured information of the lane lines to generate extended data. The extended data includes scene data of multiple views and continuous frames, and the scene data includes three-dimensional annotation information of each frame image.

[0039] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the data expansion method described in the first aspect or any corresponding embodiment thereof.

[0040] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the data augmentation method described in the first aspect or any corresponding embodiment thereof. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a data augmentation method according to an embodiment of the present invention;

[0043] Figure 2 This is a structural block diagram of a data expansion device according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] According to an embodiment of the present invention, a data augmentation method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0047] This embodiment provides a data augmentation method. Figure 1 This is a flowchart of a data augmentation method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0048] Step S101: Obtain the first scene description in natural language form.

[0049] The first scene description can be obtained through manual input, calling a preset scene description library, or automatic conversion based on scene requirements. The first scene description is natural language text used to define the target scene for autonomous driving, providing semantic constraints for subsequent data generation and clarifying the traffic flow characteristics and map environment features of the scene. The first scene description includes the distribution and movement status of dynamic targets, as well as map information. As an example, the first scene description is: "Weekday morning rush hour, downtown intersection, heavy traffic, slow-moving vehicles, pedestrians crossing the crosswalk."

[0050] Step S102: Based on the first scene description, determine the matching target map area from the preset map dataset, and based on the first scene description, the target map area, and the preset traffic flow generation model, generate the three-dimensional trajectory information of the dynamic target.

[0051] The first scene description is transformed into structured data. The map environment features in the first scene description are analyzed to extract map attributes such as road type, number of lanes, and intersection location. Map feature retrieval conditions are constructed based on the selected map attributes. The preset map dataset consists of pre-collected real road map data or a standardized generated map data set. By calculating the matching degree between each map region and the retrieval conditions (such as feature similarity, attribute fit, etc.), the map region with the best matching degree is selected as the target map region.

[0052] The preset traffic flow generation model is a generative model with dynamic target interaction processing capabilities. The first scene description and the target map area are input into the preset traffic flow generation model. The preset traffic flow generation model analyzes the traffic flow features in the first scene description and combines them with the road constraints of the target map area (such as lane boundaries, intersection rules, etc.) to simulate the motion law of dynamic targets and generate the three-dimensional trajectory information of each dynamic target in the future continuous time period. The three-dimensional trajectory information includes the three-dimensional trajectory information of each dynamic target in the future continuous time period.

[0053] Step S103: Based on the 3D trajectory information and the target vehicle, perform coordinate transformation to generate 3D bounding box structured information and lane line structured information corresponding to the target vehicle.

[0054] The target vehicle is selected from dynamic targets and serves as the viewpoint reference for subsequent structured information. The selection method can include random selection, manual designation, or filtering based on scene requirements. A conventional coordinate system transformation algorithm is employed to determine the target vehicle's real-time position and attitude in 3D space based on its 3D trajectory information. The 3D trajectory information of other dynamic targets and lane line information in the target map area are transformed from the global coordinate system to a local coordinate system based on the target vehicle, and then further transformed to the image coordinate system according to preset imaging parameters.

[0055] Based on the converted location information of the dynamic targets, and combined with the standardized size parameters of various dynamic targets, 3D bounding box information for each dynamic target is generated and organized into structured 3D bounding box information according to a preset data format. The converted lane line information is then classified and organized to form structured lane line information. Both types of structured information are compatible with the format requirements of subsequent data processing, ensuring data parsability and consistency.

[0056] Step S104: Process the 3D bounding box structured information and lane line structured information to generate augmented data.

[0057] The expanded data includes scene data from multiple perspectives and consecutive frames, with each frame containing 3D annotation information. A model capable of feature extraction, fusion, and generation processes the structured information of the 3D bounding boxes and lane lines. This processing includes converting the structured information into model-recognizable feature vectors and performing feature interaction fusion.

[0058] Based on the processed feature information, multi-view continuous frame scene data is generated through model decoding. The multiple views can be preset according to actual needs (such as commonly used observation views such as forward, backward, and side views), and parameters such as frame rate and duration of continuous frames can be configured according to training data requirements. During the generation of scene data, structured information such as 3D bounding boxes and lane lines are synchronously converted into 3D annotation information corresponding to each frame image (such as annotation box coordinates, category labels, lane line attributes, etc.), ensuring accurate correspondence between scene data and annotation information.

[0059] The output augmented data is a dataset containing multi-view, continuous frame scene images and corresponding 3D annotation information. The dataset can be directly used for training, validation or testing of autonomous driving models, achieving efficient augmentation of 3D autonomous driving data.

[0060] The data augmentation method provided in this embodiment includes: acquiring a first scene description in natural language form; determining a matching target map region from a preset map dataset based on the first scene description; generating three-dimensional trajectory information of a dynamic target based on the first scene description, the target map region, and a preset traffic flow generation model; performing coordinate transformation based on the three-dimensional trajectory information and the target vehicle to generate three-dimensional bounding box structured information and lane line structured information corresponding to the target vehicle; and processing the three-dimensional bounding box structured information and lane line structured information to generate augmented data. This invention matches a target map region with a first scene description in natural language form and generates a dynamic target's three-dimensional trajectory. After obtaining structured information through coordinate transformation, it generates multi-view and continuous frame scene data. This eliminates the need for real data collection and complex manual annotation, reducing the cost of acquiring 3D autonomous driving data, providing more complex scene data for autonomous driving models, and thus improving the model's generalization ability.

[0061] In some optional implementations, step S102 above, which involves determining the matching target map region from a preset map dataset based on the first scene description, includes:

[0062] Step S201: Divide the preset map dataset into multiple candidate map regions and convert the first scene description into target map structured data.

[0063] The preset map dataset M is divided into regions using methods such as geographic boundary division, feature clustering, or grid division to obtain multiple candidate map regions. The division criteria can be set according to actual application needs, such as dividing by road type or dividing by geographic coordinate grid evenly, to ensure that each candidate map region has independent and complete map features such as roads, lane lines, and intersections.

[0064] The target map structured data includes dynamic target status encoding and map information encoding. Map information encoding rules are defined based on preset structured cue words. These rules include output format specifications. Map features are transformed into a 6-dimensional vector according to these rules, specifically including: the number of lane lines in the four directions (east, south, west, north), the distance from the map center to the intersection, and the lane ID of the vehicle. For example, encoding can be done in intervals such as 0-10m and 10-20m; for instance, 10 meters corresponds to index 2.

[0065] Dynamic targets include vehicles, pedestrians, etc. The state coding rules of dynamic targets are defined by preset structured prompt words. Each dynamic target is described by an 8-dimensional vector. Taking a vehicle as an example, the 8-dimensional vector specifically includes: the vehicle's quadrant position (quadrants 1-4), the distance of the vehicle from the center of the map, the direction of travel, the speed, and the vehicle's actions in a certain period of time in the future (e.g., 4 seconds) (e.g., turning left, accelerating, decelerating, etc.).

[0066] Step S202: Calculate the matching degree between each candidate map region and the target map structured data.

[0067] Step S203: The candidate map region with the highest matching degree is determined as the target map region.

[0068] For each candidate map region, its core map features are first extracted and converted into a map representation vector (v_m) according to map information encoding rules. The absolute value of the difference between the map representation vector (v_m) of each candidate map region and the target map feature vector (target_m) is calculated, i.e., |target_m-v_m|. The smaller the difference, the higher the matching degree. The matching degree calculation results of all candidate map regions are sorted, and the candidate map region with the smallest difference and the best score is selected as the target map region to ensure the adaptability of the map environment to the first scene description. If multiple candidate map regions have the same and highest matching degree, one can be randomly selected, or further filtering can be performed using the state encoding of the dynamic target.

[0069] In some optional implementations, step S102 above, based on the first scene description, the target map area, and a preset traffic flow generation model, generates three-dimensional trajectory information of the dynamic target, including:

[0070] Step 204: Input the target map area and structured scene data into the preset traffic flow generation model, so as to extract map features based on the encoder module in the traffic flow generation model and generate an initial query vector for each dynamic target.

[0071] The encoder module in the traffic flow generation model processes map feature extraction and initial query vector generation in parallel. Specifically, it extracts features from the target map area using a convolutional neural network or a multilayer perceptron, transforming the raw map data into a low-dimensional, high-density map feature vector. This vector contains information such as road constraints (e.g., lane width, turning rules) and spatial topological relationships (e.g., intersection connection methods). For each dynamic target, its 8-dimensional state code is used as input and transformed into a high-dimensional vector that the model can process through a linear projection layer. This vector is the initial query vector for each dynamic target, containing state information such as the target's initial position, speed, and intention.

[0072] Specifically, step S204 includes:

[0073] Step S2041: Input the target map region into a map encoder based on a convolutional neural network to extract map features.

[0074] First, the raw data of the target map area is standardized. The raw data of the target map area includes information such as the coordinates of key points of lane lines, intersection boundary polygons, road curvature, and lane width. The preprocessing process includes coordinate normalization and data format conversion to ensure that the data meets the input requirements of the map encoder.

[0075] The map encoder employs a multi-layered CNN (Convolutional Neural Network) stacked design, including convolutional layers and batch normalization layers. The target map region is input into the convolutional neural network-based map encoder, which outputs map feature vectors.

[0076] Step S2042: The state encoding of each dynamic target in the structured scene data is input into a multilayer perceptron-based generator to generate an initial query vector for each dynamic target.

[0077] The state code of each dynamic target in the structured scene data is used as the input to the generator of the multilayer perceptron. The generator adopts a fully connected layer stacked design, including an input layer, a hidden layer and an output layer. For each dynamic target, the generator outputs a corresponding fixed-dimensional initial query vector. The initial query vector transforms the discrete state information of the dynamic target into continuous high-dimensional features, including the initial position, motion state and action intention of the dynamic target.

[0078] Step 205: Calculate the initial query vectors of all dynamic targets based on the self-attention mechanism to fuse the dynamic targets and generate the first intermediate data.

[0079] An initial query vector set is constructed for all dynamic targets and input into a self-attention layer. The self-attention mechanism calculates the similarity between each dynamic target query vector and all other dynamic target query vectors. Based on attention weights, the query vectors of all dynamic targets are weighted and summed, fusing the state information of other dynamic targets into the feature vector of the current dynamic target. After multiple layers of self-attention calculation, a feature vector set that integrates the interaction information of all dynamic targets is output, i.e., the first intermediate data. The first intermediate data ensures that the trajectory planning of each dynamic target can respond to the state changes of other dynamic targets.

[0080] Step 206: Calculate the first intermediate data and map features based on the cross-attention mechanism to fuse the dynamic targets and the map, and generate the second intermediate data.

[0081] Using map feature vectors as keys and values ​​for the cross-attention layer, and the first intermediate data as the query, the attention weights of the fused feature vectors of each dynamic target (the first intermediate data) and the map feature vectors are calculated. Based on these attention weights, the map feature vectors are weighted and summed to fuse map constraint information into the feature vectors of the dynamic targets. The output is a set of feature vectors that fuses dynamic target interaction information and map constraint information, i.e., the second intermediate data. This second intermediate data simultaneously satisfies the motion requirements of the scene description, the interaction rules of the dynamic targets, and the environmental constraints of the map.

[0082] Step 207: Decode the second intermediate data to generate the three-dimensional trajectory information of the dynamic target.

[0083] The second intermediate data is input into the decoder module of the model. The decoder employs a Multi-Layer Perceptron (MLP) or Transformer decoder structure (a deep neural network architecture based on a self-attention mechanism, consisting of an encoder and a decoder stacked together). It predicts the spatial position of the dynamic target in a continuous future time period through stepwise regression. It outputs the 3D coordinates (x, y, z) of the dynamic target in the global coordinate system at each preset time step (e.g., 0.1 seconds / step), where the x and y coordinates represent the position on the horizontal plane, and the z coordinate represents the height (adapted for scenarios such as pedestrians and bridges). The output coordinate sequence is smoothed to eliminate abrupt changes in the trajectory, ensuring the continuity of the dynamic target's movement. Finally, it outputs the 3D coordinate sequence of each dynamic target in a continuous future time period (e.g., the next 10 seconds), i.e., the 3D trajectory information of the dynamic target.

[0084] In some alternative implementations, step S103 includes:

[0085] Step S301: Transform the three-dimensional trajectory information of other dynamic targets from the global coordinate system to a vehicle coordinate system centered on the target vehicle. The target vehicle is not included among the other dynamic targets.

[0086] The global coordinate system is a preset fixed coordinate system (such as a geodetic coordinate system or a custom world coordinate system) used to uniformly describe the spatial position of all dynamic targets and map areas. Based on 3D trajectory information, the position coordinates and attitude angles of the target vehicle in the global coordinate system are acquired in real time. The attitude angles can be calculated by fitting the trajectory direction of the dynamic target. The position coordinates and attitude angles of other dynamic targets are then transformed to a vehicle coordinate system centered on the target vehicle.

[0087] Step S302: Based on preset camera parameters, project the three-dimensional trajectory information of other dynamic targets in the vehicle coordinate system to the image coordinate system to generate three-dimensional bounding box structured information.

[0088] The preset camera parameters include intrinsic and extrinsic parameters. The intrinsic parameters are calibrated based on the hardware parameters of the acquisition device, while the extrinsic parameters define the camera's installation position and attitude in the vehicle coordinate system. All parameters are preset and stored in the system.

[0089] For each other dynamic target, based on its real-time position coordinates in the vehicle coordinate system and combined with preset dynamic target size parameters, the coordinates of the vertices of the 3D bounding box in the vehicle coordinate system are calculated. Taking vehicle-type dynamic targets as an example, the vertex coordinates are calculated by offsetting the origin. The vertex coordinates of the 3D bounding box are projected from the vehicle coordinate system to the image coordinate system, ultimately generating the structured information of the 3D bounding box.

[0090] Step S303: The lane line information in the target map area is transformed from the global coordinate system to the vehicle coordinate system, and projected onto the image coordinate system based on preset camera parameters to generate lane line structured information.

[0091] Extract the key point coordinate sequence of lane lines in the target map area and store it according to the line type. Convert the global coordinates of the key points of the lane lines to coordinates in the vehicle coordinate system to ensure that the lane line position is consistent with the dynamic target viewpoint.

[0092] Based on preset camera parameters, the coordinates of key points of lane lines in the vehicle coordinate system are projected onto the image coordinate system to obtain a pixel coordinate sequence. The pixel coordinate sequence is then organized according to the line type, and lane line types are distinguished by different channels or labels. Finally, the data is output according to a preset data format to obtain lane line structured information.

[0093] In some alternative implementations, step S104 includes:

[0094] Step S401: Encode the structured information of the three-dimensional bounding box and the structured information of the lane lines to obtain spatial condition features.

[0095] Before encoding the 3D bounding box and lane line structured information, the information is first standardized, including data dimension alignment and numerical normalization. A convolutional neural network or variational autoencoder structure is used as the input to the 3D bounding box structured information to extract features such as the spatial location, category, size, and multi-view projection of the dynamic target, outputting a bounding box feature map with unified dimensions. A network architecture originating from the bounding box encoding branch is used as the input to the lane line structured information to extract features such as the lane line type, position, direction, and relative spatial relationship with the dynamic target, outputting a lane line feature map. Based on a diffusion model, the encoded bounding box and lane line features are used as diffusion conditions, and a progressive denoising process is employed to generate images as spatial conditional features. The dimensions of the spatial conditional features can be preset according to actual needs.

[0096] Step S402: Encode the text environment description to obtain semantic condition features.

[0097] The text environment description is in natural language form and can be input simultaneously with the first scene description, or it can be input according to data expansion requirements. A pre-trained model with natural language semantic understanding capabilities is used as the encoder. The pre-processed text environment description is fused into the diffusion model in the form of cross-attention, influencing the diffusion direction, and finally generating a scene image that fits the text requirements, which serves as the semantic conditional feature of the text environment description.

[0098] Step S403: Generate augmented data based on spatial condition features and semantic condition features.

[0099] Further, step S403 includes:

[0100] Step S4031: Based on the self-attention mechanism, perform interactive calculations on random noise in the spatial and temporal dimensions to generate intermediate features.

[0101] Gaussian white noise is generated according to a preset dimension and used as the initial input for video sequence generation. The dimension of the noise remains consistent with the dimension of the subsequently generated scene video data. During training, a spatial attention module allows the model to capture the spatial relationships of objects in the image, ensuring the consistency of generated objects within a single frame across different viewpoints. The noise features of consecutive frames are arranged chronologically, and the attention weights of the noise features of the current frame and its preceding and following frames are calculated, fusing the temporal correlation information between frames. After multi-layer spatiotemporal self-attention computation, intermediate features with spatial correlation and temporal coherence are output.

[0102] Step S4032: The intermediate features, spatial condition features and semantic condition features are fused together.

[0103] The intermediate features are encoded using Gaussian white noise. These intermediate features are concatenated with the spatial conditional features to obtain a fused feature. This fused feature is then input into both the spatial attention module and the temporal attention module. The spatial attention module interacts with features at different spatial locations within the same time frame to enhance the spatial structural consistency of the generated content. The temporal attention module interacts with features at the same spatial location across consecutive time frames to ensure the temporal continuity and motion rationality of the generated video. Through processing by the spatial and temporal attention modules, intermediate features aligned with spatial layout and temporal dynamics are output.

[0104] The intermediate features after spatial-temporal alignment are used as queries, and the semantic condition features are used as keys and values. Through a cross-attention mechanism, the output is a fusion feature that simultaneously carries spatiotemporal consistency, spatial layout constraints, and semantic environment constraints.

[0105] Step S4033: Decode the fused features and use multi-view and continuous frame autonomous driving scene video data as augmented data.

[0106] A decoder architecture combining multi-layer transposed convolution and upsampling is adopted. First, the output dimension of the fused features is gradually increased through transposed convolutional layers, mapping the low-dimensional fused features into high-dimensional feature maps. Then, the feature map size is enlarged through bilinear interpolation upsampling algorithm, and the spatial resolution of the scene video is restored to the preset specifications simultaneously. During the decoding process, the camera parameters of six preset viewpoints (forward, backward, front left, front right, back left, and back right) are switched to ensure that the scene content at the same time step under different viewpoints matches each other. Finally, the decoded multi-view continuous frame feature maps are converted into autonomous driving scene video data. The autonomous driving scene video data is accompanied by the three-dimensional annotation information corresponding to each frame image as supplementary data output.

[0107] This embodiment also provides a data expansion device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0108] This embodiment provides a data expansion device, such as... Figure 2 As shown, it includes:

[0109] The data acquisition module is used to acquire a first scene description in natural language form;

[0110] The trajectory information generation module is used to determine the matching target map area from the preset map dataset based on the first scene description, and generate three-dimensional trajectory information of the dynamic target based on the first scene description, the target map area and the preset traffic flow generation model. The three-dimensional trajectory information includes the three-dimensional trajectory information of each dynamic target in a continuous time period in the future.

[0111] The information generation module is used to perform coordinate transformation based on the three-dimensional trajectory information and the target vehicle to generate three-dimensional bounding box structured information and lane line structured information corresponding to the target vehicle.

[0112] The data generation module is used to process the structured information of the three-dimensional bounding box and the structured information of the lane lines to generate extended data. The extended data includes scene data of multiple views and continuous frames, and the scene data includes three-dimensional annotation information of each frame image.

[0113] In some optional implementations, the trajectory information generation module includes:

[0114] A map partitioning unit is used to divide the preset map dataset into multiple candidate map regions and convert the first scene description into target map structured data, wherein the target map structured data includes the state encoding of dynamic targets and map information encoding;

[0115] A matching degree calculation unit is used to calculate the matching degree between each candidate map region and the target map structured data;

[0116] The target area determination unit is used to determine the candidate map area with the highest matching degree as the target map area.

[0117] In some optional implementations, the trajectory information generation module includes:

[0118] The vector generation unit is used to input the target map area and structured scene data into the preset traffic flow generation model, so as to extract map features based on the encoder module in the traffic flow generation model and generate an initial query vector for each dynamic target.

[0119] The first data generation unit is used to calculate the initial query vector of all dynamic targets based on the self-attention mechanism, so as to fuse the dynamic targets and generate the first intermediate data.

[0120] The second data generation unit is used to calculate the first intermediate data and the map features based on the cross-attention mechanism, so as to fuse the dynamic targets and the map to generate the second intermediate data.

[0121] The data decoding unit is used to decode the second intermediate data to generate the three-dimensional trajectory information of the dynamic target.

[0122] In some alternative implementations, the vector generation unit includes:

[0123] The map feature extraction subunit is used to input the target map region into a map encoder based on a convolutional neural network to extract map features;

[0124] An initial vector generation subunit is used to encode the state of each dynamic target in the structured scene data and input it into a multilayer perceptron-based generator to generate the initial query vector for each dynamic target.

[0125] In some alternative implementations, the information generation module includes:

[0126] The first transformation unit is used to transform the three-dimensional trajectory information of other dynamic targets from the global coordinate system to the vehicle coordinate system centered on the target vehicle. The other dynamic targets do not include the target vehicle.

[0127] The projection unit is used to project the three-dimensional trajectory information of other dynamic targets in the vehicle coordinate system to the image coordinate system based on preset camera parameters, and generate three-dimensional bounding box structured information.

[0128] The second conversion unit is used to convert the lane line information in the target map area from the global coordinate system to the vehicle coordinate system, and project it onto the image coordinate system based on preset camera parameters to generate the lane line structured information.

[0129] In some alternative implementations, the data generation module includes:

[0130] The first encoding unit is used to encode the structured information of the three-dimensional bounding box and the structured information of the lane lines to obtain spatial condition features;

[0131] The second encoding unit is used to encode the text environment description to obtain semantic condition features;

[0132] The data augmentation unit is used to generate augmented data based on the spatial condition features and the semantic condition features.

[0133] In some alternative implementations, the data augmentation unit includes:

[0134] The intermediate feature generation subunit is used to perform interactive computation on random noise in the spatial and temporal dimensions based on the self-attention mechanism to generate intermediate features.

[0135] The fusion subunit is used to fuse the intermediate features, the spatial condition features, and the semantic condition features;

[0136] The feature decoding subunit is used to decode the fused features and uses multi-view and continuous frame autonomous driving scene video data as the augmented data.

[0137] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0138] In this embodiment, the data expansion device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0139] This invention also provides a computer device having the above-described features. Figure 2 The data expansion device shown.

[0140] Please see Figure 3 , Figure 3This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 3 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 3 Take a processor 10 as an example.

[0141] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0142] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0143] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0145] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0146] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0147] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0148] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the invention.

Claims

1. A data augmentation method, characterized in that, The method includes: Obtain a first-level scene description in natural language form; Based on the first scene description, a matching target map area is determined from a preset map dataset. Based on the first scene description, the target map area, and a preset traffic flow generation model, three-dimensional trajectory information of the dynamic target is generated. The three-dimensional trajectory information includes the three-dimensional trajectory information of each dynamic target in a future continuous time period. Based on the 3D trajectory information and the target vehicle, coordinate transformation is performed to generate 3D bounding box structured information and lane line structured information corresponding to the target vehicle. The structured information of the three-dimensional bounding box and the structured information of the lane lines are processed to generate augmented data. The augmented data includes scene data from multiple perspectives and continuous frames, and the scene data includes three-dimensional annotation information of each frame image. The process of processing the structured information of the three-dimensional bounding box and the structured information of the lane lines to generate augmented data includes: The structured information of the three-dimensional bounding box and the structured information of the lane lines are encoded to obtain spatial condition features; The text environment description is encoded to obtain semantic condition features; Expanded data is generated based on the spatial condition features and the semantic condition features.

2. The data augmentation method according to claim 1, characterized in that, The step of determining the matching target map region from the preset map dataset based on the first scene description includes: The preset map dataset is divided into multiple candidate map regions, and the first scene description is converted into target map structured data, which includes the state encoding of dynamic targets and map information encoding. Calculate the matching degree between each candidate map region and the target map structured data; The candidate map region with the highest matching degree is determined as the target map region.

3. The data augmentation method according to claim 2, characterized in that, The step of generating three-dimensional trajectory information of a dynamic target based on the first scene description, the target map region, and a preset traffic flow generation model includes: The target map area and structured scene data are input into the preset traffic flow generation model to extract map features based on the encoder module in the traffic flow generation model and generate an initial query vector for each dynamic target. The initial query vectors of all dynamic targets are calculated based on the self-attention mechanism to fuse the dynamic targets and generate the first intermediate data. The first intermediate data and the map features are calculated based on the cross-attention mechanism to fuse the dynamic targets and the map, and generate the second intermediate data. The second intermediate data is decoded to generate the three-dimensional trajectory information of the dynamic target.

4. The data augmentation method according to claim 3, characterized in that, The step of inputting the target map region and structured scene data into the preset traffic flow generation model, and extracting map features based on the encoder module in the traffic flow generation model to generate an initial query vector for each dynamic target, includes: The target map region is input into a map encoder based on a convolutional neural network to extract map features; The state encoding of each dynamic target in the structured scene data is input into a multilayer perceptron-based generator to generate the initial query vector for each dynamic target.

5. The data augmentation method according to claim 1, characterized in that, The process of generating 3D bounding box structured information and lane line structured information corresponding to the target vehicle by performing coordinate transformation based on 3D trajectory information and the target vehicle includes: The three-dimensional trajectory information of other dynamic targets is transformed from the global coordinate system to a vehicle coordinate system centered on the target vehicle. The other dynamic targets do not include the target vehicle. Based on preset camera parameters, the three-dimensional trajectory information of other dynamic targets in the vehicle coordinate system is projected onto the image coordinate system to generate three-dimensional bounding box structured information; The lane line information in the target map area is transformed from the global coordinate system to the vehicle coordinate system, and projected onto the image coordinate system based on preset camera parameters to generate the lane line structured information.

6. The data augmentation method according to claim 1, characterized in that, The generation of augmented data based on the spatial condition features and the semantic condition features includes: Based on the self-attention mechanism, random noise is interactively computed in the spatial and temporal dimensions to generate intermediate features; The intermediate features, spatial condition features, and semantic condition features are fused together; The fused features are decoded, and the multi-view and continuous frame autonomous driving scene video data is used as the augmented data.

7. A data expansion device, characterized in that, The device includes: The data acquisition module is used to acquire a first scene description in natural language form; The trajectory information generation module is used to determine the matching target map area from the preset map dataset based on the first scene description, and generate three-dimensional trajectory information of the dynamic target based on the first scene description, the target map area and the preset traffic flow generation model. The three-dimensional trajectory information includes the three-dimensional trajectory information of each dynamic target in a continuous time period in the future. The information generation module is used to perform coordinate transformation based on the three-dimensional trajectory information and the target vehicle to generate three-dimensional bounding box structured information and lane line structured information corresponding to the target vehicle. The data generation module is used to process the structured information of the three-dimensional bounding box and the structured information of the lane lines to generate extended data. The extended data includes scene data of multiple views and continuous frames, and the scene data includes three-dimensional annotation information of each frame image. The data generation module includes: The first encoding unit is used to encode the structured information of the three-dimensional bounding box and the structured information of the lane lines to obtain spatial condition features; The second encoding unit is used to encode the text environment description to obtain semantic condition features; The data augmentation unit is used to generate augmented data based on the spatial condition features and the semantic condition features.

8. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the data expansion method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the data augmentation method according to any one of claims 1 to 6.

Images