A trajectory generation method, a target trajectory generation model training method, device, equipment, medium and product

Abstract

The application discloses a trajectory generation method, a target trajectory generation model training method, a device, equipment, a medium and a product. The method comprises the following steps: acquiring an environment image of a current vehicle; wherein the environment image comprises continuous multiple frames of images; inputting the environment image into a target trajectory generation model, processing the environment image through a Gaussian coding module in the target trajectory generation model to output a three-dimensional image Gaussian feature, and then processing the three-dimensional image Gaussian feature to output a predicted trajectory of the current vehicle. Through the technical scheme of the application, the vehicle trajectory can be accurately predicted.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of vehicle technology, and in particular to a trajectory generation method, a target trajectory generation model training method, an apparatus, a device, a medium, and a product. Background Technology

[0002] With the rapid development of artificial intelligence across various industries, autonomous driving technology, as an important application of AI, is gradually emerging. Autonomous driving refers to the ability to guide and make decisions regarding vehicle operation without requiring a driver to perform physical driving operations, and to replace the driver's control to ensure safe driving. The goal is to enable vehicles to autonomously travel along roads without human intervention, reaching their destination as quickly as possible while ensuring the vehicle's own safety and preventing any direct or indirect threats to the safety of other road users.

[0003] How to accurately predict vehicle trajectories is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a trajectory generation method, a target trajectory generation model training method, an apparatus, a device, a medium, and a product that can accurately predict vehicle trajectories.

[0005] According to one aspect of the present invention, a trajectory generation method is provided, comprising:

[0006] Acquire an environmental image of the current vehicle; wherein the environmental image includes multiple consecutive frames;

[0007] The environmental image is input into the target trajectory generation model, processed by the Gaussian coding module in the target trajectory generation model to output the Gaussian features of the three-dimensional image, and then processed based on the Gaussian features of the three-dimensional image to output the predicted trajectory of the current vehicle.

[0008] In some embodiments of the present invention, the target trajectory generation model includes: a Gaussian coding module, a perception module, and a decoder;

[0009] Accordingly, the environmental image is input into the target trajectory generation model, processed by the Gaussian coding module in the target trajectory generation model to output three-dimensional image Gaussian features, and then processed based on the three-dimensional image Gaussian features to output the predicted trajectory of the current vehicle, including:

[0010] The environmental image is input into the Gaussian encoding module to output Gaussian features of the three-dimensional image;

[0011] The Gaussian features of the three-dimensional image are input into the perception module to obtain the perception features output by the feature extraction layer of the perception module;

[0012] The perceived features and the Gaussian features of the 3D image are input into the decoder to output the predicted trajectory of the current vehicle.

[0013] According to another aspect of the present invention, a method for training a target trajectory generation model is provided, comprising:

[0014] Obtain a target sample set, wherein the target sample set includes: environmental image samples and labels carried by the environmental image samples, the environmental image samples include: consecutive multi-frame image samples, the labels carried by the environmental image samples include: obstacle information, road structure information, voxel category information and vehicle trajectory, and the voxel category information includes: category information of each voxel in the three-dimensional space corresponding to the consecutive multi-frame image samples;

[0015] The environmental image samples in the target sample set are input into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information and predicted vehicle trajectory.

[0016] Based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, the parameters of the initial trajectory generation model are trained to obtain the target trajectory generation model.

[0017] In some embodiments of the present invention, the initial trajectory generation model includes: a Gaussian encoding module, a Gaussian-to-voxel jet module, a sensing module, and a decoder;

[0018] The environmental image samples from the target sample set are input into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory, including:

[0019] The environmental image samples in the target sample set are input into the Gaussian encoding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples;

[0020] The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the Gaussian-to-voxel jetting module to obtain the predicted voxel category information.

[0021] The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the perception module to obtain predicted obstacle information and predicted road structure information.

[0022] The Gaussian features of the 3D image corresponding to the environmental image sample and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

[0023] In some embodiments of the present invention, the Gaussian coding module includes: a feature extraction unit and a Gaussian coding unit;

[0024] The environmental image samples in the target sample set are input into the Gaussian encoding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples, including:

[0025] The environmental image samples in the target sample set are input into the feature extraction unit to obtain multi-scale image features;

[0026] The initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0027] In some embodiments of the present invention, the feature extraction unit includes, from input to output, a backbone layer and a feature pyramid layer;

[0028] The environmental image samples in the target sample set are input into the feature extraction unit to obtain multi-scale image features, including:

[0029] The environmental image samples from the target sample set are input into the backbone layer to obtain initial image features;

[0030] The initial image features are input into the feature pyramid layer to obtain multi-scale image features.

[0031] In some embodiments of the present invention, the Gaussian coding unit includes, from input to output, an autoencoder layer, an attention layer, and a temporal coding layer in sequence.

[0032] The initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples, including:

[0033] Initial Gaussian noise is input into the autoencoder layer for enhancement processing to output the first Gaussian feature;

[0034] The first Gaussian feature and the multi-scale image feature are input into the attention layer to fuse the first Gaussian feature and the multi-scale image feature to output the initial image Gaussian feature.

[0035] The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing, so as to output the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0036] In some embodiments of the present invention, the timing coding layer includes, from input to output, a timing fusion layer and a correction layer in sequence;

[0037] The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing to output three-dimensional image Gaussian features, including:

[0038] The initial image Gaussian features are input into the temporal fusion layer to obtain the first image Gaussian features;

[0039] The Gaussian features of the first image are input into the correction layer for correction processing, so as to output the Gaussian features of the three-dimensional image.

[0040] In some embodiments of the present invention, the perception module includes: a feature extraction layer, a detection head, and a self-attention layer, and the obstacle information includes at least one of: obstacle position, obstacle size, and obstacle trajectory;

[0041] The Gaussian features of the 3D image are input into the perception module to obtain predicted obstacle information and predicted road structure information, including:

[0042] The Gaussian features of the three-dimensional image are voxelized to obtain the voxelized Gaussian features of the three-dimensional image.

[0043] The Gaussian features of the voxelized 3D image are input into the feature extraction layer to obtain perceptual features;

[0044] The sensing features are input into the detection head to obtain the predicted obstacle location, predicted obstacle size, and predicted road structure information;

[0045] The perceived features are input into the self-attention layer to obtain the predicted obstacle trajectory.

[0046] In some embodiments of the present invention, the Gaussian features of the three-dimensional image corresponding to the environmental image sample and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory, including:

[0047] Based on the attribute information of each Gaussian sphere in the Gaussian features of the three-dimensional image corresponding to the environmental image sample, the predicted three-dimensional image Gaussian features at multiple preset times are obtained.

[0048] The Gaussian features of the three-dimensional images at multiple preset times and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

[0049] In some embodiments of the present invention, the attribute information of the Gaussian sphere includes at least two of the following: the position of the Gaussian sphere, the size of the Gaussian sphere, the orientation of the Gaussian sphere, the category information of the Gaussian sphere, and the predicted offset of the Gaussian sphere at multiple preset times.

[0050] According to another aspect of the present invention, a trajectory generation apparatus is provided, the apparatus comprising:

[0051] An environmental image acquisition module is used to acquire environmental images of the current vehicle; wherein, the environmental images include multiple consecutive frames of images;

[0052] The trajectory generation module is used to input the environmental image into the target trajectory generation model, process it through the Gaussian coding module in the target trajectory generation model to output the Gaussian features of the three-dimensional image, and then process it based on the Gaussian features of the three-dimensional image to output the predicted trajectory of the current vehicle.

[0053] According to another aspect of the present invention, a target trajectory generation model training apparatus is provided, the apparatus comprising:

[0054] The target sample set acquisition module is used to acquire a target sample set, wherein the target sample set includes: environmental image samples and labels carried by the environmental image samples. The environmental image samples include: consecutive multi-frame image samples. The labels carried by the environmental image samples include: obstacle information, road structure information, voxel category information and vehicle trajectory. The voxel category information includes: category information of each voxel in the three-dimensional space corresponding to the consecutive multi-frame image samples.

[0055] The module for predicting obstacle information, road structure information, voxel category information, and vehicle trajectory determination is used to input environmental image samples from the target sample set into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory.

[0056] The training module is used to train the parameters of the initial trajectory generation model based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, so as to obtain the target trajectory generation model.

[0057] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0058] At least one processor; and

[0059] A memory communicatively connected to the at least one processor; wherein,

[0060] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method described in any embodiment of the present invention.

[0061] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the method described in any embodiment of the present invention.

[0062] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements the method described in any of the embodiments of the present invention.

[0063] The technical solution of this embodiment involves inputting the environmental image of the current vehicle into a target trajectory generation model. The Gaussian encoding module within the target trajectory generation model processes the image to output three-dimensional Gaussian features. Further processing based on these three-dimensional Gaussian features yields a predicted trajectory for the current vehicle. Since useless features have already been removed from the acquired three-dimensional Gaussian features, processing these features to obtain the predicted trajectory reduces model overhead. Furthermore, because acquiring three-dimensional Gaussian features takes into account all environmental factors affecting the vehicle's trajectory, the accuracy of trajectory prediction is improved.

[0064] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart of the first trajectory generation method in this embodiment of the invention;

[0067] Figure 2 This is a flowchart of the first target trajectory generation model training method in this embodiment of the invention;

[0068] Figure 3This is a flowchart of the second target trajectory generation model training method in this embodiment of the invention;

[0069] Figure 4 This is a schematic diagram of the structure of a trajectory generation device according to an embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of the structure of a target trajectory generation model training device according to an embodiment of the present invention;

[0071] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0072] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0073] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0074] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0075] In existing technologies, trajectory prediction is performed using traditional end-to-end autonomous driving models. Traditional end-to-end autonomous driving models are generally based on bird's-eye view (BEV) perception, and the most important aspect of BEV perception is transferring 2D features into 3D space.

[0076] The aforementioned end-to-end autonomous driving models typically suffer from the following drawbacks:

[0077] In BEV perception, all 2D features are transferred to 3D space, regardless of whether objects exist. In other words, BEV perception will have a lot of useless features in the process of transferring 2D features to 3D space, which will increase the model overhead and reduce the accuracy of trajectory prediction.

[0078] Example 1

[0079] Figure 1 This is a flowchart of a first trajectory generation method provided in an embodiment of the present invention. This embodiment is applicable to trajectory generation. The method can be executed by the trajectory generation device in this embodiment of the present invention, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0080] S101. Obtain the environmental image of the current vehicle.

[0081] In this embodiment, the environmental image includes multiple consecutive frames. These multiple consecutive frames may include images captured by the camera at time T, images captured by the camera at time T-1, images captured by the camera at time T-2, and images captured by the camera at time T-3, etc. The camera can be a camera mounted on the vehicle, for example, a left front camera, a front-view camera, a right front camera, a rear-view camera, a left rear camera, and a right rear camera. It should be noted that the number of left front cameras, front-view cameras, right front cameras, rear-view cameras, left rear cameras, and right rear cameras can be one or more; this embodiment of the invention does not impose any limitation on this.

[0082] S102. The environmental image is input into the target trajectory generation model, processed by the Gaussian coding module in the target trajectory generation model to output the Gaussian features of the three-dimensional image, and then processed based on the Gaussian features of the three-dimensional image to output the predicted trajectory of the current vehicle.

[0083] In this embodiment, the target trajectory generation model includes a Gaussian coding module, a perception module, and a decoder.

[0084] In this embodiment, the target trajectory generation model can be obtained by iteratively training an initial trajectory generation model using a target sample set. The target sample set includes: environmental image samples and labels carried by the environmental image samples. The environmental image samples include: multiple consecutive frames of images. The labels carried by the environmental image samples include: obstacle information, road structure information, voxel category information, and vehicle trajectory. The voxel category information includes: category information of each voxel in the three-dimensional space corresponding to the multiple consecutive frames of images. The initial trajectory generation model includes: a Gaussian encoding module, a Gaussian-to-voxel injection module, a perception module, and a decoder.

[0085] In this embodiment, the method of processing the Gaussian features of the three-dimensional image to output the predicted trajectory of the current vehicle can be as follows: feature extraction is performed on the Gaussian features of the three-dimensional image to obtain perceptual features, and the perceptual features and the Gaussian features of the three-dimensional image are decoded to obtain the predicted trajectory of the current vehicle.

[0086] The technical solution of this embodiment involves inputting the environmental image of the current vehicle into a target trajectory generation model. The Gaussian encoding module within the target trajectory generation model processes the image to output three-dimensional Gaussian features. Further processing based on these three-dimensional Gaussian features yields a predicted trajectory for the current vehicle. Since useless features have already been removed from the acquired three-dimensional Gaussian features, processing these features to obtain the predicted trajectory reduces model overhead. Furthermore, because acquiring three-dimensional Gaussian features takes into account all environmental factors affecting the vehicle's trajectory, the accuracy of trajectory prediction is improved.

[0087] Optionally, the target trajectory generation model includes: a Gaussian coding module, a perception module, and a decoder.

[0088] Accordingly, the environmental image is input into the target trajectory generation model, processed by the Gaussian coding module in the target trajectory generation model to output three-dimensional image Gaussian features, and then processed based on the three-dimensional image Gaussian features to output the predicted trajectory of the current vehicle, including:

[0089] The environmental image is input into the Gaussian encoding module to output Gaussian features of the three-dimensional image.

[0090] In this embodiment, the Gaussian coding module includes a feature extraction unit and a Gaussian coding unit.

[0091] In this embodiment, the method of inputting the environmental image into the Gaussian coding module to output the Gaussian features of the three-dimensional image can be as follows: inputting the environmental image into the feature extraction unit to obtain multi-scale image features; inputting the initial Gaussian noise and the multi-scale image features into the Gaussian coding unit to obtain the Gaussian features of the three-dimensional image.

[0092] In this embodiment, the feature extraction unit includes a backbone layer and a feature pyramid layer from input to output.

[0093] In this embodiment, the environmental image is input into the feature extraction unit to obtain multi-scale image features, including:

[0094] The environmental image is input into the backbone layer to obtain initial image features;

[0095] The initial image features are input into the feature pyramid layer to obtain multi-scale image features.

[0096] In this embodiment, the Gaussian coding unit includes, from input to output, an autoencoder layer, an attention layer, and a temporal coding layer.

[0097] In this embodiment, the initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features, including:

[0098] Initial Gaussian noise is input into the autoencoder layer for enhancement processing to output the first Gaussian feature;

[0099] The first Gaussian feature and the multi-scale image feature are input into the attention layer to fuse the first Gaussian feature and the multi-scale image feature to output the initial image Gaussian feature.

[0100] The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing to output the three-dimensional image Gaussian features.

[0101] In this embodiment, the timing coding layer includes, from input to output, a timing fusion layer and a correction layer.

[0102] The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing to output three-dimensional image Gaussian features, including:

[0103] The initial image Gaussian features are input into the temporal fusion layer to obtain the first image Gaussian features;

[0104] The Gaussian features of the first image are input into the correction layer for correction processing, so as to output the Gaussian features of the three-dimensional image.

[0105] The Gaussian features of the 3D image are input into the perception module to obtain the perception features output by the feature extraction layer of the perception module.

[0106] In this embodiment, the perception module includes: a feature extraction layer, a detection head, and a self-attention layer; the obstacle information includes: obstacle position, obstacle size, and obstacle trajectory.

[0107] The Gaussian features of the 3D image are input into the perception module to obtain the perception features output by the feature extraction layer of the perception module, including:

[0108] The Gaussian features of the three-dimensional image are voxelized to obtain the voxelized Gaussian features of the three-dimensional image.

[0109] The Gaussian features of the voxelized 3D image are input into the feature extraction layer to obtain perceptual features.

[0110] The perceived features and the Gaussian features of the 3D image are input into the decoder to output the predicted trajectory of the current vehicle.

[0111] In this embodiment, the perceived features and the Gaussian features of the 3D image are input into the decoder to output the predicted trajectory of the current vehicle, including:

[0112] Based on the attribute information of each Gaussian sphere in the Gaussian features of the three-dimensional image, Gaussian features of the three-dimensional image at multiple preset times are obtained.

[0113] The Gaussian features of the three-dimensional images at the multiple preset times and the perception features are input into the decoder to obtain the predicted trajectory of the current vehicle.

[0114] It should be noted that the Gaussian features of the three-dimensional image at the preset time can include the Gaussian features of the three-dimensional image in the next three seconds, such as the Gaussian features of the three-dimensional image at time T+0.5, time T+1, time T+1.5, time T+2, time T+2.5, and time T+3.

[0115] In this embodiment, the attribute information of the Gaussian sphere includes: the position of the Gaussian sphere, the size of the Gaussian sphere, the orientation of the Gaussian sphere, the category information of the Gaussian sphere, and the predicted offset of the Gaussian sphere at multiple preset times.

[0116] In this embodiment, the predicted Gaussian ball offsets at multiple preset times can be the Gaussian ball offsets for the next 3 seconds, for example, the Gaussian ball offsets at time T+0.5, T+1, T+1.5, T+2, T+2.5, and T+3.

[0117] The technical solution of this embodiment involves inputting the environmental image into the Gaussian encoding module to output Gaussian features of the three-dimensional image; inputting the Gaussian features of the three-dimensional image into the perception module to obtain the perception features output by the feature extraction layer of the perception module; and inputting the perception features and the Gaussian features of the three-dimensional image into the decoder to output the predicted trajectory of the current vehicle. This method allows for the acquisition of Gaussian features of the three-dimensional image through the Gaussian encoder. Since useless features have been removed from the acquired Gaussian features, decoding the perception features and the Gaussian features of the three-dimensional image to obtain the predicted trajectory can improve the accuracy of trajectory prediction while reducing model overhead.

[0118] Example 2

[0119] Figure 2 This is a flowchart of the first target trajectory generation model training method in this embodiment of the invention. This embodiment is applicable to the training of target trajectory generation models. This method can be executed by the target trajectory generation model training device in this embodiment of the invention, which can be implemented in software and / or hardware. Figure 2 As shown, the method specifically includes the following steps:

[0120] S201. Obtain the target sample set.

[0121] In this embodiment, the target sample set includes: environmental image samples and labels carried by the environmental image samples. The environmental image samples include: consecutive multi-frame image samples. The labels carried by the environmental image samples include: obstacle information, road structure information, voxel category information, and vehicle trajectory. The voxel category information includes: category information of each voxel in the three-dimensional space corresponding to the consecutive multi-frame image samples.

[0122] In this embodiment, the continuous multi-frame image samples included in the environmental image samples are historically acquired multi-frame images. These continuous multi-frame image samples may include: images captured by the camera at time K, images captured by the camera at time K-1, images captured by the camera at time K-2, and images captured by the camera at time K-3. It should be noted that the images captured by the camera at time K, time K-1, time K-2, and time K-3 are all historically acquired images. The cameras can be cameras installed on the vehicle, such as a left front camera, a front-view camera, a right front camera, a rear-view camera, a left rear camera, and a right rear camera. It should be noted that the number of left front cameras, front-view cameras, right front cameras, rear-view cameras, left rear cameras, and right rear cameras can be one or more; this embodiment of the invention does not impose any limitation on this.

[0123] In this embodiment, the obstacle information may include at least one of the following: the position of the obstacle within the detection frame, the size of the obstacle within the detection frame, and obstacle trajectories at multiple preset times. The position of the obstacle within the detection frame can be the position of the obstacle within the detection frame in the image captured by the camera at time K. The size of the obstacle within the detection frame can be the size of the obstacle within the detection frame in the image captured by the camera at time K. The multiple preset obstacle trajectories can be obstacle trajectories for the next 3 seconds, for example, the positions of the obstacle within the detection frame at time K+0.5, K+1, K+1.5, K+2, K+2.5, and K+3.

[0124] In this embodiment, the vehicle trajectory can be the vehicle trajectory at multiple preset times, and the vehicle trajectory at multiple preset times can be the vehicle trajectory for the next 3 seconds. For example, it can be the vehicle position at time K+0.5, vehicle position at time K+1, vehicle position at time K+1.5, vehicle position at time K+2, vehicle position at time K+2.5, and vehicle position at time K+3.

[0125] In this embodiment, the voxel category information in the label can be obtained as follows: A LiDAR (one or more units can be installed on the vehicle; if one unit is used, it is installed on the vehicle roof to collect point cloud data of the vehicle's environment from 360 degrees) is used to obtain the point cloud data of the vehicle's environment. This point cloud data is then input into a laser segmentation model to obtain the category information corresponding to each point cloud. The point cloud is then voxelized to obtain the category information of each voxel in the image space, thus obtaining the voxel category information label.

[0126] S202. Input the environmental image samples in the target sample set into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information and predicted vehicle trajectory.

[0127] Optionally, the initial trajectory generation model includes: a Gaussian encoding module, a Gaussian-to-voxel jetting module, a perception module, and a decoder;

[0128] The environmental image samples from the target sample set are input into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory, including:

[0129] The environmental image samples in the target sample set are input into the Gaussian encoding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0130] In this embodiment, the Gaussian coding module includes a feature extraction unit and a Gaussian coding unit.

[0131] In this embodiment, the method of inputting environmental image samples from the target sample set into the Gaussian coding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples can be as follows: inputting environmental image samples from the target sample set into the feature extraction unit to obtain multi-scale image features; inputting initial Gaussian noise and the multi-scale image features into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0132] The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the Gaussian-to-voxel jetting module to obtain the predicted voxel category information.

[0133] In this embodiment, the predicted voxel category information includes: the predicted category information of each voxel in three-dimensional space.

[0134] In this embodiment, the Gaussian-to-voxel jetting module is used to convert Gaussian features to obtain voxels in three-dimensional space and the predicted category information of each voxel.

[0135] The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the perception module to obtain predicted obstacle information and predicted road structure information.

[0136] In this embodiment, the perception module includes a feature extraction layer, a detection head, and a self-attention layer. The obstacle information includes obstacle position, obstacle size, and obstacle trajectory. The predicted obstacle trajectory can be the obstacle trajectory for the next 3 seconds, for example, the position of the obstacle in the detection box at time K+0.5, the position of the obstacle in the detection box at time K+1, the position of the obstacle in the detection box at time K+1.5, the position of the obstacle in the detection box at time K+2, the position of the obstacle in the detection box at time K+2.5, and the position of the obstacle in the detection box at time K+3.

[0137] In this embodiment, the method of inputting the Gaussian features of the three-dimensional image corresponding to the environmental image sample into the perception module to obtain the predicted obstacle information and the predicted road structure information can be as follows: inputting the Gaussian features of the three-dimensional image corresponding to the environmental image sample into the feature extraction layer to obtain the perception features; inputting the perception features into the detection head to obtain the predicted obstacle position, predicted obstacle size, and predicted road structure information; and inputting the perception features into the self-attention layer to obtain the predicted obstacle trajectory.

[0138] The Gaussian features of the 3D image corresponding to the environmental image sample and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

[0139] In this embodiment, the predicted vehicle trajectory can be the vehicle trajectory for the next 3 seconds, such as the vehicle position at time K+0.5, time K+1, time K+1.5, time K+2, time K+2.5, and time K+3.

[0140] The technical solution of this invention involves inputting environmental image samples from the target sample set into the Gaussian encoding module to obtain the 3D Gaussian features corresponding to the environmental image samples; inputting the 3D Gaussian features corresponding to the environmental image samples into the Gaussian-to-voxel jetting module to obtain predicted voxel category information; inputting the 3D Gaussian features corresponding to the environmental image samples into the perception module to obtain predicted obstacle information and predicted road structure information; and inputting the 3D Gaussian features corresponding to the environmental image samples and the perception features output by the feature extraction layer of the perception module into the decoder to obtain the predicted vehicle trajectory. This approach takes all environmental factors affecting the vehicle trajectory into account, thereby improving the accuracy of the trajectory predicted by the model.

[0141] Optionally, the Gaussian coding module includes: a feature extraction unit and a Gaussian coding unit;

[0142] The environmental image samples in the target sample set are input into the Gaussian encoding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples, including:

[0143] The environmental image samples in the target sample set are input into the feature extraction unit to obtain multi-scale image features.

[0144] In this embodiment, the feature extraction unit includes a backbone layer and a feature pyramid layer from input to output.

[0145] In this embodiment, the method of inputting environmental image samples from the target sample set into the feature extraction unit to obtain multi-scale image features can be as follows: inputting environmental image samples from the target sample set into the backbone layer to obtain initial image features; inputting the initial image features into the feature pyramid layer to obtain multi-scale image features.

[0146] The initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0147] In this embodiment, the Gaussian coding unit includes, from input to output, an autoencoder layer, an attention layer, and a temporal coding layer.

[0148] In this embodiment, the method of inputting the initial Gaussian noise and the multi-scale image features into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples can be as follows: inputting the initial Gaussian noise into the autoencoder layer for enhancement processing to output the first Gaussian feature; inputting the first Gaussian feature and the multi-scale image features into the attention layer for fusion processing to output the initial image Gaussian feature; and inputting the initial image Gaussian feature into the temporal coding layer for temporal information fusion processing to output the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0149] The technical solution of this invention involves inputting environmental image samples from the target sample set into the feature extraction unit to obtain multi-scale image features; inputting initial Gaussian noise and the multi-scale image features into the Gaussian coding unit to obtain three-dimensional Gaussian features corresponding to the environmental image samples. This method can fuse image features and Gaussian noise to obtain three-dimensional Gaussian features corresponding to the environmental image samples. The obtained three-dimensional Gaussian features have already eliminated useless features. Therefore, trajectory prediction based on the above-mentioned three-dimensional Gaussian features can improve the accuracy of trajectory prediction while reducing model overhead.

[0150] Optionally, the feature extraction unit includes, from input to output, a backbone layer and a feature pyramid layer;

[0151] The environmental image samples in the target sample set are input into the feature extraction unit to obtain multi-scale image features, including:

[0152] The environmental image samples from the target sample set are input into the backbone layer to obtain initial image features.

[0153] In this embodiment, the backbone layer can be a backbone neural network. A backbone neural network typically refers to the main component of a neural network, responsible for extracting features from the input data and efficiently transmitting information. The functions of a backbone neural network mainly include the following: Nonlinear mapping: Through its multi-layered structure and activation functions, the backbone network can achieve nonlinear mapping between input and output data, thereby capturing and expressing complex relationships in the data. Knowledge acquisition and representation: The weight parameters learned by the backbone neural network during training represent knowledge from input to output. This knowledge is stored and represented in the form of network parameters, facilitating subsequent reasoning and application. Associative memory: The backbone network has the ability to store and recall previously learned input data. This ability allows the network to quickly associate and match previously learned knowledge when faced with new input. Recognition, clustering, and classification: Through training, the backbone network can learn to perform tasks such as classification and clustering of input data. This is one of its core functions and is widely used in image recognition, speech recognition, natural language processing, and other fields. Optimization computation: During training, the backbone neural network continuously adjusts its internal parameters to minimize the loss function. This process involves complex optimization computation, aiming to find the optimal parameter configuration to achieve the best performance. In summary, the backbone neural network, as a key component of the neural network architecture, undertakes important tasks such as feature extraction, knowledge representation, pattern recognition, and optimization computation, and is the foundation for improving the overall performance of the neural network.

[0154] The initial image features are input into the feature pyramid layer to obtain multi-scale image features.

[0155] In this embodiment, the feature pyramid layer can be a Feature Pyramid Network (FPN), a deep neural network used for object detection and semantic segmentation. It improves the accuracy of detection and segmentation by constructing feature pyramids at multiple scales to extract object features at different scales. The core idea of ​​FPN is to construct the feature pyramid through cross-layer connections and a top-level feature pyramid, while preserving the semantic information of high-level features and the spatial information of low-level features.

[0156] The technical solution of this invention involves inputting environmental image samples from the target sample set into the backbone layer to obtain initial image features; then, inputting the initial image features into the feature pyramid layer to obtain multi-scale image features. This allows for the acquisition of image features at different scales, thereby improving the accuracy of subsequent trajectory prediction.

[0157] Optionally, the Gaussian coding unit includes, from input to output, an autoencoder layer, an attention layer, and a temporal coding layer.

[0158] The initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples, including:

[0159] Initial Gaussian noise is input into the autoencoder layer for enhancement processing to output a first Gaussian feature.

[0160] In this embodiment, the autoencoder layer includes a 3D sparse convolutional layer, which is used to convert low-dimensional initial Gaussian noise into high-dimensional Gaussian features.

[0161] In this embodiment, the size of the initial Gaussian noise is related to the detection range, which includes: the range in front and behind the vehicle, the range to the left and right, and the range in height. The initial Gaussian noise is random Gaussian noise, and the size of each Gaussian sphere in the initial Gaussian noise is within a preset size range.

[0162] The first Gaussian feature and the multi-scale image feature are input into the attention layer to fuse the first Gaussian feature and the multi-scale image feature to output the initial image Gaussian feature.

[0163] In this embodiment, the attention layer, from input to output, includes: a variable attention layer, a feedforward neural network layer, and a normalization layer. The variable attention layer can be a cross-attention layer. The attention layer is used to fuse image features and Gaussian features.

[0164] The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing, so as to output the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0165] In this embodiment, the temporal coding layer includes a temporal fusion layer and a refinement layer from input to output. The temporal fusion layer includes a 3D sparse convolutional layer and a 4D sparse convolutional layer. The temporal fusion layer is used to fuse temporal information. The refinement layer is used to refine the Gaussian features.

[0166] The technical solution of this invention involves inputting initial Gaussian noise into the autoencoder layer for enhancement processing to output a first Gaussian feature; inputting the first Gaussian feature and the multi-scale image features into the attention layer for fusion processing to output an initial image Gaussian feature; and inputting the initial image Gaussian feature into the temporal coding layer for temporal information fusion processing to output a three-dimensional image Gaussian feature corresponding to the environmental image sample. Since the obtained three-dimensional image Gaussian feature has already eliminated useless features, trajectory prediction based on the above-mentioned three-dimensional image Gaussian feature can improve the accuracy of trajectory prediction while reducing model overhead.

[0167] Optionally, the timing coding layer includes, from input to output, a timing fusion layer and a correction layer in sequence;

[0168] The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing to output three-dimensional image Gaussian features, including:

[0169] The Gaussian features of the initial image are input into the temporal fusion layer to obtain the Gaussian features of the first image.

[0170] In this embodiment, the first image Gaussian feature is the image Gaussian feature after fusing temporal features.

[0171] The Gaussian features of the first image are input into the correction layer for correction processing, so as to output the Gaussian features of the three-dimensional image.

[0172] In this embodiment, the correction layer can be a multilayer perceptron layer used to correct the Gaussian features of the image.

[0173] The technical solution of this invention involves inputting the initial image Gaussian features into the temporal fusion layer to obtain first image Gaussian features; then inputting the first image Gaussian features into the correction layer for correction processing to output three-dimensional image Gaussian features. This method obtains fused and corrected three-dimensional image Gaussian features. Based on these fused and corrected three-dimensional image Gaussian features, trajectory prediction accuracy can be improved.

[0174] Optionally, the perception module includes: a feature extraction layer, a detection head, and a self-attention layer, and the obstacle information includes at least two of the following: obstacle position, obstacle size, and obstacle trajectory;

[0175] The Gaussian features of the 3D image are input into the perception module to obtain predicted obstacle information and predicted road structure information, including:

[0176] The Gaussian features of the three-dimensional image are voxelized to obtain the voxelized Gaussian features of the three-dimensional image.

[0177] The Gaussian features of the voxelized 3D image are input into the feature extraction layer to obtain perceptual features.

[0178] In this embodiment, the feature extraction layer includes, from input to output, multiple 3D sparse convolutional layers and sparse max pooling layers.

[0179] The sensing features are input into the detection head to obtain the predicted obstacle location, predicted obstacle size, and predicted road structure information.

[0180] In this embodiment, the predicted obstacle position is the position of the detection frame containing the obstacle, and the predicted obstacle size is the size of the detection frame containing the obstacle. The predicted road structure includes: predicting the lane line position. It should be noted that the detection frame containing the obstacle is also known as the obstacle envelope.

[0181] The perceived features are input into the self-attention layer to obtain the predicted obstacle trajectory.

[0182] In this embodiment, the predicted obstacle trajectory can be the obstacle trajectory for the next 3 seconds.

[0183] The technical solution of this invention involves voxelizing the Gaussian features of the three-dimensional image to obtain voxelized Gaussian features; inputting the voxelized Gaussian features into the feature extraction layer to obtain perceptual features; inputting the perceptual features into the detection head to obtain predicted obstacle position, predicted obstacle size, and predicted road structure information; inputting the perceptual features into the self-attention layer to obtain predicted obstacle trajectory. This allows for the acquisition of predicted obstacle position, predicted obstacle size, predicted road structure information, and predicted obstacle trajectory. Furthermore, based on the differences between the predicted obstacle position, predicted obstacle size, predicted road structure information, and predicted obstacle trajectory and the environmental image sample labels, the parameters of the initial trajectory generation model are trained, resulting in a more accurate target trajectory generation model.

[0184] Optionally, the Gaussian features of the 3D image corresponding to the environmental image sample and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory, including:

[0185] Based on the attribute information of each Gaussian sphere in the Gaussian features of the three-dimensional image corresponding to the environmental image sample, the predicted three-dimensional image Gaussian features at multiple preset times are obtained.

[0186] The Gaussian features of the three-dimensional images at multiple preset times and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

[0187] In this embodiment, the method for obtaining the predicted Gaussian features of the three-dimensional image at multiple preset times based on the attribute information of each Gaussian sphere in the three-dimensional image Gaussian features can be as follows: The predicted Gaussian features of the three-dimensional image at multiple preset times are obtained based on the offsets of the Gaussian spheres predicted at multiple preset times in the three-dimensional image Gaussian features and the three-dimensional image Gaussian features themselves. For example, if the attribute information of each Gaussian sphere includes the offset of each Gaussian sphere in the next 3 seconds, then the three-dimensional image Gaussian features for the next 3 seconds can be obtained based on the offsets of the Gaussian spheres in the next 3 seconds and the three-dimensional image Gaussian features.

[0188] In a specific example, based on the offsets of each Gaussian ball in the next 3 seconds (the offsets of each Gaussian ball at time K+0.5, time K+1, time K+1.5, time K+2, time K+2.5, and time K+3) and the Gaussian features of the 3D image, the Gaussian features of the 3D image at time K+0.5, time K+1, time K+1.5, time K+2, time K+2.5, and time K+3 are obtained.

[0189] The technical solution of this invention obtains predicted Gaussian features of the three-dimensional image at multiple preset times based on the attribute information of each Gaussian sphere in the Gaussian features of the three-dimensional image; inputs the predicted Gaussian features of the three-dimensional image at multiple preset times and the perception features output by the feature extraction layer of the perception module into the decoder to obtain the predicted vehicle trajectory; then, based on the differences between the predicted obstacle information and the obstacle information in the label of the environmental image sample, the differences between the predicted road structure information and the road structure information in the label of the environmental image sample, the differences between the predicted voxel category information and the voxel category information in the label of the environmental image sample, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the label of the environmental image sample, the parameters of the initial trajectory generation model are trained to obtain a more accurate target trajectory generation model.

[0190] Optionally, the attribute information of the Gaussian sphere includes at least two of the following: the position of the Gaussian sphere, the size of the Gaussian sphere, the orientation of the Gaussian sphere, the category information of the Gaussian sphere, and the predicted offset of the Gaussian sphere at multiple preset times.

[0191] In this embodiment, the predicted Gaussian ball offsets at multiple preset times can be the Gaussian ball offsets for the next 3 seconds, for example, the Gaussian ball offsets at time T+0.5, T+1, T+1.5, T+2, T+2.5, and T+3.

[0192] S203. Based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, train the parameters of the initial trajectory generation model to obtain the target trajectory generation model.

[0193] In this embodiment, the parameters of the initial trajectory generation model include: the parameters of the Gaussian encoding module, the parameters of the Gaussian-to-voxel jet module, the parameters of the perception module, and the parameters of the decoder.

[0194] In this embodiment, after training the parameters of the initial trajectory generation model based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, it is necessary to return to the initial trajectory generation model and input the environmental image samples in the target sample set into the initial trajectory generation model to obtain the predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory, until the iteration termination condition is met, and the target trajectory generation model is obtained.

[0195] In this embodiment, the iteration termination condition may include at least one of the following conditions:

[0196] The number of iterations equals the threshold number.

[0197] The differences between the predicted obstacle information and the obstacle information in the label of the environmental image sample, the differences between the predicted road structure information and the road structure information in the label of the environmental image sample, the differences between the predicted voxel category information and the voxel category information in the label of the environmental image sample, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the label of the environmental image sample are all less than their respective difference thresholds.

[0198] The model accuracy is greater than the accuracy threshold.

[0199] It should be noted that after the target trajectory generation model is trained based on the above training process, S101 in the above embodiment is executed.

[0200] In existing technologies, only obstacle detection boxes and lane lines are used as sample labels, while other environmental factors (such as buildings, plants, etc.) are not taken into account. These other environmental factors can indirectly affect the vehicle trajectory in complex driving scenarios, resulting in low accuracy of the predicted trajectory.

[0201] In a specific example, such as Figure 3As shown, the initial trajectory generation model includes: a Gaussian coding module (not shown in the figure), a Gaussian-to-voxel jetting module, a perception module, and a decoder. The Gaussian coding module includes: a Gaussian coding unit and a feature extraction unit. The Gaussian coding unit includes: an autoencoder layer, an attention layer, and a temporal coding layer. The temporal coding layer includes: a temporal fusion layer and a correction layer. The feature extraction unit includes: a backbone layer and a feature pyramid layer. The perception module includes: a feature extraction layer, a detection head, and a self-attention layer. Environmental image samples at time K, K-1, K-2, and K-3 are input into the backbone layer to obtain initial image features. These initial image features are then input into the feature pyramid layer to obtain multi-scale image features. Initial Gaussian noise is input into the autoencoder layer for enhancement processing to output a first Gaussian feature. The first Gaussian feature and the multi-scale image features are input into the attention layer for fusion processing to output an initial image Gaussian feature. This initial image Gaussian feature is then input into the temporal fusion layer to obtain a first image Gaussian feature. The first image Gaussian feature is then input into the correction layer for correction processing to output a three-dimensional image Gaussian feature. Finally, the three-dimensional image Gaussian feature is input into the Gaussian-to-voxel jetting module to obtain predicted voxel category information. The Gaussian features of the 3D image are voxelized to obtain voxelized Gaussian features of the 3D image; the voxelized Gaussian features of the 3D image are input into the feature extraction layer to obtain perceptual features; the perceptual features are input into the detection head to obtain predicted obstacle position, predicted obstacle size, and predicted road structure information; based on the attribute information of each Gaussian sphere in the Gaussian features of the 3D image, predicted Gaussian features of the 3D image at multiple preset times are obtained; the predicted Gaussian features of the 3D image at multiple preset times and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory. The parameters of the initial trajectory generation model are trained based on the differences between the predicted obstacle location and the obstacle location in the label of the environmental image sample, the differences between the predicted obstacle size and the obstacle size in the label of the environmental image sample, the differences between the predicted obstacle trajectory and the obstacle trajectory in the label of the environmental image sample, the differences between the predicted road structure information and the road structure information in the label of the environmental image sample, the differences between the predicted voxel category information and the voxel category information in the label of the environmental image sample, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the label of the environmental image sample.

[0202] The technical solution of this embodiment involves inputting environmental image samples from a target sample set into an initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory. Based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, the parameters of the initial trajectory generation model are trained. This approach considers all environmental factors that directly or indirectly affect vehicle trajectory, thereby improving the accuracy of the trajectory predicted by the model.

[0203] Example 3

[0204] Figure 4 This is a schematic diagram of a trajectory generation device provided in an embodiment of the present invention. This embodiment is applicable to trajectory generation applications. The device can be implemented using software and / or hardware, and can be integrated into any device that provides trajectory generation functionality, such as… Figure 4 As shown, the trajectory generation device specifically includes an environmental image acquisition module 401 and a trajectory generation module 402.

[0205] The environmental image acquisition module is used to acquire the environmental image of the current vehicle; wherein the environmental image includes multiple consecutive frames.

[0206] The trajectory generation module is used to input the environmental image into the target trajectory generation model, process it through the Gaussian coding module in the target trajectory generation model to output the Gaussian features of the three-dimensional image, and then process it based on the Gaussian features of the three-dimensional image to output the predicted trajectory of the current vehicle.

[0207] Optionally, the target trajectory generation model includes: a Gaussian encoding module, a perception module, and a decoder;

[0208] The trajectory generation module is specifically used for:

[0209] The environmental image is input into the Gaussian encoding module to output Gaussian features of the three-dimensional image;

[0210] The Gaussian features of the three-dimensional image are input into the perception module to obtain the perception features output by the feature extraction layer of the perception module;

[0211] The perceived features and the Gaussian features of the 3D image are input into the decoder to output the predicted trajectory of the current vehicle.

[0212] The above-mentioned products can execute the trajectory generation method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of the execution method.

[0213] Example 4

[0214] Figure 5 This is a schematic diagram of a target trajectory generation model training device provided in an embodiment of the present invention. This embodiment is applicable to target trajectory generation model training. The device can be implemented using software and / or hardware, and can be integrated into any device that provides target trajectory generation model training functionality, such as… Figure 5 As shown, the target trajectory generation model training device specifically includes: a target sample set acquisition module 501, a module for predicting obstacle information, predicting road structure information, predicting voxel category information, and predicting vehicle trajectory determination 502, and a training module 503.

[0215] The target sample set acquisition module is used to acquire a target sample set, wherein the target sample set includes: environmental image samples and labels carried by the environmental image samples. The environmental image samples include: consecutive multi-frame image samples. The labels carried by the environmental image samples include: obstacle information, road structure information, voxel category information and vehicle trajectory. The voxel category information includes: category information of each voxel in the three-dimensional space corresponding to the consecutive multi-frame image samples.

[0216] The module for predicting obstacle information, road structure information, voxel category information, and vehicle trajectory determination is used to input environmental image samples from the target sample set into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory.

[0217] The training module is used to train the parameters of the initial trajectory generation model based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, so as to obtain the target trajectory generation model.

[0218] Optionally, the initial trajectory generation model includes: a Gaussian encoding module, a Gaussian-to-voxel jetting module, a perception module, and a decoder;

[0219] The trajectory generation module is also used for:

[0220] The environmental image samples in the target sample set are input into the Gaussian encoding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples;

[0221] The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the Gaussian-to-voxel jetting module to obtain the predicted voxel category information.

[0222] The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the perception module to obtain predicted obstacle information and predicted road structure information.

[0223] The Gaussian features of the 3D image corresponding to the environmental image sample and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

[0224] Optionally, the Gaussian coding module includes: a feature extraction unit and a Gaussian coding unit;

[0225] The trajectory generation module is also used for:

[0226] The environmental image samples in the target sample set are input into the feature extraction unit to obtain multi-scale image features;

[0227] The initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0228] Optionally, the feature extraction unit includes, from input to output, a backbone layer and a feature pyramid layer;

[0229] The trajectory generation module is also used for:

[0230] The environmental image samples from the target sample set are input into the backbone layer to obtain initial image features;

[0231] The initial image features are input into the feature pyramid layer to obtain multi-scale image features.

[0232] Optionally, the Gaussian coding unit includes, from input to output, an autoencoder layer, an attention layer, and a temporal coding layer.

[0233] The trajectory generation module is also used for:

[0234] Initial Gaussian noise is input into the autoencoder layer for enhancement processing to output the first Gaussian feature;

[0235] The first Gaussian feature and the multi-scale image feature are input into the attention layer to fuse the first Gaussian feature and the multi-scale image feature to output the initial image Gaussian feature.

[0236] The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing, so as to output the three-dimensional image Gaussian features corresponding to the environmental image samples.

[0237] Optionally, the timing coding layer includes, from input to output, a timing fusion layer and a correction layer in sequence;

[0238] The trajectory generation module is also used for:

[0239] The initial image Gaussian features are input into the temporal fusion layer to obtain the first image Gaussian features;

[0240] The Gaussian features of the first image are input into the correction layer for correction processing, so as to output the Gaussian features of the three-dimensional image.

[0241] Optionally, the perception module includes: a feature extraction layer, a detection head, and a self-attention layer, and the obstacle information includes at least one of: obstacle position, obstacle size, and obstacle trajectory;

[0242] The trajectory generation module is also used for:

[0243] The Gaussian features of the three-dimensional image are voxelized to obtain the voxelized Gaussian features of the three-dimensional image.

[0244] The Gaussian features of the voxelized 3D image are input into the feature extraction layer to obtain perceptual features;

[0245] The sensing features are input into the detection head to obtain the predicted obstacle location, predicted obstacle size, and predicted road structure information;

[0246] The perceived features are input into the self-attention layer to obtain the predicted obstacle trajectory.

[0247] Optionally, the trajectory generation module is also used for:

[0248] Based on the attribute information of each Gaussian sphere in the Gaussian features of the three-dimensional image corresponding to the environmental image sample, the predicted three-dimensional image Gaussian features at multiple preset times are obtained.

[0249] The Gaussian features of the three-dimensional images at multiple preset times and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

[0250] Optionally, the attribute information of the Gaussian sphere includes at least two of the following: the position of the Gaussian sphere, the size of the Gaussian sphere, the orientation of the Gaussian sphere, the category information of the Gaussian sphere, and the predicted offset of the Gaussian sphere at multiple preset times.

[0251] The above-mentioned products can execute the target trajectory generation model training method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of the execution method.

[0252] Example 5

[0253] Figure 6 A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0254] like Figure 6 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 or a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded from storage unit 68 into the RAM 63. The RAM 63 can also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0255] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0256] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as trajectory generation methods or target trajectory generation model training methods.

[0257] In some embodiments, the trajectory generation method, or the target trajectory generation model training method, may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the trajectory generation method or the target trajectory generation model training method described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to execute the trajectory generation method or the target trajectory generation model training method by any other suitable means (e.g., by means of firmware).

[0258] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0259] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0260] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (e.g., non-volatile memory (EPROM) or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0261] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0262] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0263] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0264] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the trajectory generation method or the target trajectory generation model training method of any embodiment of the present invention.

[0265] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0266] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0267] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A trajectory generation method, characterized in that, include: Acquire an environmental image of the current vehicle; wherein the environmental image includes multiple consecutive frames; The environmental image is input into the target trajectory generation model, processed by the Gaussian coding module in the target trajectory generation model to output the Gaussian features of the three-dimensional image, and then processed based on the Gaussian features of the three-dimensional image to output the predicted trajectory of the current vehicle.

2. The method according to claim 1, characterized in that, The target trajectory generation model includes: a Gaussian encoding module, a perception module, and a decoder; Accordingly, the environmental image is input into the target trajectory generation model, processed by the Gaussian coding module in the target trajectory generation model to output three-dimensional image Gaussian features, and then processed based on the three-dimensional image Gaussian features to output the predicted trajectory of the current vehicle, including: The environmental image is input into the Gaussian encoding module to output Gaussian features of the three-dimensional image; The Gaussian features of the three-dimensional image are input into the perception module to obtain the perception features output by the feature extraction layer of the perception module; The perceived features and the Gaussian features of the 3D image are input into the decoder to output the predicted trajectory of the current vehicle.

3. A method for training a target trajectory generation model, characterized in that, include: Obtain a target sample set, wherein the target sample set includes: environmental image samples and labels carried by the environmental image samples, the environmental image samples include: consecutive multi-frame image samples, the labels carried by the environmental image samples include: obstacle information, road structure information, voxel category information and vehicle trajectory, and the voxel category information includes: category information of each voxel in the three-dimensional space corresponding to the consecutive multi-frame image samples; The environmental image samples in the target sample set are input into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information and predicted vehicle trajectory. Based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, the parameters of the initial trajectory generation model are trained to obtain the target trajectory generation model.

4. The method according to claim 3, characterized in that, The initial trajectory generation model includes: a Gaussian encoding module, a Gaussian-to-voxel jetting module, a sensing module, and a decoder; The environmental image samples from the target sample set are input into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory, including: The environmental image samples in the target sample set are input into the Gaussian encoding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples; The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the Gaussian-to-voxel jetting module to obtain the predicted voxel category information. The Gaussian features of the three-dimensional image corresponding to the environmental image sample are input into the perception module to obtain predicted obstacle information and predicted road structure information. The Gaussian features of the 3D image corresponding to the environmental image sample and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

5. The method according to claim 4, characterized in that, The Gaussian coding module includes: a feature extraction unit and a Gaussian coding unit; The environmental image samples in the target sample set are input into the Gaussian encoding module to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples, including: The environmental image samples in the target sample set are input into the feature extraction unit to obtain multi-scale image features; The initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples.

6. The method according to claim 5, characterized in that, The feature extraction unit includes, from input to output, a backbone layer and a feature pyramid layer. The environmental image samples in the target sample set are input into the feature extraction unit to obtain multi-scale image features, including: The environmental image samples from the target sample set are input into the backbone layer to obtain initial image features; The initial image features are input into the feature pyramid layer to obtain multi-scale image features.

7. The method according to claim 5 or 6, characterized in that, The Gaussian coding unit, from input to output, includes, in sequence, an autoencoder layer, an attention layer, and a temporal coding layer; The initial Gaussian noise and the multi-scale image features are input into the Gaussian coding unit to obtain the three-dimensional image Gaussian features corresponding to the environmental image samples, including: Initial Gaussian noise is input into the autoencoder layer for enhancement processing to output the first Gaussian feature; The first Gaussian feature and the multi-scale image feature are input into the attention layer to fuse the first Gaussian feature and the multi-scale image feature to output the initial image Gaussian feature. The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing, so as to output the three-dimensional image Gaussian features corresponding to the environmental image samples.

8. The method according to claim 7, characterized in that, The timing coding layer, from input to output, includes: a timing fusion layer and a correction layer; The initial image Gaussian features are input into the temporal coding layer for temporal information fusion processing to output three-dimensional image Gaussian features, including: The initial image Gaussian features are input into the temporal fusion layer to obtain the first image Gaussian features; The Gaussian features of the first image are input into the correction layer for correction processing, so as to output the Gaussian features of the three-dimensional image.

9. The method according to any one of claims 4-8, characterized in that, The perception module includes a feature extraction layer, a detection head, and a self-attention layer. The predicted obstacle information includes at least one of the following: obstacle position, obstacle size, and obstacle trajectory. The Gaussian features of the 3D image are input into the perception module to obtain predicted obstacle information and predicted road structure information, including: The Gaussian features of the three-dimensional image are voxelized to obtain the voxelized Gaussian features of the three-dimensional image. The Gaussian features of the voxelized 3D image are input into the feature extraction layer to obtain perceptual features; The sensing features are input into the detection head to obtain the predicted obstacle location, predicted obstacle size, and predicted road structure information; The perceived features are input into the self-attention layer to obtain the predicted obstacle trajectory.

10. The method according to any one of claims 4-9, characterized in that, The Gaussian features of the 3D image corresponding to the environmental image sample and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory, including: Based on the attribute information of each Gaussian sphere in the Gaussian features of the three-dimensional image corresponding to the environmental image sample, the predicted three-dimensional image Gaussian features at multiple preset times are obtained. The Gaussian features of the three-dimensional images at multiple preset times and the perceptual features output by the feature extraction layer of the perception module are input into the decoder to obtain the predicted vehicle trajectory.

11. The method according to claim 10, characterized in that, The attribute information of the Gaussian sphere includes at least two of the following: the position of the Gaussian sphere, the size of the Gaussian sphere, the orientation of the Gaussian sphere, the category information of the Gaussian sphere, and the predicted offset of the Gaussian sphere at multiple preset times.

12. A trajectory generation device, characterized in that, include: An environmental image acquisition module is used to acquire environmental images of the current vehicle; wherein, the environmental images include multiple consecutive frames of images; The trajectory generation module is used to input the environmental image into the target trajectory generation model, process it through the Gaussian coding module in the target trajectory generation model to output the Gaussian features of the three-dimensional image, and then process it based on the Gaussian features of the three-dimensional image to output the predicted trajectory of the current vehicle.

13. A target trajectory generation model training device, characterized in that, include: The target sample set acquisition module is used to acquire a target sample set, wherein the target sample set includes: environmental image samples and labels carried by the environmental image samples. The environmental image samples include: consecutive multi-frame image samples. The labels carried by the environmental image samples include: obstacle information, road structure information, voxel category information and vehicle trajectory. The voxel category information includes: category information of each voxel in the three-dimensional space corresponding to the consecutive multi-frame image samples. The module for predicting obstacle information, road structure information, voxel category information, and vehicle trajectory determination is used to input environmental image samples from the target sample set into the initial trajectory generation model to obtain predicted obstacle information, predicted road structure information, predicted voxel category information, and predicted vehicle trajectory. The training module is used to train the parameters of the initial trajectory generation model based on the differences between the predicted obstacle information and the obstacle information in the labels of the environmental image samples, the differences between the predicted road structure information and the road structure information in the labels of the environmental image samples, the differences between the predicted voxel category information and the voxel category information in the labels of the environmental image samples, and the differences between the predicted vehicle trajectory and the vehicle trajectory in the labels of the environmental image samples, so as to obtain the target trajectory generation model.

14. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-11.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to perform the method of any one of claims 1-11.

16. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-11.

Images