Driving track generation method, electronic equipment, storage medium and program product

Abstract

The invention discloses a driving track generation method, an electronic device, a storage medium and a program product, and relates to the technical field of automatic driving, and the method comprises the steps: obtaining an environment image of a target vehicle, semantic feature extraction indication information and a basic driving track, and then obtaining a driving track; and inputting the environment image and the extraction indication information of the semantic features into the language model, so that the language model can extract the target semantic features required for generating the driving track. Secondly, target image features are directly extracted from the environment image; furthermore, the target semantic features and the target image features can serve as guidance conditions for track generation, and after the basic driving track is adjusted, a target driving track meeting the target conditions is obtained. Therefore, in the process of generating the driving track, the driving track is generated by adopting the environment image acquired in real time instead of using a high-definition map, so that the application range is wider.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to a driving trajectory generation method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] In the technical field of automatic driving, the accuracy of trajectory prediction is directly related to driving safety. The trajectory prediction technology is mainly based on the current vehicle state, surrounding environment information and driving intention to predict the future driving path, so that the current vehicle can drive based on the predicted driving path.

[0003] The current trajectory prediction technology generally depends on a high-definition map. When predicting a trajectory, the real-time captured surrounding environment information of the vehicle and the pre-stored high-definition map are fused to accurately position the vehicle on the accurate position of the high-definition map and obtain accurate traffic information, such as road topology, lane lines and traffic light positions. However, the production cost of the high-definition map is high, the maintenance cycle is long, it cannot cover all roads, and the application conditions are relatively harsh, that is, it can only be applied to road scenes where a high-definition map has been constructed. SUMMARY

[0004] The present application provides a driving trajectory generation method, device, electronic device, storage medium and program product to solve the problem of limited application range of high-definition maps.

[0005] The present application provides a driving trajectory generation method, comprising: obtaining an environment image of a target vehicle, extraction indication information of semantic features, and a pre-generated basic driving trajectory; inputting the environment image and the extraction indication information of the semantic features into a pre-trained language model to obtain target semantic features output by the language model; extracting target image features from the environment image; generating a target condition based on the target image features and the target semantic features; generating a target driving trajectory meeting the target condition based on the target condition and the basic driving trajectory to guide the driving operation of the target vehicle.

[0006] The present application also provides a driving trajectory generation device, comprising: an acquisition module configured to obtain an environment image of a target vehicle, extraction indication information of semantic features, and a pre-generated basic driving trajectory; a semantic feature extraction module configured to input the environment image and the extraction indication information of the semantic features into a pre-trained language model to obtain target semantic features output by the language model; an image feature extraction module configured to extract target image features from the environment image; The generating module is configured to generate a target condition based on the target image feature and the target semantic feature, and generate a target driving track meeting the target condition based on the target condition and the basic driving track, so as to guide the driving operation of the target vehicle.

[0007] The application further provides an electronic device, including a memory for storing a computer program, and a processor for executing the computer program to implement the steps of the driving track generation method.

[0008] The application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the driving track generation method.

[0009] The application further provides a computer program product, which includes a computer program, and the computer program is executed by a processor to implement the steps of the driving track generation method.

[0010] According to the application, after obtaining the environment image of the target vehicle, the extraction indication information of the semantic feature and the basic driving track, the environment image and the extraction indication information of the semantic feature are first input into the language model, so that the language model can extract the target semantic feature required for generating the driving track. Then, the target image feature is directly extracted from the environment image. Further, the target semantic feature and the target image feature can be used as the guiding condition for track generation, and the basic driving track is adjusted to obtain the target driving track meeting the target condition. In this way, in the process of generating the driving track, the real-time obtained environment image is used to generate the driving track, which can be applied to any road scene and has a wider application range. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the application, the drawings required in the embodiments will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 An architecture schematic diagram of a driving track generation system provided by an embodiment of the application; Figure 2 A flow schematic diagram of a driving track generation method provided by an embodiment of the application; Figure 3 A data flow schematic diagram of a language model provided by an embodiment of the application; Figure 4 A data flow schematic diagram of a target image feature generation provided by an embodiment of the application; Figure 5 A schematic diagram of a data flow of a diffusion model provided for an embodiment of the present application is shown in FIG. 1. Figure 6 A schematic diagram of a data flow of generating a target driving trajectory provided for an embodiment of the present application is shown in FIG. 2. Figure 7 A flowchart of a driving trajectory generation device provided for an embodiment of the present application is shown in FIG. 3. Figure 8 A structural schematic diagram of an electronic device provided for an embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0013] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0014] It should be noted that, in the description of the present application, the terms “comprise”, “contain” or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. The terms “first”, “second” and the like in the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0015] In order for those skilled in the art to better understand the technical solutions of the present application, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0016] The driving trajectory generation method provided by the present application can be realized by a driving trajectory generation system, as shown in FIG. 1. Figure 1 As shown in FIG. 1, the driving trajectory generation system can include an image acquisition device, a vehicle terminal, and a driving trajectory generation platform.

[0017] The image acquisition device can be a surround camera (including front view, surround view, rear view camera, etc.), radar, etc., used to acquire environmental images around the vehicle and transmit them to the vehicle terminal.

[0018] The vehicle terminal can be used to receive the environmental images sent by the image acquisition device and perform driving trajectory generation operations, or the received environmental images can also be sent to the driving trajectory generation platform.

[0019] The driving track generation platform can be a server, a server cluster, etc., and can be configured to generate a driving track based on an environment image sent by the vehicle terminal and send the generated driving track to the vehicle terminal.

[0020] Embodiments of the present application provide a driving track generation method, which can be executed by the vehicle terminal or the driving track generation platform described above (hereinafter, the vehicle terminal is taken as an example for description), as shown in Figure 2 The specific processing steps of the driving track generation method can include: In step S201, an environment image of a target vehicle, extraction indication information of semantic features, and a pre-generated basic driving track are obtained.

[0021] The environment image can include images of multiple angles, such as front view images, rear view images, side view images, etc. The images can include image elements such as pedestrians, sky, traffic lights, roads, etc.

[0022] The extraction indication information of semantic features can include extraction prompt information of semantic features of multiple semantic types, such as traffic congestion degree, time period (morning, noon, evening, night, etc.), weather condition (rain, sunny, etc.), traffic light state (red light, green light, yellow light), longitudinal driving intention (speed, acceleration), lateral driving intention (straight, left turn), etc. The extraction prompt information is used to guide the extraction process of semantic features. For example, “What is the weather condition?” “What is the traffic congestion degree?”

[0023] The basic driving track can be a driving track generated by adding noise to multiple typical driving tracks included in a trajectory anchor vocabulary (Trajectory Anchor Vocabulary) in advance, or can be a pre-constructed driving track, such as a straight driving track.

[0024] Specifically, the image acquisition device of the target vehicle can periodically acquire the environment image around the target vehicle and transmit it to the vehicle terminal. After receiving the environment image, the vehicle terminal can read the extraction prompt information of semantic features and the pre-generated basic driving track from a pre-set storage location (which can be a memory of itself or a driving track generation platform).

[0025] In step S202, the environment image and the extraction indication information of semantic features are input into a pre-trained language model to obtain target semantic features output by the language model.

[0026] The language model can include a Mixture of Experts (MoE) model and a Vision-Language Model (VLM). For example, the Vision-Language Model can be QwenVL2.5, InternVL3, etc.

[0027] Specifically, the vehicle terminal can input the extraction indication information of the environmental image and the semantic feature into the language model, and the language model can extract the target semantic feature from the environmental image according to the extraction indication information.

[0028] Step S203: Extracting a target image feature from the environmental image.

[0029] Specifically, the vehicle terminal can extract the target image feature from the environmental image by using a pre-trained image feature extraction model. The target image feature can include one or more types of image features, such as basic image features, bird's eye view features, map features, etc. The basic image features can be textures, colors, etc.

[0030] Step S204: Generating a target condition based on the target image feature and the target semantic feature.

[0031] Specifically, the vehicle terminal can splice or aggregate the target image feature and the target semantic feature to obtain the target condition. The target condition can be in the form of a vector, a matrix, etc.

[0032] Step S205: Generating a target driving trajectory that meets the target condition based on the target condition and a basic driving trajectory, to guide the driving operation of the target vehicle.

[0033] Specifically, the vehicle terminal can adjust the basic driving trajectory by taking the target image feature and the target semantic feature indicated in the target condition as a reference to obtain the target driving trajectory that meets the target condition. In this way, the vehicle terminal can display the target driving trajectory, so that the driver can perform the driving operation based on the target driving trajectory, or the driving process of the target vehicle can be directly controlled according to the target driving trajectory.

[0034] The method for generating a driving trajectory of an embodiment of the present application, after obtaining the environment image of the target vehicle, the extraction indication information of the semantic feature, and the basic driving trajectory, first inputs the environment image and the extraction indication information of the semantic feature into the language model, so that the language model can extract the target semantic feature required for generating the driving trajectory. Then, the target image feature is directly extracted from the environment image. Further, the target semantic feature and the target image feature can be used as a guiding condition for trajectory generation to adjust the basic driving trajectory to obtain the target driving trajectory that meets the target condition. In this way, in the process of generating the driving trajectory, the driving trajectory is generated by using the environment image obtained in real time, and the application range is wider.

[0035] In addition, due to the frequent changes of roads, for example, when there is temporary construction, traffic sign adjustment, newly repaired road, etc., the high-definition map may not be updated in time, resulting in inaccurate generated driving trajectory, and even serious safety problems. In addition, the high-definition map cannot inform the vehicle of dynamic information such as whether traffic congestion is occurring in front of the vehicle, whether there is temporary traffic control at a certain intersection, whether pedestrians are crossing the road, or whether an ambulance is approaching with a siren, etc., resulting in inaccurate generated driving trajectory and safety problems. However, the present scheme generates the driving trajectory based on the real-time captured environment image, and further, the driving decision based on the accurate driving trajectory can avoid the occurrence of such safety problems.

[0036] Due to the complex production process of the high-definition map, a professional high-precision surveying and mapping vehicle needs to be equipped for large-scale map data collection, and a large amount of manual labeling and correction is needed, which is costly. However, the present scheme does not need to maintain the high-definition map, and the cost is lower.

[0037] In some optional embodiments, as shown in Figure 3 The language model can include a first image encoder, a text encoder, a language model decoder (LM Decoder), and a semantic feature generator of multiple semantic types, wherein the first image encoder, the text encoder, and the language model decoder can be component models of the above-mentioned vision-language model. Correspondingly, in the above step S202, the vehicle terminal can specifically determine the target semantic feature by the following steps: Step one, input the environment image into the first image encoder to obtain the image feature output by the first image encoder.

[0038] Step two, input the extraction indication information of the semantic feature into the text encoder to obtain the text feature output by the text encoder.

[0039] Step three, input the image features and text features output by the first image encoder into the language model decoder to obtain fusion features output by the language model decoder.

[0040] Step four, input the fusion features into a plurality of semantic feature generators respectively to obtain semantic features corresponding to their own semantic types respectively output by the plurality of semantic feature generators.

[0041] Step five, splice the semantic features corresponding to the plurality of semantic types respectively to obtain target semantic features.

[0042] Among them, the first image encoder can also be referred to as a vision encoder (Vision Encoder). The encoder is usually a neural network structure (Transformer) architecture based on attention mechanism, for example, it can be a visual Transformer, abbreviated as ViT. The image features output by the first image encoder can be a basic image feature.

[0043] The semantic feature generators of the plurality of semantic types can all be expert mixtures (Mixture of Experts, MoE) output heads, including a traffic congestion output head, a time of day head, a weather condition head, a traffic light status head, a longitudinal action head, a latitudinal action head, etc. Among them, the traffic congestion output head can be used to analyze the vehicle density, vehicle speed, etc. in the image to determine the degree of traffic congestion. The time of day head can determine the current weather condition according to the visual clues such as light, raindrops, snowflakes or fog in the image. The traffic light status head can be used to identify the traffic light in the image and determine its current state. The longitudinal action head can be used to infer the longitudinal driving intention of the vehicle, such as accelerating, decelerating or maintaining the current speed. The latitudinal action head can be used to infer the latitudinal driving intention of the vehicle, such as straight, left turn or right turn.

[0044] Specifically, in step one, the vehicle terminal can input the environment image into the first image encoder, and obtain the image features output by the first image encoder after the first image encoder encodes the environment image. Alternatively, when the environment image includes multiple environment sub-images at different angles, the vehicle terminal can input each environment sub-image into the first image encoder respectively, and obtain the sub-image features corresponding to each environment sub-image output by the first image encoder. Then, the vehicle terminal can splice all the sub-image features to obtain the image features corresponding to the environment image, i.e. the image features output by the first image encoder.

[0045] In step two, the vehicle terminal can input the extraction indication information of the semantic features into the text encoder, and obtain the text features after the text encoder encodes the extraction indication information. These text features encode the semantic information of the question, and can provide guidance for the subsequent language model decoder.

[0046] In step three, the vehicle terminal can splice the text features and the image features determined in step one and input them into the language model decoder, for example, splice the text features at the back of the image features. The language model decoder can extract and fuse the image information related to the text features in the image features according to the text features, and obtain the fusion features output by the language model decoder. The fusion features can include the information corresponding to all semantic types indicated in the extraction indication information.

[0047] In step four, the vehicle terminal can input the fusion features into multiple semantic feature generators respectively, and obtain the semantic features corresponding to the semantic types of the semantic feature generators output by the semantic feature generators respectively. For example, the traffic congestion output head can output the traffic congestion degree feature, the time output head can input the time period feature, the weather condition output head can input the weather condition feature, the traffic signal light state output head can output the traffic signal light state feature, the longitudinal action output head can output the longitudinal action feature of the target vehicle, and the lateral action output head can output the lateral action feature of the target vehicle.

[0048] In step five, the vehicle terminal can splice all the semantic features to obtain the target semantic features including the semantic information of multiple semantic types.

[0049] Generally, it is difficult to comprehensively and accurately perceive a complex road environment based on a model of a convolutional neural network or a Transformer alone. For example, in poor weather conditions such as insufficient light at night, rain, snow, and fog, or in the presence of occlusion and reflection, the image quality decreases, making it difficult for the model to accurately identify the positions of road boundaries, lane lines, drivable areas, and dynamic obstacles, resulting in a large error in the generated driving trajectory. However, the present scheme uses the extracted semantic features as guidance to extract various types of key semantic information from the environment image, obtain target semantic features, and then generate a target driving trajectory using the target semantic features containing rich semantic information (for example, weather conditions, time period, traffic congestion, etc.), thereby improving the accuracy of the generated driving trajectory in a map-free scenario.

[0050] In some optional embodiments, in step S203 described above, the vehicle terminal can sequentially extract image features of different dimensions, such as basic image features, bird's eye view features, and map features, from the environment image. The environment image can include environment sub-images corresponding to different shooting angles. Accordingly, as shown in FIG. 2, the vehicle terminal can specifically extract target image features from the environment image by the following steps: Figure 4 Step one, input each environment sub-image into a pre-constructed second image encoder to obtain sub-image features corresponding to each environment sub-image output by the second image encoder.

[0051] Wherein, the sub-image features corresponding to all environment sub-images constitute the basic image features.

[0052] Step two, input the sub-image features corresponding to each environment sub-image into a pre-constructed view conversion model to obtain bird's eye view features output by the view conversion model.

[0053] Step three, input the bird's eye view features into a pre-constructed mapping model to obtain map features output by the mapping model.

[0054] Wherein, the sub-image features corresponding to all environment sub-images, the bird's eye view features, and the map features constitute the target image features.

[0055] Wherein, the second image encoder can be an image encoder based on a residual network (Residual Network, ResNet) architecture. The view conversion model can be a bird's eye view perception model (Bird's Eye View, BEV), such as BEVFormer, PETRv2, BEVDepth, etc. The mapping (Map) model can be MapTR, MapTRv2, StreamMapNet, etc. ​

[0056] Specifically, the vehicle terminal can input each of the environment sub-images into the pre-constructed second image encoder respectively, to obtain the sub-image features corresponding to each of the environment sub-images output by the second image encoder. Further, the vehicle terminal can input the sub-image features corresponding to each of the environment sub-images into the view conversion model, and obtain the bird's eye view features after the view conversion operation of the view conversion model. The bird's eye view features contain the depth, semantic, geometric information and the like of the environment. Finally, the vehicle terminal can input the bird's eye view features into the pre-constructed mapping model, to obtain the map features output by the mapping model.

[0057] In this way, by extracting image features from different dimensions, the target image features can include basic image information, overhead information, and map information, which can generate a more accurate target driving trajectory when used as a target condition subsequently. In addition, the map features are extracted based on the real-time captured environment images, so that the map features consistent with the actual situation can be used to guide the generation process of the driving trajectory subsequently, so that the generated driving trajectory is more accurate and the occurrence of safety problems is avoided.

[0058] In some optional embodiments, the mapping model described above can include a queryer and a third image encoder. Accordingly, in step three of step S203 described above, the vehicle terminal can obtain the map features based on the bird's eye view features by using the following steps: Step one, input the bird's eye view features into the queryer to obtain the structured information corresponding to each of the at least one map elements output by the queryer.

[0059] Step two, input the structured information of the target map element into the third image encoder to obtain the graph encoding vector corresponding to the target map element output by the third image encoder.

[0060] Step three, after determining the graph encoding vector corresponding to each of the at least one map elements, construct the map features according to the graph encoding vector corresponding to each of the at least one map elements.

[0061] The map elements can be lane lines, road boundaries, drivable areas, obstacles, etc. The structured information can be a series of coordinate points. For example, a lane line can be represented as an ordered two-dimensional point sequence L1 = {(x1, y1), (x2, y2),..., (xn, yn)}, which defines the shape and position of the lane line. The road boundary can be represented as a series of coordinate points RB1 = {(x'1, y'1), (x'2, y'2),..., (xm, y'm)}. The drivable area is usually modeled as a binary semantic segmentation mask, i.e., a two-dimensional matrix with the same size as the BEV feature, where each pixel value represents whether the position belongs to the drivable area. For example, a pixel value of 0 indicates that the position belongs to the non-drivable area, and a pixel value of 1 indicates that the position belongs to the drivable area. The structured information of an obstacle (such as another vehicle) can be represented by its center point coordinate (xc, yc) and orientation angle (θ) in the BEV space Obstacle1 = (xc, yc, θ).

[0062] Specifically, the vehicle terminal can input the bird's eye view feature into the queryer, which can use its own query mechanism to predict at least one map element in parallel and determine the structured information of each map element. Further, the vehicle terminal can input the structured information of each map element into the third image encoder respectively, so that the third image encoder can use a preset encoding method (for example, the VectorNet method) to perform encoding operation on each map element respectively, obtain the graph encoding vector (also commonly referred to as subgraph encoding) corresponding to each map element, and then use the self-attention mechanism to capture the relationship between nodes, and then construct the map feature containing all map elements based on the relationship between nodes.

[0063] In some optional embodiments, in the step S201, the vehicle terminal can specifically generate the basic driving trajectory by using the following steps, including: Step one, obtaining a plurality of driving trajectories.

[0064] Step two, obtaining the basic driving trajectory by performing a noise adding operation on each of the plurality of driving trajectories.

[0065] Specifically, the plurality of driving trajectories can be driving trajectories in the trajectory anchor point vocabulary, and the vehicle terminal can add noise to each of the plurality of driving trajectories after obtaining the plurality of driving trajectories, and the plurality of driving trajectories after adding noise constitute the basic driving trajectory.

[0066] In some optional embodiments, in the case that the basic driving trajectory is generated by adding noise to a plurality of typical driving trajectories included in the trajectory anchor point vocabulary in advance, the vehicle terminal can specifically perform the following steps on the plurality of driving trajectories respectively after adding noise to obtain the basic driving trajectory, including: Step 1, in the current noise adding round, based on the second preset variance and the pre-constructed Gaussian distribution function, random sampling is performed to obtain the noise of the current noise adding round, and the driving trajectory to be processed in the current noise adding round is determined.

[0067] Step 2, the noise scheduling parameter and the signal retention ratio corresponding to the current noise adding round are extracted from the preset parameter sequence.

[0068] Step 3, based on the noise of the current noise adding round, and the noise scheduling parameter and the signal retention ratio corresponding to the current noise adding round, noise operation is performed on the driving trajectory to be processed to obtain the driving trajectory after noise adding in the current noise adding round.

[0069] Step 4, when it is determined that the current noise adding round is the last noise adding round, the noise adding process is stopped.

[0070] Alternatively, step 5, when it is determined that the current noise adding round is not the last noise adding round, the next noise adding round is entered until the noise adding process of the last noise adding round is completed, and then the noise adding process is stopped.

[0071] Wherein, when the current noise adding round is the first noise adding round, the driving trajectory to be processed is the first driving trajectory, and the first driving trajectory is any one of the plurality of driving trajectories, or when the current noise adding round is not the first noise adding round, the driving trajectory to be processed is the second driving trajectory, and the second driving trajectory is the driving trajectory obtained after the noise operation of the historical noise adding round before the current noise adding round on the first driving trajectory. The pre-constructed Gaussian distribution function can be a standard Gaussian distribution function, that is, the mean of the Gaussian distribution function is 1 and the variance is 1. The plurality of driving trajectories after noise adding determined by the last noise adding round constitute the basic driving trajectory.

[0072] Specifically, the vehicle terminal can use Markov Chain to perform noise operation on each driving trajectory in the trajectory anchor point vocabulary for a preset number of noise adding rounds.

[0073] For example, in the first driving trajectory, in the current noise adding round, the vehicle terminal can input the second preset variance into the Gaussian distribution function, and obtain the noise added in the current noise adding round after performing the sampling operation. When the current noise adding round is the first noise adding round, the vehicle terminal can determine the first driving trajectory as the driving trajectory to be processed in the current noise adding round. Alternatively, when the current noise adding round is not the first noise adding round, the vehicle terminal can determine the second driving trajectory as the driving trajectory to be processed in the current noise adding round. The second driving trajectory has been processed in one or more noise adding rounds.

[0074] In addition, the vehicle terminal can pre-store a preset parameter sequence, and the preset parameter sequence can include noise scheduling parameters and signal reservation ratios corresponding to a preset number of noise adding rounds. The vehicle terminal can extract the noise scheduling parameters and the signal reservation ratios corresponding to the order of the current noise adding round from the preset parameter sequence, that is, extract the noise scheduling parameters and the signal reservation ratios corresponding to the current noise adding round. Since the noise scheduling parameters can indicate the noise adding intensity in the noise adding process, the signal reservation ratio can indicate the proportion of the original signal (that is, the driving trajectory to be processed), and the noise added noise can indicate the noise amount, the vehicle terminal can perform the noise adding operation on the driving trajectory to be processed based on the noise added noise in the current noise adding round and the noise scheduling parameters and the signal reservation ratio corresponding to the current noise adding round, to obtain the driving trajectory after noise adding in the current noise adding round.

[0075] When it is determined that the order of the current noise adding round is equal to the preset order (equal to the preset number), it is determined that the current noise adding round is the last noise adding round, and at this time, the noise adding process can be stopped. Alternatively, when it is determined that the order of the current noise adding round is less than the preset order, the next noise adding round is entered, and the noise adding process is stopped after completing the noise adding process of the last noise adding round.

[0076] For each driving trajectory in the trajectory anchor point vocabulary, a similar manner can be used for the preset number of noise adding rounds, so that the driving trajectories after noise adding determined by the last noise adding round constitute the basic driving trajectory. As shown in the noise adding process of Figure 5 , the plurality of driving trajectories generate the basic driving trajectory including more noise after noise adding.

[0077] For example, the above step 3 can use the following expression: (1) wherein, is the driving trajectory after noise adding, is the noise added in the current noise adding round, is the driving trajectory to be processed (that is, the driving trajectory after noise adding in the last noise adding round), a signal reservation ratio for the current noise adding round, a noise scheduling parameter for the current noise adding round, a cumulative product of the signal reservation ratios for the current noise adding round and the historical noise adding rounds before the current noise adding round, a signal reservation ratio for the s2th noise adding round.

[0078] In some optional embodiments, in the case where the base driving trajectory is a driving trajectory generated by adding noise to a plurality of typical driving trajectories included in a trajectory anchor point vocabulary in advance, in step S205 described above, the vehicle terminal can specifically generate the target driving trajectory by performing the following steps on the base driving trajectory after denoising processing: Step one, in the current denoising round, determine the initial driving trajectory of the current denoising round.

[0079] Step two, according to the order of the current denoising round, the initial driving trajectory of the current denoising round, and the target condition, determine the final driving trajectory of the current denoising round.

[0080] Step three, when it is determined that the current denoising round is the last denoising round, stop the denoising processing, and determine the final driving trajectory determined by the last denoising round as the target driving trajectory.

[0081] Alternatively, step four, when it is determined that the current denoising round is not the last denoising round, enter the processing of the next denoising round until the denoising processing of the last denoising round is completed, then stop the denoising processing, and determine the final driving trajectory determined by the last denoising round as the target driving trajectory.

[0082] Wherein, when the current denoising round is the first denoising round, the initial driving trajectory is the base driving trajectory, or when the current denoising round is not the first denoising round, the initial driving trajectory is the final driving trajectory determined by the last denoising round before the current denoising round.

[0083] Specifically, the vehicle terminal can use a pre-constructed diffusion model (DiffusionModel) to perform denoising operation on the base driving trajectory for a preset number of denoising rounds. For example, the diffusion model can be DiffusionDrive, HydraMDP, etc.

[0084] ​In the first denoising round, the vehicle terminal can directly determine the basic driving trajectory as the initial driving trajectory. In the non-first denoising round, the vehicle terminal can determine the final driving trajectory generated after the denoising processing of the last denoising round as the initial driving trajectory of the current denoising round. Since the number of denoising processes affects the denoising process, the order of the current denoising round can also be used as a reference condition, so that the initial driving trajectory of the current denoising round can be adjusted based on the order of the current denoising round and the target condition, and the final driving trajectory of the current denoising round can be obtained after the adjustment. After determining that the order of the denoising process is equal to the preset number, it can be determined that the current denoising round is the last denoising round, at which time the denoising process can be stopped, and the final driving trajectory determined by the last denoising round is determined as the target driving trajectory. Alternatively, when it is determined that the order of the denoising process is less than the preset number, it can be determined that the current denoising round is not the last denoising round, and the next denoising round is processed until the denoising process of the last denoising round is completed, and then the denoising process is stopped, and the final driving trajectory determined by the last denoising round is determined as the target driving trajectory. As shown in the denoising process of FIG. 8, the basic driving trajectory full of noise is denoised to obtain a clear target driving trajectory. Figure 5

[0085] In this way, the denoising process of each step is based on the denoising result of the last step, and the denoising process of each step introduces a target condition, which can ensure that each step of optimization can be performed in the direction of meeting the driving scene, so that the generated target driving trajectory is more accurate and meets the actual situation. For example, the target condition indicates that the vehicle is turning, and the noise that causes the trajectory to be straight can be removed in the denoising process, thereby guiding the trajectory to develop in the correct curved direction. For another example, the target condition indicates that the weather condition is rainy and the traffic is congested, and a trajectory with low speed and more stability is generated. For another example, the target condition indicates that the current road section is an open road section and the driving intention is to overtake, and a more aggressive and faster lane-changing trajectory can be generated.

[0086] In some optional embodiments, in step two of step S205, the vehicle terminal can use the following specific steps to determine the final driving trajectory of the current denoising round based on the order of the current denoising round, the initial driving trajectory of the current denoising round, and the target condition: Step 1: input the order of the current denoising round, the initial driving trajectory of the current denoising round, and the target condition into the pre-constructed noise prediction model to obtain the predicted noise removal corresponding to the current denoising round output by the noise prediction model.

[0087] Step 2: extract the noise scheduling parameter and signal retention ratio corresponding to the current denoising round from the preset parameter sequence.

[0088] ​Step 3, determining an average driving trajectory of the current denoising round according to the initial driving trajectory of the current denoising round, the predicted removed noise, the noise scheduling parameter, and the signal reservation ratio.

[0089] Step 4, obtaining a random noise corresponding to the current denoising round.

[0090] Step 5, generating a final driving trajectory of the current denoising round according to the average driving trajectory, the first preset variance, and the random noise.

[0091] Specifically, the vehicle terminal can input the order of the current denoising round, the initial driving trajectory of the current denoising round, and the target condition into the pre-constructed noise prediction model (which can be a component of the diffusion model described above, or also referred to as a denoising network in the diffusion model) to obtain the predicted removed noise corresponding to the current denoising round output by the noise prediction model. The preset parameter sequence can include noise scheduling parameters and signal reservation ratios corresponding to a preset number of denoising rounds. Then, the vehicle terminal can extract the noise scheduling parameter and the signal reservation ratio corresponding to the order of the current denoising round from the preset parameter sequence according to the order of the current denoising round, and determine the noise scheduling parameter and the signal reservation ratio corresponding to the current denoising round. In the denoising process, the noise scheduling parameter can indicate the noise intensity eliminated in the denoising process. Further, the vehicle terminal can perform denoising processing on the initial driving trajectory of the current denoising round according to the predicted removed noise, the noise scheduling parameter, and the signal reservation ratio to obtain the average driving trajectory of the current denoising round. In addition, the vehicle terminal can randomly sample a noise from a standard Gaussian noise to obtain a random noise corresponding to the current denoising round. Finally, the vehicle terminal can adjust the average driving trajectory according to the first preset variance and the random noise to obtain the final driving trajectory of the current denoising round.

[0092] In this way, in each denoising round, the predicted removed noise is determined by the target condition, the order of the current denoising round, and the initial driving trajectory of the current denoising round, which can make the denoising direction consistent with the target condition. Furthermore, the historical denoising information (the order of the current denoising round and the initial driving trajectory of the current denoising round) is used as a reference, which can make the predicted removed noise take into account the characteristics of the historical denoising operation, so that the final driving trajectory after removing noise based on the predicted removed noise conforms to the actual scene, is more accurate, and avoids various safety problems.

[0093] For example, step 3 can use the following expression: (2) wherein, is the initial driving trajectory of the current denoising round, t1 is the current denoising round, is the average driving trajectory of the current denoising round, a signal reservation ratio for the current denoising round, a noise scheduling parameter for the current denoising round, for predicting denoising, a cumulative product of the signal reservation ratio for the current denoising round and the signal reservation ratio for a historical denoising round before the current denoising round, , a signal reservation ratio for the s1th denoising round.

[0094] Step 5 can adopt the following expression: (3) wherein, a final driving trajectory for the current denoising round, an average driving trajectory, a first preset variance, a random noise.

[0095] In this way, the random noise can simulate the uncertainty factor in the actual traffic environment, and by adding the random noise to the average driving trajectory, the generated final driving trajectory can be more realistic and accurate.

[0096] On the basis of the above embodiment, as shown in Figure 6 the vehicle terminal inputs the environment image into the second image encoder to obtain the base image feature output by the second image encoder, inputs the base image feature into the view conversion model to obtain the bird's eye view feature. In addition, the vehicle terminal can input the extraction prompt information of the environment image and the semantic feature into the language model to obtain the target semantic feature. Finally, the vehicle terminal can input the target semantic feature, the bird's eye view feature, the map feature, and the trajectory anchor vocabulary into the diffusion model to obtain the target driving trajectory.

[0097] It should be noted that the letters in the above formula (1) to formula (3) are all dimensionless vectors or matrices, etc.

[0098] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment.

[0099] The embodiments of the present application also provide a driving trajectory generation device, as shown in Figure 7 comprising: The acquisition module 710 is configured to acquire an environment image of a target vehicle, extraction indication information of a semantic feature, and a pre-generated base driving trajectory. The semantic feature extraction module 720 is configured to input the environment image and the semantic feature extraction indication information into the pre-trained language model to obtain target semantic features output by the language model. The image feature extraction module 730 is configured to extract target image features from the environment image. The generation module 740 is configured to generate a target condition based on the target image features and the target semantic features, and generate a target driving trajectory meeting the target condition based on the target condition and the basic driving trajectory, so as to guide the driving operation of the target vehicle.

[0100] In some optional embodiments, the generation module 740 is specifically configured to: In the current denoising round, an initial driving trajectory of the current denoising round is determined, wherein when the current denoising round is the first denoising round, the initial driving trajectory is the basic driving trajectory, or when the current denoising round is a non-first denoising round, the initial driving trajectory is a final driving trajectory determined by a previous denoising round before the current denoising round; According to the order of the current denoising round, the initial driving trajectory of the current denoising round, and the target condition, a final driving trajectory of the current denoising round is determined; When it is determined that the current denoising round is the last denoising round, the denoising process is stopped, and the final driving trajectory determined by the last denoising round is determined as the target driving trajectory; Or, when it is determined that the current denoising round is not the last denoising round, the processing of the next denoising round is entered, until the denoising process of the last denoising round is completed, and then the denoising process is stopped, and the final driving trajectory determined by the last denoising round is determined as the target driving trajectory.

[0101] In some optional embodiments, the generation module 740 is specifically configured to: The order of the current denoising round, the initial driving trajectory of the current denoising round, and the target condition are input into a pre-constructed noise prediction model to obtain predicted removed noise corresponding to the current denoising round output by the noise prediction model; The noise scheduling parameter and the signal retention ratio corresponding to the current denoising round are extracted from a preset parameter sequence; According to the initial driving trajectory of the current denoising round, the predicted removed noise, the noise scheduling parameter, and the signal retention ratio, an average driving trajectory of the current denoising round is determined; Random noise corresponding to the current denoising round is obtained; According to the average driving trajectory, the first preset variance, and the random noise, a final driving trajectory of the current denoising round is generated.

[0102] In some optional embodiments, the average driving track of the current denoising round is determined according to the initial driving track of the current denoising round, the predicted removed noise, the noise scheduling parameter, and the signal reservation ratio, and the following expression is used:

[0103] wherein, is the initial driving track of the current denoising round, t1 is the current denoising round, is the average driving track of the current denoising round, is the signal reservation ratio of the current denoising round, is the noise scheduling parameter of the current denoising round, is the predicted removed noise, is the cumulative product of the signal reservation ratios of the current denoising round and the historical denoising rounds before the current denoising round.

[0104] In some optional embodiments, the final driving track of the current denoising round is generated according to the average driving track, the first preset variance, and the random noise, and the following expression is used:

[0105] wherein, is the final driving track of the current denoising round, is the average driving track, is the first preset variance, is the random noise.

[0106] In some optional embodiments, the pre-generated basic driving track is obtained, including: a plurality of driving tracks are obtained; after performing the noise adding operation on the plurality of driving tracks respectively, the basic driving track is obtained.

[0107] In some optional embodiments, the generation module 740 is specifically configured to: in the current noise adding round, random sampling is performed based on the second preset variance and the pre-constructed Gaussian distribution function to obtain the noise adding noise of the current noise adding round, and a to-be-processed driving track of the current noise adding round is determined, wherein when the current noise adding round is the first noise adding round, the to-be-processed driving track is the first driving track, the first driving track is any one of the plurality of driving tracks, or when the current noise adding round is a non-first noise adding round, the to-be-processed driving track is the second driving track, the second driving track is a driving track obtained after the first driving track is subjected to the noise adding operation of the historical noise adding round before the current noise adding round; the noise scheduling parameter and the signal reservation ratio corresponding to the current noise adding round are extracted from the preset parameter sequence; performing a noise adding operation on the to-be-processed driving track based on the noise adding noise of the current noise adding round and the noise scheduling parameter and the signal reservation ratio corresponding to the current noise adding round, to obtain the driving track after noise adding in the current noise adding round; When it is determined that the current noise adding round is the last noise adding round, stop the noise adding processing; Or, when it is determined that the current noise adding round is not the last noise adding round, enter the next noise adding round, until the noise adding processing of the last noise adding round is completed, and then stop the noise adding processing; The plurality of driving tracks after noise adding determined by the last noise adding round constitute the basic driving track.

[0108] In some optional embodiments, the noise adding operation on the to-be-processed driving track based on the noise adding noise of the current noise adding round and the noise scheduling parameter and the signal reservation ratio corresponding to the current noise adding round is performed to obtain the driving track after noise adding in the current noise adding round, and the following expression is used:

[0109] wherein, is the driving track after noise adding, is the noise adding noise of the current noise adding round, is the to-be-processed driving track, is the signal reservation ratio of the current noise adding round, is the noise scheduling parameter of the current noise adding round, is the cumulative product of the signal reservation ratios of the current noise adding round and the historical noise adding round before the current noise adding round.

[0110] In some optional embodiments, the language model comprises a first image encoder, a text encoder, a language model decoder, and a plurality of semantic feature generators of different semantic types; and the semantic feature extraction module 720 is specifically configured to: input the environmental image into the first image encoder to obtain image features output by the first image encoder; input the extraction instruction information of the semantic features into the text encoder to obtain text features output by the text encoder; input the image features output by the first image encoder and the text features into the language model decoder to obtain fusion features output by the language model decoder; input the fusion features into the plurality of semantic feature generators respectively to obtain semantic features corresponding to the semantic types of the plurality of semantic feature generators respectively and output by the plurality of semantic feature generators respectively; splice the semantic features corresponding to the plurality of semantic types respectively to obtain the target semantic features.

[0111] The features of the embodiments of the device for generating a driving trajectory can be referred to the related descriptions of the embodiments of the method for generating a driving trajectory, which will not be repeated here.

[0112] The embodiments of the present application also provide an electronic device, as shown in the figure, comprising a memory 10 and a processor 20, the memory 10 stores a computer program, and the processor 20 is configured to run the computer program to perform the steps in any of the above-mentioned embodiments of the method for generating a driving trajectory. Figure 8

[0113] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, wherein the computer program is configured to perform the steps in any of the above-mentioned embodiments of the method for generating a driving trajectory when running.

[0114] In an example embodiment, the above-mentioned computer readable storage medium can include but is not limited to: a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store computer programs.

[0115] The embodiments of the present application also provide a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the method for generating a driving trajectory.

[0116] The embodiments of the present application also provide another computer program product, which comprises a non-volatile computer readable storage medium, and the non-volatile computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps in any of the above-mentioned embodiments of the method for generating a driving trajectory.

[0117] The skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0118] ​The above introduces in detail a driving track generation method, device, electronic device, storage medium and program product provided by the present application. The principles and implementation manners of the present application are described by applying specific examples, and the above example is only used to help understand the method of the present application and its core idea. It should be pointed out that, for ordinary skilled persons in the technical field, some improvements and modifications can be made to the present application without departing from the principles of the present application, and these improvements and modifications also fall within the protection scope of the present application.

Claims

1. A method of generating a travel trajectory, characterized by, The method comprises the following steps: acquiring an environment image of a target vehicle, extraction indication information of semantic features, and a pre-generated basic driving track; inputting the environment image and the extraction indication information of the semantic features into a pre-trained language model to obtain target semantic features output by the language model; extracting target image features from the environment image; generating a target condition based on the target image features and the target semantic features; generating a target driving track that meets the target condition based on the target condition and the basic driving track, to guide the driving operation of the target vehicle.

2. The travel trajectory generation method according to claim 1, characterized by, The method of generating a target driving track that meets the target condition based on the target condition and the basic driving track comprises the following steps: in a current denoising round, determining an initial driving track of the current denoising round, wherein when the current denoising round is the first denoising round, the initial driving track is the basic driving track, or when the current denoising round is a non-first denoising round, the initial driving track is a final driving track determined by a previous denoising round before the current denoising round; determining a final driving track of the current denoising round according to the sequence of the current denoising round, the initial driving track of the current denoising round, and the target condition; when it is determined that the current denoising round is the last denoising round, stopping the denoising process and determining the final driving track determined by the last denoising round as the target driving track; or, when it is determined that the current denoising round is not the last denoising round, entering the processing of a next denoising round until the denoising process of the last denoising round is completed, then stopping the denoising process and determining the final driving track determined by the last denoising round as the target driving track.

3. The travel trajectory generation method according to claim 2, characterized by, The method of determining a final driving track of the current denoising round according to the sequence of the current denoising round, the initial driving track of the current denoising round, and the target condition comprises the following steps: inputting the sequence of the current denoising round, the initial driving track of the current denoising round, and the target condition into a pre-constructed noise prediction model to obtain predicted removed noise corresponding to the current denoising round output by the noise prediction model; extracting noise scheduling parameters and a signal retention ratio corresponding to the current denoising round from a preset parameter sequence; determining an average driving track of the current denoising round according to the initial driving track of the current denoising round, the predicted removed noise, the noise scheduling parameters, and the signal retention ratio; acquiring random noise corresponding to the current denoising round; generating a final driving track of the current denoising round according to the average driving track, a first preset variance, and the random noise.

4. The travel trajectory generation method according to claim 3, characterized by, The method of determining an average driving track of the current denoising round according to the initial driving track of the current denoising round, the predicted removed noise, the noise scheduling parameters, and the signal retention ratio adopts the following expression: wherein, is an initial driving trajectory of the current denoising round, t1 is the current denoising round, is an average driving trajectory of the current denoising round, is a signal reservation ratio of the current denoising round, is the noise scheduling parameter of the current denoising round, is the predicted noise removal, is a cumulative product of the signal reservation ratios of the current denoising round and a historical denoising round before the current denoising round.

5. The travel trajectory generation method according to claim 3, characterized by, The final driving track of the current denoising round is generated according to the average driving track, the first preset variance, and the random noise, and the following expression is used: wherein, is the final driving trajectory of the current denoising round, is the average driving trajectory, is the first preset variance, is the random noise.

6. The travel trajectory generating method according to any one of claims 3 to 5, characterized by, The pre-generated basic driving track is obtained, including: Obtaining a plurality of driving tracks; After performing a noise adding operation on each of the plurality of driving tracks, the basic driving track is obtained.

7. The travel trajectory generation method according to claim 6, characterized by, After performing a noise adding operation on each of the plurality of driving tracks, the basic driving track is obtained. In the current noise adding round, random sampling is performed based on a second preset variance and a pre-constructed Gaussian distribution function to obtain noise of the current noise adding round, and a to-be-processed driving track of the current noise adding round is determined, wherein when the current noise adding round is a first noise adding round, the to-be-processed driving track is a first driving track, and the first driving track is any one of the plurality of driving tracks, or when the current noise adding round is a non-first noise adding round, the to-be-processed driving track is a second driving track, and the second driving track is a driving track obtained after a noise adding operation on the first driving track in a historical noise adding round before the current noise adding round; From the preset parameter sequence, noise scheduling parameters and signal retention ratios corresponding to the current noise adding round are extracted; Based on the noise of the current noise adding round and the noise scheduling parameters and the signal retention ratios corresponding to the current noise adding round, a noise adding operation is performed on the to-be-processed driving track to obtain a driving track after noise adding in the current noise adding round; When it is determined that the current noise adding round is the last noise adding round, noise adding processing is stopped; Or, when it is determined that the current noise adding round is not the last noise adding round, the next noise adding round is entered, and noise adding processing is stopped after the last noise adding round is completed. The plurality of driving tracks after noise adding determined by the last noise adding round constitute the basic driving track.

8. The travel trajectory generation method according to claim 7, characterized by, The noise adding operation performed on the to-be-processed driving track based on the noise of the current noise adding round and the noise scheduling parameters and the signal retention ratios corresponding to the current noise adding round to obtain the driving track after noise adding in the current noise adding round uses the following expression: wherein, is the current noisy driving track, is the noise of the current noisy driving track, is the driving track to be processed, is the signal reservation ratio of the current noisy driving track, is the noise scheduling parameter of the current noisy driving track, is the cumulative product of the signal reservation ratios of the current noisy driving track and the historical noisy driving track before the current noisy driving track.

9. The travel trajectory generation method according to any one of claims 1 to 5, characterized by The language model includes a first image encoder, a text encoder, a language model decoder, and a semantic feature generator of a plurality of semantic types; The language model includes a first image encoder, a text encoder, a language model decoder, and a semantic feature generator of a plurality of semantic types; The image feature output by the first image encoder is obtained by inputting the environment image into the first image encoder; The text feature output by the text encoder is obtained by inputting the extraction instruction information of the semantic feature into the text encoder; The fusion feature output by the language model decoder is obtained by inputting the image feature output by the first image encoder and the text feature into the language model decoder; and The fusion features are respectively input into a plurality of semantic feature generators to obtain semantic features corresponding to semantic types of the semantic feature generators respectively output by the semantic feature generators; The semantic features corresponding to the semantic types are spliced to obtain the target semantic feature.

10. An electronic device, comprising: The method comprises the following steps: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the method for generating a driving track according to any one of claims 1 to 9.

Images