Method and device for generating aerial view

Abstract

The invention discloses a method and a device for generating an aerial view. A pre-constructed instruction-track data set is obtained. Then, a multi-frame aerial view feature map corresponding to the vehicle control instruction is generated; and finally, inputting the multiple frames of aerial view feature maps into a feature fusion module, so that the feature fusion module performs feature alignment on the first frame of aerial view feature map based on a self-vehicle motion conversion matrix to obtain an aligned first frame of aerial view feature map, and performs feature fusion on the aligned first frame of aerial view feature map and the second frame of aerial view feature map to obtain a second frame of aerial view feature map. And generating a second frame of fused aerial view feature map, continuously inputting the second frame of fused aerial view feature map into the feature fusion module to execute the same action, and so on until the last frame of fused aerial view feature map is generated. Therefore, feature fusion is performed on the multiple frames of aerial view feature maps through the real-time positions of the autonomous vehicle in different tracks, so that the fused aerial view feature maps can more comprehensively capture the change of the surrounding environment of the autonomous vehicle.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more particularly to a method and device for generating a bird's-eye view. Background Art

[0002] Currently, autonomous vehicles can generate bird's-eye view feature maps through sensor data in the nuScenes dataset to improve the vehicle's environmental perception capabilities.

[0003] However, the sensor data in the nuScenes dataset is collected when the vehicle is following a predetermined or natural driving trajectory. This means that the generated bird's-eye view feature map can reflect the environmental characteristics under a specific trajectory. However, this data collection method that relies on a fixed trajectory also means that the resulting bird's-eye view feature map has certain limitations in generalization ability. As a result, the accuracy of the decision made by the autonomous vehicle when facing complex or unknown environments is reduced. Summary of the invention

[0004] This specification provides a method, device, storage medium and electronic device for generating a bird's-eye view to improve the generalization ability of the bird's-eye view feature map, thereby improving the accuracy of the decision made by the autonomous driving vehicle when facing a complex or unknown environment.

[0005] This manual adopts the following technical solutions:

[0006] This specification provides a method for generating a bird's-eye view, the method comprising:

[0007] Acquire a pre-constructed instruction-trajectory data set; the instruction-trajectory data set includes at least one instruction-trajectory pair; the instruction-trajectory pair consists of a vehicle control instruction and a driving trajectory corresponding to the vehicle control instruction; wherein the vehicle control instruction is an instruction for controlling the autonomous driving vehicle to perform a specified driving action;

[0008] Based on the command-trajectory data set, generating a plurality of frames of bird's-eye view feature maps corresponding to at least one vehicle control command;

[0009] The multiple frames of bird's-eye view feature images are input into a feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the bird's-eye view feature image in the multiple frames of the bird's-eye view feature image based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye view feature image, and performs feature fusion on the aligned first frame of the bird's-eye view feature image and the second frame of the bird's-eye view feature image to generate a second frame of fused bird's-eye view feature image, and the second frame of fused bird's-eye view feature image is continuously input into the feature fusion module to perform the same action, and so on, until the last frame of fused bird's-eye view feature image in the multiple frames of the bird's-eye view feature image is generated.

[0010] Optionally, before acquiring the pre-built instruction-trajectory dataset, the method further includes:

[0011] Obtaining an autonomous driving data set to be labeled and at least one vehicle control instruction;

[0012] constructing at least one command-trajectory pair based on the at least one vehicle control command and the driving trajectory corresponding to the at least one vehicle control command;

[0013] Based on the at least one instruction-trajectory pair, the autonomous driving dataset is labeled to construct an instruction-trajectory dataset.

[0014] Optionally, before obtaining the autonomous driving data set to be labeled and at least one vehicle control instruction, the method further includes:

[0015] Get the audio data input by the user;

[0016] Based on a preset audio recognition algorithm, the audio data is recognized to obtain text data corresponding to the audio data;

[0017] The text data is input into a pre-trained large language model to determine a vehicle control instruction corresponding to the text data.

[0018] Optionally, inputting the text data into a pre-trained large language model to determine a vehicle control instruction corresponding to the text data includes:

[0019] Inputting the text data into a pre-trained large language model to determine keywords corresponding to the text data;

[0020] The keyword is post-processed to determine the vehicle control instruction corresponding to the keyword; wherein the post-processing at least includes: screening, sorting, and formatting.

[0021] Optionally, the driving trajectory is composed of a set number of trajectory points, and the trajectory points include coordinates and heading angles of the autonomous driving vehicle.

[0022] Optionally, generating a multi-frame bird's-eye view feature map based on the instruction-trajectory dataset includes:

[0023] Performing feature extraction on the command-trajectory data set to obtain a multi-frame feature map;

[0024] The multi-frame feature maps are perspective transformed to obtain multi-frame bird's-eye view feature maps.

[0025] Optionally, the autonomous driving dataset includes: a nuScenes dataset, and the feature fusion module adopts the temporal architecture of BEVFormer.

[0026] This specification provides a device for generating a bird's-eye view, the device comprising:

[0027] An acquisition module, used to acquire a pre-constructed instruction-trajectory data set; the instruction-trajectory data set includes at least one instruction-trajectory pair; the instruction-trajectory pair consists of a vehicle control instruction and a driving trajectory corresponding to the vehicle control instruction; wherein the vehicle control instruction is an instruction for controlling the autonomous driving vehicle to perform a specified driving action;

[0028] A generating module, configured to generate a plurality of frames of bird's-eye view feature images corresponding to the at least one vehicle control instruction based on the instruction-trajectory data set;

[0029] A fusion module is used to input the multi-frame bird's-eye feature map into a feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the multi-frame bird's-eye feature map based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye feature map, and performs feature fusion on the aligned first frame of the bird's-eye feature map and the second frame of the bird's-eye feature map to generate a second frame of fused bird's-eye feature map, and continues to input the second frame of fused bird's-eye feature map into the feature fusion module to perform the same action, and so on, until the last frame of fused bird's-eye feature map of the multi-frame bird's-eye feature map is generated.

[0030] This specification provides an electronic device, including a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus;

[0031] The memory stores machine-readable instructions, and the processor executes the above-mentioned method for generating a bird's-eye view by calling the machine-readable instructions.

[0032] This specification provides a machine-readable storage medium, which stores machine-readable instructions. When the machine-readable instructions are called and executed by a processor, the above-mentioned method for generating a bird's-eye view is implemented.

[0033] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0034] In this specification, a pre-built command-trajectory dataset is obtained. The command-trajectory dataset containing a variety of driving trajectories provides better generalization capability for the subsequently generated bird's-eye view feature map.

[0035] Then, based on the command-trajectory data set, a multi-frame bird's-eye view feature map corresponding to at least one vehicle control command is generated. Thus, the construction of a multi-trajectory bird's-eye view feature map is realized, allowing the autonomous driving vehicle to handle more complex and changeable traffic scenes.

[0036] Next, the multi-frame bird's-eye view feature map is input into the feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the multi-frame bird's-eye view feature map based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye view feature map, and performs feature fusion on the aligned first frame of the bird's-eye view feature map and the second frame of the bird's-eye view feature map to generate the second frame of the fused bird's-eye view feature map, and the second frame of the fused bird's-eye view feature map is continuously input into the feature fusion module to perform the same action, and so on, until the last frame of the fused bird's-eye view feature map in the multi-frame bird's-eye view feature map is generated. Thus, the position changes of the autonomous driving vehicle at different time points are taken into account, and the various driving paths that the autonomous driving vehicle may take in a complex traffic environment are simulated. Through the real-time position of the autonomous driving vehicle under different trajectories, the multi-frame bird's-eye view feature map is feature fused, so that the fused bird's-eye view feature map can more comprehensively capture the changes in the environment around the autonomous driving vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The illustrative embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation on this specification. In the drawings:

[0038] Figure 1 is a flow chart of a method for generating a bird's-eye view shown in an exemplary embodiment;

[0039] Figure 2 is a schematic diagram of a driving trajectory shown in an exemplary embodiment;

[0040] Figure 3 is a schematic diagram of a heading angle of an autonomous driving vehicle shown in an exemplary embodiment;

[0041] Figure 4 is a schematic diagram of a predicted trajectory shown in an exemplary embodiment;

[0042] Figure 5 is a structural diagram of an electronic device in which a bird's-eye view generating device is located, shown in an exemplary embodiment;

[0043] Figure 6 It is a structural diagram of a device for generating a bird's-eye view, shown in an exemplary embodiment. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0045] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0046] Currently, autonomous vehicles can generate bird's-eye view feature maps through sensor data in the nuScenes dataset to improve the vehicle's environmental perception capabilities.

[0047] However, the sensor data in the nuScenes dataset is collected when the vehicle is following a predetermined or natural driving trajectory. This means that the generated bird's-eye view feature map can reflect the environmental characteristics under a specific trajectory. However, this data collection method that relies on a fixed trajectory also means that the resulting bird's-eye view feature map has certain limitations in generalization ability. As a result, the accuracy of the decision made by the autonomous vehicle when facing complex or unknown environments is reduced.

[0048] Based on this, this specification proposes a technical solution for generating a multi-frame bird's-eye feature map based on a pre-constructed instruction-trajectory data set, and inputting the multi-frame bird's-eye feature map into a feature fusion module to generate the last frame fused bird's-eye feature map in the multi-frame bird's-eye feature map. Thus, by including an instruction-trajectory data set with a variety of driving trajectories, a better generalization capability is provided for the subsequently generated bird's-eye feature map, and the position changes of the autonomous driving vehicle at different time points are taken into account, simulating a variety of driving paths that the autonomous driving vehicle may take in a complex traffic environment. By using the real-time position of the autonomous driving vehicle under different trajectories, the multi-frame bird's-eye feature map is feature fused, so that the fused bird's-eye feature map can more comprehensively capture the changes in the environment surrounding the autonomous driving vehicle.

[0049] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0050] Figure 1The present invention is a flowchart of a method for generating a bird's-eye view according to an exemplary embodiment, wherein the method specifically comprises the following steps:

[0051] S100: Acquire a pre-constructed instruction-trajectory data set; the instruction-trajectory data set contains at least one instruction-trajectory pair; the instruction-trajectory pair consists of a vehicle control instruction and a driving trajectory corresponding to the vehicle control instruction; wherein the vehicle control instruction is an instruction for controlling the autonomous driving vehicle to perform a specified driving action.

[0052] In an embodiment of the present specification, a pre-constructed instruction-trajectory data set is obtained. The instruction-trajectory data set includes at least one instruction-trajectory pair. The instruction-trajectory pair consists of a vehicle control instruction and a driving trajectory corresponding to the vehicle control instruction.

[0053] The vehicle control command is a command for controlling the autonomous driving vehicle to perform a specified driving action. For example, the vehicle control command may include: turn left, turn right, go straight, change lane left, change lane right, turn around, etc.

[0054] In the embodiment of this specification, the method for constructing the instruction-trajectory data set is specifically as follows:

[0055] First, an autonomous driving data set to be labeled and at least one vehicle control instruction are obtained.

[0056] Then, at least one instruction-trajectory pair is constructed based on the at least one vehicle control instruction and the driving trajectory corresponding to the at least one vehicle control instruction.

[0057] Next, based on at least one instruction-trajectory pair, the autonomous driving dataset is labeled to construct an instruction-trajectory dataset.

[0058] In practical applications, "Drive Language" (DL) means using the power of a large language model to achieve natural and intuitive communication between the driver and the autonomous vehicle, so that the autonomous vehicle can understand and respond to voice commands and control the driving of the autonomous vehicle. Thus, voice control of the autonomous vehicle is achieved.

[0059] However, due to the personalized way users express themselves when giving voice commands, the same intention may be expressed in a variety of different sentence structures and vocabulary choices, which may result in lower recognition accuracy of the user's voice commands by the autonomous driving vehicle.

[0060] In the embodiment of the present specification, audio data input by the user is obtained.

[0061] Then, based on a preset audio recognition algorithm, the audio data is recognized to obtain text data corresponding to the audio data. The audio recognition algorithm may refer to an automatic speech recognition algorithm (Automatic Speech Recognition, ASR).

[0062] Next, the text data is input into a pre-trained large language model to determine the vehicle control instruction corresponding to the text data. For example, the text data may be "the vehicle ahead is driving too slowly, overtake the vehicle ahead", and the vehicle control instruction may be "overtake". For another example, the text data may be "the traffic ahead is too congested, change the route, turn left at the next intersection", and the vehicle control instruction may be "turn left". This specification does not limit the specific content of the text data.

[0063] Furthermore, although the large language model can extract keywords from text data, the initially extracted keywords may be ambiguous or inaccurate, and may also contain repeated or irrelevant keywords. In addition, vehicle control instructions also need to follow specific formats and standards.

[0064] In the embodiment of the present specification, text data is input into a pre-trained large language model to determine keywords corresponding to the text data.

[0065] Then, the keywords are post-processed to determine the vehicle control instructions corresponding to the keywords. The post-processing includes at least operations such as screening, sorting, and formatting. The screening operation may refer to screening out keywords related to vehicle control from the extracted keywords. The sorting operation may refer to sorting the keywords according to the priority or order of the vehicle control instructions. The formatting operation may refer to converting the extracted keywords into a standard instruction format that can be recognized by the autonomous driving vehicle.

[0066] For example, the text data is "The traffic ahead is too congested, change the route, and turn left at the next intersection." First, extract the keywords in the text data to obtain keywords. For example, "next intersection", "turn left." Then, filter the keywords to obtain the keyword "turn left" related to vehicle control. Next, sort the keywords and determine that "turn left" is the first keyword. Finally, convert the extracted keywords into a standard instruction format that can be recognized by the autonomous driving vehicle.

[0067] It should be noted that the large language model may include models such as ChatGLM, ChatGPT, BERT, etc. This specification does not limit the large language model.

[0068] Among them, the autonomous driving dataset may refer to the nuScenes dataset. The nuScenes dataset is a large dataset that provides a full set of sensor data for autonomous driving vehicles. The sensors on the autonomous driving vehicles include: 6 cameras, 1 lidar, 5 millimeter-wave radars, GPS, and IMU. Furthermore, the nuScenes dataset consists of 1,000 scenes, each of which is 20 seconds long. In each scene, there are 40 key frames, that is, 2 key frames per second. For the object detection task, 3D bounding boxes are annotated with 10 classes and 8 attributes.

[0069] As can be seen from the above embodiments, the personalized voice commands issued by the user are recognized through a large language model to obtain vehicle control commands that can be recognized by the autonomous driving vehicle. Thus, the recognition accuracy of the user's voice commands by the autonomous driving vehicle is improved. In addition, the decision-making process of the autonomous driving vehicle is made more transparent, allowing users or organizations to understand and explain the decision-making of the autonomous driving vehicle.

[0070] In practical applications, the sensor data in the nuScenes dataset is collected by the vehicle along a set driving trajectory. This means that the generated bird's-eye view feature map can reflect the environmental characteristics under a specific trajectory. However, this data collection method that relies on a fixed trajectory also means that the resulting bird's-eye view feature map has certain limitations in generalization ability. As a result, the accuracy of the decision made by the autonomous vehicle when facing complex or unknown environments is reduced.

[0071] In the embodiments of this specification, the driving trajectory is composed of a set number of trajectory points, and the trajectory points include the coordinates and heading angles of the autonomous driving vehicle. For example, each driving trajectory has 10 trajectory points, and the coordinates and heading angles of each trajectory point are (x, y, yaw). Figure 2 shown.

[0072] Figure 2 It is a schematic diagram of a driving trajectory shown in an exemplary embodiment.

[0073] exist Figure 2 In the figure, the yellow box is used to represent the trajectory point. The yellow box forms two driving trajectories, one corresponding to the vehicle control instruction "turn right" and the other corresponding to the vehicle control instruction "turn around".

[0074] It should be noted that the trajectory points can be marked manually.

[0075] Among them, the method for determining the heading angle of the autonomous driving vehicle is as follows: Figure 3 shown.

[0076] Figure 3It is a schematic diagram of a heading angle of an autonomous driving vehicle shown in an exemplary embodiment.

[0077] In Figure (a), the autonomous vehicle is in the initial state, with the autonomous vehicle as the origin. The positive direction of the y-axis is the direction of the front of the autonomous vehicle, and the positive direction of the x-axis is the right side of the front of the autonomous vehicle. The heading angle corresponding to the positive direction of the y-axis is π. The heading angle corresponding to the negative direction of the y-axis is 0. The coordinates of the autonomous vehicle in Figure (a) are (0, 0), and the heading angle is -π.

[0078] As can be seen in Figure (b), the heading angle can be defined as the angle between the heading of the autonomous driving vehicle and the negative direction of the y-axis, and the range of the heading angle is -2π to 0.

[0079] S102: Based on the instruction-trajectory data set, generate a multi-frame bird's-eye view feature map corresponding to the at least one vehicle control instruction.

[0080] In an embodiment of the present specification, a multi-frame bird's-eye view feature map corresponding to at least one vehicle control instruction is generated based on the instruction-trajectory data set.

[0081] Specifically, feature extraction is performed on the instruction-trajectory dataset to obtain multi-frame feature maps.

[0082] Then, the multi-frame feature maps are perspective transformed to obtain multi-frame bird's-eye view feature maps. The perspective transformation mentioned here is a geometric transformation that converts the feature map from the vehicle's perspective to a bird's-eye view. For example, the LSS (Lift, Splat, Shoot) or BEVFormer projection method is used to convert the multi-view 2D features into a bird's-eye view feature map.

[0083] In the embodiments of this specification, there are multiple methods for feature extraction of instruction-trajectory data sets. For example, the PointPillars model can be used. The PointPillars model consists of three parts: a point cloud column feature extraction network (Pillar FeatureNet), a two-dimensional feature extraction backbone network (Backbone (2D CNN)), and a target detection head (Detection Head (SSD)). Among them, the point cloud column feature extraction network is used to perform point cloud column (Pillar) clustering on the input point cloud, perform bird's-eye view plane projection on the point cloud column, and output the final projection result as a pseudo image (Pseudo Image). The two-dimensional feature extraction backbone network extracts two-dimensional image features from the pseudo image based on the traditional multi-level downsampling convolutional network. The target detection head classifies the extracted bird's-eye view features, and restores the classification results to the original point cloud data, adding semantic features to each point in the original point cloud data.

[0084] The point cloud pillar feature extraction network can divide the point cloud data into several grids according to the x-axis and y-axis coordinates of each point in the point cloud. The point cloud data divided in the same grid constitutes a pillar. Each point can be represented by a D = 9-dimensional vector, that is, (x, y, z, r, x c ,y c , z c , x p ,y p ), where (x, y, z) can be used to represent the three-dimensional coordinates of the point in the laser system. r can be used to represent the reflection intensity of the point. c ,y c , z c ) can be used to represent the geometric center of all points on the pillar where the point cloud is located. (x p ,y p ) can be used to indicate the relative position of the point and the geometric center.

[0085] Assuming that each frame of point cloud has P non-empty pillars and each pillar has N point cloud data, the frame of point cloud can be represented as a tensor of (D, P, N). In order to ensure that each pillar has N point cloud data, the strategy of selecting more and filling less is adopted. When the number of point clouds in each pillar exceeds N, N points are randomly sampled. When the number of point clouds in each pillar is less than N, the less part is filled with 0 vectors. Thus, the irregular point cloud data is converted into stacked Pillars with fixed shape.

[0086] Next, a simplified PointNet is used to extract features from pillars to obtain a tensor of shape (C, P, N). Then, a MaxPooling operation is performed on the dimension where pillar is located to obtain a feature map of shape (C, P). In order to realize the pseudo-image feature, P is converted to (H, W) to obtain a pseudo-image of shape (C, H, W).

[0087] Then, the two-dimensional feature extraction backbone network extracts two-dimensional image features from the pseudo-image to obtain a feature map.

[0088] For another example, a deep learning network may be used to extract features from the instruction-trajectory data set. The deep learning network mentioned here may include: a convolutional neural network, a recurrent neural network, a Transformer, etc. This specification does not specifically limit the method of feature extraction.

[0089] S106: Input the multiple frames of bird's-eye view feature maps into a feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the bird's-eye view feature map in the multiple frames of the bird's-eye view feature map based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye view feature map, and performs feature fusion on the aligned first frame of the bird's-eye view feature map and the second frame of the bird's-eye view feature map to generate a second frame of fused bird's-eye view feature map, and continues to input the second frame of fused bird's-eye view feature map into the feature fusion module to perform the same action, and so on, until the last frame of fused bird's-eye view feature map in the multiple frames of the bird's-eye view feature map is generated.

[0090] In practical applications, the sensor data in the nuScenes dataset is collected by the vehicle along a set driving trajectory. The collected sensor data are all in the vehicle coordinate system. Therefore, the bird's-eye view feature map generated based on the sensor data in the nuScenes dataset is vehicle-centric, providing an intuitive view of the surrounding environment for the autonomous driving vehicle. Furthermore, in the process of feature fusion of multiple frames of bird's-eye view feature maps, the autonomous driving vehicle is always located at the center of the bird's-eye view feature map, so that the positions of other vehicles around the autonomous driving vehicle are consistent with the actual situation, and there is no need to consider the problem of position deviation of other vehicles around the autonomous driving vehicle.

[0091] However, the driving trajectories in the command-trajectory dataset constructed in this specification are manually annotated, which means that in the bird's-eye view feature map generated based on the sensor data in the command-trajectory dataset, the position of the autonomous driving vehicle will be adjusted according to the manually annotated trajectory. However, the positions of other vehicles around the autonomous driving vehicle in the bird's-eye view feature map are still based on the standard of the nuScenes dataset. Therefore, in the process of feature fusion of multiple frames of bird's-eye view feature maps, the positions of other vehicles around the autonomous driving vehicle will be deviated. The specific formula is as follows:

[0092]

[0093] In the above formula, T can be used to represent the current moment, and T-1 can be used to represent the moment before the current moment. It can be used to represent the position of a stationary vehicle in the vehicle coordinate system at time T. It can be used to represent the position of the stationary vehicle at time T-1 in the vehicle coordinate system. The calculation result of the above formula can be used to represent the position deviation of the stationary vehicle from time T to time T-1. It can be found that due to The existence of , the position deviation is not 0.

[0094] Therefore, in the process of fusing the bird's-eye view feature map of the current frame with the bird's-eye view feature map of the previous frame, it is necessary to add the vehicle motion transformation matrix from time T to time T-1 Thus, the positions of other vehicles around the autonomous driving vehicle are prevented from deviating. The specific formula is as follows:

[0095]

[0096] In the above formula, It can be used to represent the position of a stationary vehicle at the i-th trajectory point in the vehicle coordinate system at time T. It can be used to represent the position of the stationary vehicle at the jth trajectory point in the vehicle coordinate system at time T-1.

[0097] Combined with the above formula, multiple frames of bird's-eye view feature maps are input into the feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the bird's-eye view feature map in the multiple frames of bird's-eye view feature map based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye view feature map, and performs feature fusion on the aligned first frame of the bird's-eye view feature map and the second frame of the bird's-eye view feature map to generate a second frame of fused bird's-eye view feature map, and continues to input the second frame of the fused bird's-eye view feature map into the feature fusion module to perform the same action, and so on, until the last frame of the fused bird's-eye view feature map in the multiple frames of the bird's-eye view feature map is generated.

[0098] It should be noted that the feature fusion module can adopt the time architecture of BEVFormer.

[0099] In the embodiment of this specification, the last frame of the fused bird's-eye view feature map in the multi-frame bird's-eye view feature map is input into the trajectory prediction module to obtain the predicted trajectory corresponding to the vehicle control command. Figure 4 shown.

[0100] Figure 4 It is a schematic diagram of a predicted trajectory shown in an exemplary embodiment.

[0101] exist Figure 4 It can be seen that if the vehicle control command is not received, the predicted trajectory tends to continue to go straight, for example, Figure (a). When the vehicle control command is to turn left, the predicted trajectory tends to turn left, for example, Figure (b). When the vehicle control command is to turn right, the predicted trajectory tends to turn right, for example, Figure (c). Of course, the vehicle control command can be a U-turn, and the predicted trajectory tends to turn U-turn. The vehicle control command can also be a left lane change, and the predicted trajectory tends to change lanes to the left. The vehicle control command can also be a right lane change, and the predicted trajectory tends to change lanes to the right.

[0102] It can be seen from the above method that obtaining a pre-built command-trajectory dataset includes a variety of driving trajectories, which provides better generalization ability for the subsequently generated bird's-eye view feature map.

[0103] Then, based on the command-trajectory data set, a multi-frame bird's-eye view feature map corresponding to at least one vehicle control command is generated. Thus, the construction of a multi-trajectory bird's-eye view feature map is realized, allowing the autonomous driving vehicle to handle more complex and changeable traffic scenes.

[0104] Next, the multi-frame bird's-eye view feature map is input into the feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the multi-frame bird's-eye view feature map based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye view feature map, and performs feature fusion on the aligned first frame of the bird's-eye view feature map and the second frame of the bird's-eye view feature map to generate the second frame of the fused bird's-eye view feature map, and the second frame of the fused bird's-eye view feature map is continuously input into the feature fusion module to perform the same action, and so on, until the last frame of the fused bird's-eye view feature map in the multi-frame bird's-eye view feature map is generated. Thus, the position changes of the autonomous driving vehicle at different time points are taken into account, and the various driving paths that the autonomous driving vehicle may take in a complex traffic environment are simulated. Through the real-time position of the autonomous driving vehicle under different trajectories, the multi-frame bird's-eye view feature map is feature fused, so that the fused bird's-eye view feature map can more comprehensively capture the changes in the environment around the autonomous driving vehicle.

[0105] Corresponding to the above-mentioned embodiment of the method based on bird's-eye view generation, this specification also provides an embodiment of a device for generating a bird's-eye view.

[0106] See also Figure 5 , Figure 5 It is a structural diagram of an electronic device in which a bird's-eye view generating device is located, shown in an exemplary embodiment. At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and of course may also include other required hardware. One or more embodiments of this specification can be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0107] See also Figure 6 , Figure 6 FIG. 1 is a structural diagram of a device for generating a bird's-eye view according to an exemplary embodiment. The device for generating a bird's-eye view can be applied to Figure 5 In the electronic device shown in the figure, the technical solution of this specification is implemented. The device may include:

[0108] The acquisition module 600 is used to acquire a pre-constructed instruction-trajectory data set; the instruction-trajectory data set includes at least one instruction-trajectory pair; the instruction-trajectory pair consists of a vehicle control instruction and a driving trajectory corresponding to the vehicle control instruction; wherein the vehicle control instruction is an instruction for controlling the autonomous driving vehicle to perform a specified driving action;

[0109] A generating module 602, configured to generate a plurality of frames of bird's-eye view feature images corresponding to the at least one vehicle control instruction based on the instruction-trajectory data set;

[0110] The fusion module 604 is used to input the multi-frame bird's-eye view feature map into the feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the multi-frame bird's-eye view feature map based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye view feature map, and performs feature fusion on the aligned first frame of the bird's-eye view feature map and the second frame of the bird's-eye view feature map to generate a second frame of fused bird's-eye view feature map, and continues to input the second frame of fused bird's-eye view feature map into the feature fusion module to perform the same action, and so on, until the last frame of fused bird's-eye view feature map in the multi-frame bird's-eye view feature map is generated.

[0111] Optionally, the acquisition module 600 is further specifically used to obtain an autonomous driving data set to be labeled and at least one vehicle control instruction; construct at least one instruction-trajectory pair based on the at least one vehicle control instruction and the driving trajectory corresponding to the at least one vehicle control instruction; based on the at least one instruction-trajectory pair, label the autonomous driving data set to construct an instruction-trajectory data set.

[0112] Optionally, the acquisition module 600 is further specifically used to acquire audio data input by a user; based on a preset audio recognition algorithm, the audio data is recognized to obtain text data corresponding to the audio data; the text data is input into a pre-trained large language model to determine the vehicle control instructions corresponding to the text data.

[0113] Optionally, the acquisition module 600 is further used to input the text data into a pre-trained large language model to determine keywords corresponding to the text data; post-process the keywords to determine vehicle control instructions corresponding to the keywords; wherein the post-processing includes at least: screening, sorting, and formatting.

[0114] Optionally, the driving trajectory is composed of a set number of trajectory points, and the trajectory points include coordinates and heading angles of the autonomous driving vehicle.

[0115] Optionally, the generating module 602 is specifically used to perform feature extraction on the instruction-trajectory data set to obtain a multi-frame feature map; and perform perspective transformation on the multi-frame feature map to obtain a multi-frame bird's-eye view feature map.

[0116] Optionally, the autonomous driving dataset includes: a nuScenes dataset, and the feature fusion module adopts the temporal architecture of BEVFormer.

[0117] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0118] Based on the same concept as the above method, this specification also provides an electronic device, including: a processor; a memory for storing processor executable instructions; wherein the processor implements the steps of the method described in any of the above embodiments by running the executable instructions.

[0119] Based on the same concept as the above method, this specification also provides a computer-readable storage medium on which computer instructions are stored. When the instructions are executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0120] Based on the same concept as the above method, this specification also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any of the above embodiments when executed by a processor.

[0121] It should be understood that although the terms first, second, third, etc. may be used to describe various information in one or more embodiments of this specification, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0122] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for generating a bird's-eye view, characterized in that: The method comprises: Acquire a pre-constructed instruction-trajectory data set; the instruction-trajectory data set includes at least one instruction-trajectory pair; the instruction-trajectory pair consists of a vehicle control instruction and a driving trajectory corresponding to the vehicle control instruction; wherein the vehicle control instruction is an instruction for controlling the autonomous driving vehicle to perform a specified driving action; Based on the command-trajectory data set, generating a plurality of frames of bird's-eye view feature maps corresponding to at least one vehicle control command; The multiple frames of bird's-eye view feature images are input into a feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the bird's-eye view feature image in the multiple frames of the bird's-eye view feature image based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye view feature image, and performs feature fusion on the aligned first frame of the bird's-eye view feature image and the second frame of the bird's-eye view feature image to generate a second frame of fused bird's-eye view feature image, and the second frame of fused bird's-eye view feature image is continuously input into the feature fusion module to perform the same action, and so on, until the last frame of fused bird's-eye view feature image in the multiple frames of the bird's-eye view feature image is generated.

2. The method according to claim 1, characterized in that Before acquiring the pre-built instruction-trajectory dataset, the method further includes: Obtaining an autonomous driving data set to be labeled and at least one vehicle control instruction; constructing at least one command-trajectory pair based on the at least one vehicle control command and the driving trajectory corresponding to the at least one vehicle control command; Based on the at least one instruction-trajectory pair, the autonomous driving dataset is labeled to construct an instruction-trajectory dataset.

3. The method according to claim 2, characterized in that Before obtaining the autonomous driving data set to be labeled and at least one vehicle control instruction, the method further includes: Get the audio data input by the user; Based on a preset audio recognition algorithm, the audio data is recognized to obtain text data corresponding to the audio data; The text data is input into a pre-trained large language model to determine a vehicle control instruction corresponding to the text data.

4. The method according to claim 3, characterized in that Inputting the text data into a pre-trained large language model, and determining a vehicle control instruction corresponding to the text data, including: Inputting the text data into a pre-trained large language model to determine keywords corresponding to the text data; The keyword is post-processed to determine the vehicle control instruction corresponding to the keyword; wherein the post-processing at least includes: screening, sorting, and formatting.

5. The method according to claim 2, characterized in that The driving trajectory is composed of a set number of trajectory points, and the trajectory points include the coordinates and heading angle of the autonomous driving vehicle.

6. The method according to claim 1, characterized in that Based on the command-trajectory data set, a multi-frame bird's-eye view feature map is generated, including: Performing feature extraction on the command-trajectory data set to obtain a multi-frame feature map; The multi-frame feature maps are perspective transformed to obtain multi-frame bird's-eye view feature maps.

7. The method according to claim 1, characterized in that The autonomous driving dataset includes: a nuScenes dataset, and the feature fusion module adopts the BEVFormer time architecture.

8. A device for generating a bird's-eye view, characterized in that: The device comprises: An acquisition module, used to acquire a pre-constructed instruction-trajectory data set; the instruction-trajectory data set includes at least one instruction-trajectory pair; the instruction-trajectory pair consists of a vehicle control instruction and a driving trajectory corresponding to the vehicle control instruction; wherein the vehicle control instruction is an instruction for controlling the autonomous driving vehicle to perform a specified driving action; A generating module, configured to generate a plurality of frames of bird's-eye view feature images corresponding to the at least one vehicle control instruction based on the instruction-trajectory data set; A fusion module is used to input the multi-frame bird's-eye feature map into a feature fusion module, so that the feature fusion module performs feature alignment on the first frame of the multi-frame bird's-eye feature map based on the vehicle motion conversion matrix to obtain the aligned first frame of the bird's-eye feature map, and performs feature fusion on the aligned first frame of the bird's-eye feature map and the second frame of the bird's-eye feature map to generate a second frame of fused bird's-eye feature map, and continues to input the second frame of fused bird's-eye feature map into the feature fusion module to perform the same action, and so on, until the last frame of fused bird's-eye feature map of the multi-frame bird's-eye feature map is generated.

9. An electronic device, characterized in that: It includes a communication interface, a processor, a memory and a bus, wherein the communication interface, the processor and the memory are interconnected via the bus; The memory stores machine-readable instructions, and the processor executes the method according to any one of claims 1 to 7 by calling the machine-readable instructions.

10. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-readable instructions, and when the machine-readable instructions are called and executed by the processor, the method according to any one of claims 1 to 7 is implemented.

Images