Road environment sensing method and sensing model training method

Abstract

The invention provides a road environment perception method and a perception model training method, and relates to the technical field of image processing, in particular to the fields of automatic driving, computer vision, deep learning and neural network models. The specific implementation scheme of the road environment sensing method comprises the following steps of: obtaining image features from a look-around image of a vehicle by utilizing a feature extraction network of a first sensing model; wherein the look-around image is obtained according to a plurality of image acquisition devices at different angles; determining a bird's-eye view feature map according to the image features and the bird's-eye view query vector by using a coding network of a first perception model; and respectively determining output results of the plurality of perception tasks according to the bird's-eye view feature map by using the plurality of head networks of the first perception model. According to the technical scheme provided by the invention, computing resources consumed by simultaneous execution of multiple visual tasks can be effectively reduced, and the real-time performance and task effects of the multiple visual tasks are ensured.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to the fields of autonomous driving, computer vision, deep learning, and neural network models. Background Art

[0002] Object detection and occupancy grid tasks are two common environmental perception tasks for autonomous driving. Object detection is to identify objects from images and mark their locations. The occupancy grid task is a task that represents the real world by predicting a three-dimensional grid. It predicts a three-dimensional grid to indicate whether each position in the environment is occupied. Each grid can have different states, indicating that the position is empty, occupied by a certain type of object, or inaccessible. The occupancy grid task can help the autonomous driving system effectively handle obstacles of different shapes and sizes and provide more accurate environmental perception.

[0003] The detection network required for the target detection task and the detection network required for the occupancy grid task are both very resource-intensive, resulting in a high time consumption when the two tasks are run on the computing platform. Summary of the invention

[0004] The present disclosure provides a road environment perception method, a perception model training method, an apparatus, a device and a storage medium.

[0005] According to a first aspect of the present disclosure, a road environment perception method is provided, comprising:

[0006] Using a feature extraction network of the first perception model, image features are obtained from a surround view image of the vehicle; wherein the surround view image is obtained based on image acquisition devices at multiple different angles;

[0007] Determine a bird's-eye view feature map according to the image features and the bird's-eye view query vector using the encoding network of the first perception model;

[0008] The multiple head networks of the first perception model are used to respectively determine output results of multiple perception tasks according to the bird's-eye view feature map.

[0009] According to a second aspect of the present disclosure, a method for training a perception model is provided, comprising:

[0010] Using a feature extraction network of a second perception model, sample features are obtained from a sample image; wherein the sample image is obtained according to image acquisition devices at multiple different angles;

[0011] Determine a bird's-eye view feature map according to the sample features and the bird's-eye view query vector using the encoding network of the second perception model;

[0012] Using the multiple head networks of the second perception model respectively, and determining the prediction results of the perception tasks corresponding to the respective head networks according to the bird's-eye view feature map; wherein the multiple head networks include a first head network for performing the target detection task and a second head network for performing the grid occupation task;

[0013] Determine the loss values ​​of multiple perception tasks according to the prediction results of multiple perception tasks and the corresponding annotation results;

[0014] According to the loss values ​​and corresponding weight coefficients of the multiple perception tasks, the parameters of the second perception model are updated to obtain a trained target perception model.

[0015] According to a third aspect of the present disclosure, a road environment perception device is provided, comprising:

[0016] A first feature extraction module is used to obtain image features from a surround view image of the vehicle using a feature extraction network of a first perception model; wherein the surround view image is obtained based on image acquisition devices at multiple different angles;

[0017] A first encoding module, configured to determine a bird's-eye view feature map according to the image features and the bird's-eye view query vector by using an encoding network of the first perception model;

[0018] An inference module is used to use multiple head networks of the first perception model to determine output results of multiple perception tasks according to the bird's-eye view feature map.

[0019] According to a fourth aspect of the present disclosure, a training device for a perception model is provided, comprising:

[0020] A second feature extraction module is used to obtain sample features from a sample image using a feature extraction network of a second perception model; wherein the sample image is obtained according to image acquisition devices at multiple different angles;

[0021] A second encoding module, configured to determine a bird's-eye view feature map according to the sample features and the bird's-eye view query vector by using an encoding network of the second perception model;

[0022] A prediction module, used to respectively use multiple head networks of the second perception model and determine the prediction results of the perception tasks corresponding to each head network according to the bird's-eye view feature map; wherein the multiple head networks include a first head network for performing the target detection task and a second head network for performing the grid occupation task;

[0023] A loss determination module, used to determine the loss values ​​of multiple perception tasks according to the prediction results of the multiple perception tasks and the corresponding annotation results;

[0024] A parameter updating module is used to update the parameters of the second perception model according to the loss values ​​and corresponding weight coefficients of the multiple perception tasks to obtain a trained target perception model.

[0025] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0026] at least one processor; and

[0027] a memory communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0029] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute any method according to the embodiments of the present disclosure.

[0030] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0031] According to the technical solution provided by the present disclosure, the computing resources consumed by executing multiple visual tasks simultaneously can be effectively reduced, ensuring the real-time performance and task effects of multiple visual tasks.

[0032] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0034] Figure 1 is a system architecture diagram of a road environment perception method according to an embodiment of the present disclosure;

[0035] Figure 2 is a flowchart of a road environment perception method according to an embodiment of the present disclosure;

[0036] Figure 3 is a structural schematic diagram of a first perception model according to an embodiment of the present disclosure;

[0037] Figure 4 is a flowchart of a perception model training method according to an embodiment of the present disclosure;

[0038] Figure 5 is a schematic structural diagram of a road environment perception device according to an embodiment of the present disclosure;

[0039] Figure 6 is a structural schematic diagram of a perception model training device according to an embodiment of the present disclosure;

[0040] Figure 7 is a block diagram of an electronic device for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0041] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0042] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the road environment perception method and perception model training method, apparatus, electronic device, and computer-readable storage medium of the present application can be applied.

[0043] like Figure 1 As shown, the system architecture 100 may include one or more servers (e.g., servers 101-105), one or more vehicles (e.g., vehicles 107-109), and a network 106 for communication between the server and the multiple vehicles. The vehicle is provided with an environmental perception device, such as an image acquisition device, so as to acquire image data around the vehicle. By cooperating with multiple image acquisition devices at different angles, a 360° range around the vehicle can be covered to obtain a surround image of the vehicle. The network 106 is used as a medium to provide a communication link between the autonomous driving vehicle and the server. There is also a communication network between the servers, which may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0044] One or more vehicles may interact with the server via the network 106 to receive or send messages, etc. For example, the autonomous driving vehicle and the server may be installed with various applications for realizing information communication between the two, such as target detection applications, model calling applications, model building applications, etc.

[0045] Autonomous driving vehicles and servers are usually different types of hardware devices. In special cases, they can also be software or virtual products of software in simulation scenarios.

[0046] The server can provide various services through various built-in applications, and can also provide real-time target detection and obstacle detection services for the autonomous driving vehicle 107 while driving. Taking the road environment perception application as an example, the server 101 can achieve the following effects when running the road environment perception application: first, the surround view image captured by the image acquisition device of the target autonomous driving vehicle 107 is obtained through the network 106; then, the surround view image is input into the first perception model to obtain the labeling results of the position and type of the target object and the labeling results of each three-dimensional grid output by the first perception model.

[0047] The first perception model can be deployed on the vehicle terminal side corresponding to any vehicle, or on the server side. If the first perception model is deployed on the vehicle terminal side corresponding to the vehicle, the environment perception task is performed based on the vehicle terminal. If the first perception model is deployed on the server side, the server side can adopt a distributed cluster system, such as Figure 1 As shown, the distributed cluster system includes multiple nodes (such as server cluster 101, server 102, server cluster 103, server 104, and server 105), and the multiple nodes can jointly perform one or more environmental perception tasks. Optionally, the multiple nodes in the distributed cluster system can perform part of the processing flow in the image processing task, or can perform the entire processing flow in the image processing task. Optionally, after each round of data processing tasks is completed, data can be exchanged (such as data synchronization) between the multiple nodes.

[0048] Figure 2 The following is a flow chart of a road environment perception method provided by an embodiment of the present disclosure. Figure 2 As shown, the method comprises at least the following steps:

[0049] S201, using a feature extraction network of a first perception model to obtain image features from a surround view image of the vehicle, wherein the surround view image is obtained using image acquisition devices at multiple different angles.

[0050] In the disclosed embodiment, the vehicle-side sensor may include any visual sensor such as a monocular visual sensor, a binocular visual sensor, a laser visual sensor, a structured light visual sensor, but is not limited thereto, and may also include other types of sensors such as a laser radar sensor and a millimeter wave radar sensor. The disclosed embodiment does not limit the type of vehicle-side sensor. The surround view image of the vehicle includes a single-view image captured by a plurality of visual sensors or image acquisition devices at different angles. Multiple single-view images may constitute a surround view image.

[0051] The feature extraction network can be a convolutional feature extraction network based on a convolutional neural network (CNN) architecture (such as ResNet) or a feature extraction network based on a transformer model architecture (such as ViT, DETR). Other network architectures in the prior art can also be used.

[0052] S202. Determine a bird's-eye view feature map according to image features and a bird's-eye view query vector using the encoding network of the first perception model.

[0053] In the disclosed embodiment, the encoding network uses the Transformer Encoder architecture to generate a Bird's Eye View (BEV) feature map. BEV refers to a scene image presented from a bird's eye view. In autonomous driving, BEV images are usually synthesized from the perspectives of multiple cameras to form a two-dimensional plane map that can clearly show the environment around the vehicle. This perspective helps to improve the accuracy of target detection because it reduces the complexity of depth information, allowing the vehicle to better understand the location and shape of surrounding obstacles.

[0054] Bird's-eye-view query vectors (BEV Queries) can also be translated into Chinese as bird's-eye-view queries or BEV queries. The "query" here refers to the features or information that the model focuses on when processing data. In the process of generating BEV feature maps, these queries are used to extract and fuse features related to the environment, which can be regarded as a focus on a specific location or area.

[0055] The BEV feature map is a two-dimensional feature representation generated by processing BEV Queries, usually displayed in the form of a grid. Each grid cell (or grid) represents the feature information of a specific location. It can indicate whether the location is occupied, usually including states such as occupied, free, and unknown. In some implementations, the feature map may also contain category information about the occupied objects (for example, vehicles, pedestrians, etc.) to facilitate subsequent target detection and understanding.

[0056] BEV Queries are high-dimensional representations used to generate feature maps, which contain important environmental information. BEV feature maps are the final two-dimensional feature representations that provide a holistic understanding of the environment and can be used to support the perception and decision-making process in autonomous driving.

[0057] S203. Using the multiple head networks of the first perception model, according to the bird's-eye view feature map, respectively determine the output results of the multiple perception tasks.

[0058] In the disclosed embodiment, the BEV feature map obtained by the encoding network is output to multiple detection heads to complete multiple different perception tasks, such as target detection tasks and occupancy grid tasks. It can also be understood that multiple detection heads share the feature extraction network and the encoding network.

[0059] According to the scheme of the embodiment of the present disclosure, multiple detection heads used to perform different visual perception tasks share a feature extraction network and an encoding network, which can effectively reduce the computing resources consumed by the simultaneous execution of multiple visual tasks and ensure the real-time performance and task effects of multiple visual tasks, thereby allowing multiple visual tasks to be run on a low-computing power platform (such as the vehicle side).

[0060] In a possible implementation, step S203 uses multiple head networks of the first perception model to determine output results of multiple perception tasks according to the bird's-eye view feature map, and further includes the steps of:

[0061] S2031. Determine a detection result of the target detection task according to a bird's-eye view feature map by using a first head network of the first perception model for performing the target detection task.

[0062] S2032. Determine a detection result of the grid occupation task according to the bird's-eye view feature map using the second head network of the first perception model for performing the grid occupation task.

[0063] In the disclosed embodiment, the encoding network is connected to two heads, namely, a first head network for performing target detection tasks, detection head, and a second head network for performing occupancy grid tasks, occ head.

[0064] The BEV feature map can also be used for a variety of environmental perception tasks such as semantic segmentation, instance segmentation, path planning, self-localization and map construction, motion estimation and tracking, behavior prediction, depth estimation, traffic sign detection, lane line detection, etc. The first perception model can also include a head for performing one or more of the above tasks.

[0065] According to the solution of the embodiment of the present disclosure, the target detection task and the occupancy grid task are executed simultaneously by sharing the feature extraction network and the encoding network. On the one hand, it can effectively reduce resource consumption, and on the other hand, the features of the two tasks are fully interactively integrated, thereby improving the effects of the two visual tasks.

[0066] In a possible implementation, step SS2032, using the second head network of the first perception model for performing the grid occupation task, and determining the detection result of the grid occupation task according to the bird's-eye view feature map, further includes the steps of:

[0067] The second head network of the first perception model is used to upsample the bird's-eye view feature map to obtain a first feature map of a preset size, wherein the preset size is larger than the original size of the bird's-eye view feature map.

[0068] According to the first feature map, a detection result of the grid occupation task is obtained.

[0069] In the disclosed embodiment, compared with the detection task, the occupancy prediction task requires a larger size of the BEV feature map, an example size of 200*200 pixels. But the problem is that the 200*200 BEV feature map will sharply increase the computational consumption of the Transformer Encoder part. In order to solve this problem, the disclosed embodiment generates a smaller BEV feature map in the Transformer encoder part, such as 50*50 pixels, to save computing resources. Then in the occ head part, the BEV feature map is upsampled and the size is adjusted to a preset size, for example, from 50*50 to 200*200. In the detection head part, the BEV feature map of the original size can be used directly.

[0070] The specific steps of upsampling can be to increase the spatial resolution of the feature map by using interpolation methods (such as bilinear interpolation) or deconvolution (Transposed Convolution), thereby expanding the horizontal and vertical resolution of the BEV viewing angle.

[0071] According to the scheme of the embodiment of the present disclosure, the encoding network outputs a BEV feature map of smaller size, which is then upsampled to meet the size requirement of the occupancy grid task, which can significantly save computing resources while ensuring the output resolution of occupancy.

[0072] In a possible implementation, the step of obtaining a detection result of an occupied grid task according to the first feature map further includes:

[0073] The height direction features of the first feature map are expanded to obtain the second feature map.

[0074] The second feature map is input into the linear layer of the second head network to obtain the detection result of the occupied grid task.

[0075] In the embodiment of the present disclosure, the height direction feature can be expanded by adding an extra channel to each grid to represent the height information of the grid in the real world, which can be achieved by adding height embedding in the BEV feature map.

[0076] The height-direction feature expansion can also be achieved by fusing feature maps at different heights, which can represent the height level of objects in a higher dimension. Different upsampling rates or convolution kernel sizes are applied to the original BEV feature map to obtain feature maps of different scales. These feature maps of different scales are combined with the upsampled BEV feature map to form a feature map containing height information.

[0077] After the height feature is expanded, a linear layer is used to predict the expanded feature map grid by grid to obtain the category information of each grid. The linear layer in this step integrates all channel information, including BEV and height features, and outputs the category distribution of each grid.

[0078] According to the solution of the embodiment of the present disclosure, in the process of upsampling the BEV feature map and expanding the height direction features, the model's understanding of the three-dimensional space is enriched by adding feature channels or introducing additional dimensional information along the height direction.

[0079] In a possible implementation, step S202 uses the encoding network of the first perception model to determine the bird's-eye view feature map according to the image features and the bird's-eye view query vector, further comprising the steps of:

[0080] S2021. Initialize a bird's-eye view query vector.

[0081] S2022. Using at least one encoder layer of the first perception model, obtain a bird's-eye view feature map according to the bird's-eye view query vector, the historical frame feature map, and the image features of the target frame surround image, wherein the historical frame feature map is obtained according to the bird's-eye view feature map of the previous surround image.

[0082] S2023. When the encoder layer also includes a next encoder layer, input the bird's-eye view feature map as a bird's-eye view query vector into the next encoder layer for updating to obtain an updated bird's-eye view feature map.

[0083] In the embodiment of the present disclosure, the first perceptual model has multiple encoder Transformer Encoder layers, and the multiple encoder layers are connected in sequence, and the output of the previous layer is used as the input of the next layer. Figure 3 As shown, a 6-layer encoder layer structure can be used, where:

[0084] Layer 1: The input is the image features of the target frame surround image and the initialized BEV query (initialized bird's-eye view query vector), and the output is the BEV query after the self-attention mechanism and feedforward network update. The initialized BEV query is pre-defined at the beginning of model training, usually a randomly initialized vector, or a representation generated by some position encoding method.

[0085] Layers 2 to 5: The input of each layer is the image features and the BEV query of the previous layer, and the output is the updated BEV query after multiple self-attention, temporal self-attention, spatial cross-attention and feedforward network processing.

[0086] Layer 6: The final output BEV query will serve as the BEV feature map of the target frame, which contains comprehensive information of spatial, temporal and image features, providing predictions for subsequent target detection or occupancy grid tasks.

[0087] Afterwards, the BEV feature map of the target frame is used as the historical frame feature map. According to the timestamp of the surround view image, the next surround view image is used as the target frame. The image features of the new target frame and the historical frame feature map are input to the first encoder layer. After the BEV feature map is output from the sixth encoder layer, the above cycle is repeated.

[0088] The above encoder structure is similar to the encoder structure provided by the BEVFormer model. Through the multi-layer Transformer encoder, the understanding of spatial and temporal context can be enhanced layer by layer, so that the final BEV feature map can be efficiently used for environmental perception tasks in autonomous driving, such as object detection, grid prediction, etc.

[0089] It should be noted that in BEVFormer, queries are a vector used in Transformer to guide the model to focus on a specific spatial position or feature. The query itself is not information directly obtained from the image, but a representation used to interact with the input features. It is associated with the image features through the self-attention mechanism of the Transformer, thereby guiding the model to learn and capture the target information. These queries are usually used to specify the spatial position of the model from a bird's-eye view, and update their own representation by interacting with the input features.

[0090] According to the solution of the embodiment of the present disclosure, in a multi-layer Transformer encoder, the query interacts with the image features through multiple self-attention calculations and gradually generates a more accurate representation, thereby improving the effect of tasks such as target detection or grid prediction.

[0091] In a possible implementation, step S2022 uses at least one encoder layer of the first perception model to obtain a bird's-eye view feature map according to the bird's-eye view query vector, the historical frame feature map, and the image features of the target frame surround image, further comprising the steps of:

[0092] The temporal self-attention module of the encoder layer is used to obtain the first fused feature map based on the bird’s-eye view query vector and the historical frame feature map.

[0093] The spatial cross attention module of the encoder layer is used to obtain the second fused feature map based on the first fused feature map and the image features of the current frame.

[0094] According to the second fused feature map, a bird's-eye view feature map is obtained.

[0095] In the disclosed embodiment, temporal self-attention is used in Transformer to fuse the BEV query of the current frame with the BEV feature map of the historical frame to obtain an enhanced BEV query, that is, a temporal BEV query that takes into account the temporal context. These enhanced BEV queries contain information from the historical frames, allowing the model to better understand the changes of objects in the temporal dimension.

[0096] The current frame BEV query is usually a sparse vector representing each grid cell. These queries are generated by embedding the spatial position and deep features at certain locations on the BEV feature map, representing the spatial information extracted from the current image frame.

[0097] Historical frame BEV feature maps refer to BEV queries extracted from one or several frames of surround view images in the past, which contain the spatial and dynamic information of historical time steps.

[0098] The temporal self-attention mechanism calculates the relationship between the current frame and the historical frames through self-attention. Each BEV query of the current frame interacts with the BEV feature map from the historical frame to capture the changes of the target at different time steps.

[0099] In this mechanism, the BEV query of the current frame is weightedly aggregated with the query vectors of the historical frames, and the weights are dynamically adjusted according to the similarity between the queries. This means that similar historical queries will have a greater impact on the update of the current query.

[0100] After being processed by the temporal self-attention mechanism, a temporal BEV query that integrates historical frame information is obtained. These queries contain historical context information, which can effectively help the model understand the target's motion trajectory, position changes, and long-term or short-term dynamic information.

[0101] Through temporal self-attention, the model can better capture the temporal characteristics of dynamic objects, such as moving vehicles, pedestrians, or other obstacles that change over time.

[0102] Similarly, the encoder layer also uses spatial cross-attention to fuse BEV queries with image features. This mechanism can effectively combine spatial information and deep features in the image to provide more accurate target localization and classification. The initialized BEV query becomes a BEV feature map after fusing the image features. For non-first-layer encoder layers, the bird's-eye view query vector, that is, the BEV query, is the BEV feature map output by the previous encoder layer, so the concepts of BEV query and BEV feature map can be used interchangeably in some cases.

[0103] It should be noted that in addition to the temporal self-attention module and the spatial cross-attention module, the multi-layer encoder layer will also pass through a fully connected feedforward neural network to further enhance its expressiveness. Usually, these feedforward networks consist of two fully connected layers, which are processed by nonlinear activation functions (such as ReLU) in the middle. The output of each layer of self-attention and feedforward network is layer normalized to ensure a stable gradient flow during training.

[0104] The multi-layer encoder layer also includes residual connections: the input and output of each layer are added through residual connections, which enables the network to avoid gradient vanishing and gradient exploding problems and promotes the training of deep networks.

[0105] from Figure 3 As can be seen in the figure, an Add&Norm layer is added after the temporal self-attention module, the spatial cross-attention module, and the feedforward network, so that residual connections and normalization operations are performed after each major step.

[0106] According to the solution of the embodiment of the present disclosure, through the self-attention mechanism and the cross-attention mechanism, the time and space information are fully utilized to generate an accurate BEV feature map, which can effectively integrate historical information and the current scene and better understand the surrounding environment.

[0107] In a possible implementation, the first head network includes a Group Detection Transformation (Group-DETR) network.

[0108] In the disclosed embodiment, the Group-DETR network is an improved Detection Transformer (DETR) architecture, which aims to improve the efficiency and accuracy of target detection. The DETR model processes image features through the Transformer structure and performs target detection tasks in an end-to-end manner. However, the original DETR has problems such as long training time and slow convergence. Group-DETR introduces a group query mechanism, which reduces the computational burden of the model, making the detection faster and performing target detection more efficiently.

[0109] According to the solution of the embodiment of the present disclosure, compared with the DETR network, Group-DETR can significantly improve the target detection effect without increasing the reasoning time.

[0110] In a possible implementation, step S201 uses a feature extraction network of the first perception model to obtain image features from the surround view image of the vehicle, further comprising the steps of:

[0111] A first image feature is determined based on the surround image of the vehicle using a torso portion of the first perception model, wherein the torso portion includes a deep layer aggregation DLA-34 network.

[0112] The neck part of the first perception model is used to obtain multi-scale features according to the first image features, wherein the neck part includes a secondary feature pyramid SecondFPN network.

[0113] According to the multi-scale features, the image features are obtained.

[0114] In the disclosed embodiment, the trunk backbone uses a Deep Layer Aggregation (DLA)-34 network pre-trained with depth estimation data (Toyota DDAD15M) to replace the commonly used ResNet-50, which reduces the complexity of the model while improving the effect. The first image features generated by the trunk are input to the neck part to further extract multi-scale features. The neck part uses a secondary feature pyramid (Second Feature Pyramid Network, SecondFPN) network to effectively fuse features of different scales for more accurate three-dimensional target detection, which can improve the overall effect of the model while maintaining performance.

[0115] According to the solution of the embodiment of the present disclosure, the detection effect of the perception task is improved by selecting the network model.

[0116] Figure 4 A flowchart of a method for training a perception model provided by an embodiment of the present disclosure is shown in FIG. Figure 4 As shown, the method comprises at least the following steps:

[0117] S401, using a feature extraction network of a second perception model to obtain sample features from a sample image, wherein the sample image is obtained using image acquisition devices at multiple different angles.

[0118] S402: Determine a bird's-eye view feature map using the encoding network of the second perception model according to sample features and a bird's-eye view query vector.

[0119] S403, using the multiple head networks of the second perception model respectively, and determining the prediction results of the perception tasks corresponding to the head networks according to the bird's-eye view feature map, wherein the multiple head networks include a first head network for performing the target detection task and a second head network for performing the grid occupation task.

[0120] S404: Determine loss values ​​of the multiple perception tasks according to the prediction results of the multiple perception tasks and the corresponding labeling results.

[0121] S405. Update the parameters of the second perception model according to the loss values ​​of the multiple perception tasks and the corresponding weight coefficients to obtain a trained target perception model.

[0122] In the disclosed embodiment, the second perception model is a multi-task pure vision model based on the Transformer architecture, which can be called DetOccFormer. In one example, the backbone of the model includes a DLA-34 network, the neck includes a SecondFPN, the encoding network part uses a multi-layer Transformer Encoder, and the first head network includes a Group-Detr network.

[0123] The second perception model can be understood as an initial or untrained model. The target perception model is a trained perception model, and the target perception model can be used as the first perception model in the above-mentioned road environment perception method embodiment.

[0124] For the perception model with multiple detection heads, one of the key steps in the training process is to design a multi-task loss function to measure the performance of each task and perform backpropagation to update the network parameters. The loss function of multi-task learning is usually the weighted sum of the losses of multiple subtasks, and the loss of each task has a different weight coefficient, which affects the final optimization process.

[0125] According to the solution of the embodiment of the present disclosure, the model can update the network parameters according to the loss of each task, effectively processing the loss weights between tasks to ensure that each task can be optimized together, thereby improving the overall performance.

[0126] In a possible implementation, S404 determines the loss values ​​of the multiple perception tasks according to the prediction results of the multiple perception tasks and the corresponding labeling results, further comprising the steps of:

[0127] S4041. Determine a first loss value for the difference between the prediction result of the target detection task and the corresponding annotation result.

[0128] S4042: For the difference between the prediction result of the occupied grid task and the corresponding annotation result, determine a second loss value through a focal loss function.

[0129] In the disclosed embodiment, the target detection task loss generally uses cross entropy loss (for classification tasks) and smooth L1 loss (for regression tasks) to measure the accuracy of object categories and the accuracy of object bounding boxes, respectively.

[0130] The occupancy grid task loss is usually based on focal loss or cross entropy loss to measure the accuracy of grid prediction.

[0131] In a possible implementation, S405 updates the parameters of the second perception model according to the loss values ​​of the plurality of perception tasks and the corresponding weight coefficients to obtain a trained target perception model, including:

[0132] S4051. Obtain a target detection task loss value according to the first loss value and the first weight coefficient.

[0133] S4052: Obtain a grid occupation task loss value according to a second loss value and a second weight coefficient, wherein the second weight coefficient is more than five times the first weight coefficient.

[0134] S4053. Obtain a total loss value according to the target detection task loss value and the grid occupation task loss value.

[0135] S4054. Update the parameters of the second perception model according to the total loss value to obtain a trained target perception model.

[0136] In the disclosed embodiment, when training a multi-task model, the loss functions are usually combined by weighted summation. The loss of each task has a weight coefficient, which is used to balance the impact between different tasks. During the training process, the model calculates the loss of each detection head and sums them up to get the total loss. Then, the gradient is calculated by gradient back propagation to update the parameters of the model.

[0137] Specifically, the loss of each detection head is calculated and back-propagated separately, but in actual operation, the losses of all tasks are processed at the same time, and the total loss is used for gradient calculation and parameter update. Because the output of each detection head usually shares part of the network layer (for example, the backbone network), the gradient update of one task will affect the feature representation of other tasks. Therefore, the loss optimization of the multi-task model is more complicated than the single-task loss optimization. For this reason, the second weight coefficient is more than 5 times the first weight coefficient. For example, the first weight coefficient is 1 and the second weight coefficient is 10.

[0138] The occupancy grid task is usually used to predict whether each grid in the environment is occupied by an obstacle. If the loss weight of this task is set to a higher value, it means that the model will pay more attention to the accuracy of the occupancy grid task, especially in multi-task learning, the importance of the occupancy grid task in the loss function is enhanced. Increasing the weight of this loss can make the model focus more on the accuracy of the occupancy grid, improving safety and real-time requirements (for example, in autonomous driving, accurately predicting the location of obstacles is crucial).

[0139] According to the solution of the embodiment of the present disclosure, setting a higher occupied grid loss weight helps to balance the effects of two or more visual tasks and ensure that the optimization progress of the occupied grid task meets expectations.

[0140] In a possible implementation, the method of the embodiment of the present disclosure further includes:

[0141] For the difference between the prediction result of the occupied grid task and the corresponding annotation result, a third loss value is determined by an affinity loss function. And / or

[0142] The fourth loss value is determined by a Lovász-softmax Loss function.

[0143] The above step S4052: obtaining the grid occupation task loss value according to the second loss value and the second weight coefficient, further includes:

[0144] The grid occupation task loss value is obtained according to the product of the second loss value and the second weight coefficient, the product of the third loss value and the third weight coefficient, and / or the product of the fourth loss value and the fourth weight coefficient.

[0145] In the disclosed embodiment, the third weight coefficient and the fourth weight coefficient may be the same as the first weight coefficient, that is, the second weight coefficient is also more than 5 times the third weight coefficient and the fourth weight coefficient.

[0146] Affinity Loss is mainly used to process pixel-level similarity or relationship learning, especially for dealing with sparse labels or unbalanced labels. Affinity Loss can effectively capture the coherence of spatial information by considering the relationship between neighborhood pixels. In the grid occupancy task, the predicted grids are often interconnected, so it is very important to strengthen the relationship learning of adjacent grids. For example, if a grid is predicted to be "occupied", the grids around it are also likely to be "occupied". This spatial relationship modeling helps to improve the accuracy of grid prediction.

[0147] Lovász-Softmax Loss is a loss function used to optimize pixel-level classification problems, especially for dealing with class imbalance problems. In the occupancy grid task, the grid annotation categories are usually highly unbalanced, with most grids being "empty" and a small number being "occupied". The traditional cross entropy loss function is sensitive to class imbalance problems, which can easily lead to obstacle areas with a small proportion of categories being predicted as empty. Lovász-Softmax Loss can significantly improve the imbalance problem by optimizing the region overlap (Intersection over Union, IoU), making the model pay more attention to the accuracy of the occupied grid, especially in difficult boundaries and sparse areas.

[0148] According to the solution of the embodiment of the present disclosure, Affinity Loss and Lovász-Softmax Loss are introduced simultaneously in the occupancy grid task. They can complement each other, reduce misclassification, improve the learning effect of sparse labels, and jointly improve the performance of the model.

[0149] In a possible implementation, step S403 uses multiple head networks of the second perception model respectively, and determines the prediction results of the perception tasks corresponding to each head network according to the bird's-eye view feature map, further comprising the steps of:

[0150] S4031. Utilize the first head network of the second perception model for performing the target detection task, and determine the prediction result of the target detection task according to the bird's-eye view feature map.

[0151] S4032. Determine a prediction result of the grid occupation task according to the bird's-eye view feature map using a second head network of the second perception model for performing the grid occupation task.

[0152] In a possible implementation, S4032 uses the second head network of the second perception model for performing the grid occupation task to determine the prediction result of the grid occupation task according to the bird's-eye view feature map, further comprising the steps of:

[0153] The second head network of the second perception model is used to upsample the bird's-eye view feature map to obtain a first feature map of a preset size, wherein the preset size is larger than the original size of the bird's-eye view feature map.

[0154] According to the first feature map, the prediction result of the grid occupation task is obtained.

[0155] In a possible implementation, obtaining a prediction result of the grid occupation task according to the first feature map further includes the steps of:

[0156] The height direction features of the first feature map are expanded to obtain the second feature map.

[0157] The second feature map is input into the linear layer of the second head network to obtain the prediction result of the occupancy grid task.

[0158] For the description of specific functions and examples of each module and sub-module of the device of the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above-mentioned road environment perception method embodiment, which will not be repeated here.

[0159] Figure 5 Schematic diagram of the structure of a road environment perception device provided by an embodiment of the present disclosure. Figure 5 As shown, the device 500 at least includes:

[0160] The first feature extraction module 501 is used to obtain image features from the vehicle's surround view image using a feature extraction network of the first perception model, wherein the surround view image is obtained using image acquisition devices at multiple different angles.

[0161] The first encoding module 502 is used to determine a bird's-eye view feature map according to image features and a bird's-eye view query vector by using an encoding network of a first perception model.

[0162] The reasoning module 503 is used to use the multiple head networks of the first perception model to determine the output results of multiple perception tasks respectively according to the bird's-eye view feature map.

[0163] In a possible implementation, the reasoning module 503 is used to:

[0164] A first head network of the first perception model for performing the target detection task is used to determine a detection result of the target detection task according to a bird's-eye view feature map.

[0165] The second head network of the first perception model for performing the grid occupation task is used to determine the detection result of the grid occupation task according to the bird's-eye view feature map.

[0166] In a possible implementation, the reasoning module 503 is used to:

[0167] The second head network of the first perception model is used to upsample the bird's-eye view feature map to obtain a first feature map of a preset size, wherein the preset size is larger than the original size of the bird's-eye view feature map.

[0168] According to the first feature map, a detection result of the grid occupation task is obtained.

[0169] In a possible implementation, the reasoning module 503 is further configured to:

[0170] The height direction features of the first feature map are expanded to obtain the second feature map.

[0171] The second feature map is input into the linear layer of the second head network to obtain the detection result of the occupied grid task.

[0172] In a possible implementation, the first encoding module 502 is used to:

[0173] Initialize the bird's-eye view query vector.

[0174] At least one encoder layer of the first perception model is used to obtain a bird's-eye view feature map according to the bird's-eye view query vector, the historical frame feature map, and the image features of the target frame surround image, wherein the historical frame feature map is obtained according to the bird's-eye view feature map of the previous surround image.

[0175] In the case that the encoder layer also includes a next encoder layer, the bird's-eye view feature map is input as a bird's-eye view query vector into the next encoder layer for updating to obtain an updated bird's-eye view feature map.

[0176] In a possible implementation, the first encoding module 502 is used to:

[0177] The temporal self-attention module of the encoder layer is used to obtain the first fused feature map based on the bird’s-eye view query vector and the historical frame feature map.

[0178] The spatial cross attention module of the encoder layer is used to obtain the second fused feature map according to the first fused feature map and the image features of the current frame.

[0179] According to the second fused feature map, a bird's-eye view feature map is obtained.

[0180] In a possible implementation, the first head network includes a Group-DETR network.

[0181] In a possible implementation, the first feature extraction module 501 is used to:

[0182] The first image feature is determined based on the surround image of the vehicle using the torso portion of the first perception model, wherein the torso portion includes a DLA-34 network.

[0183] The neck part of the first perception model is used to obtain multi-scale features according to the first image features, wherein the neck part includes a SecondFPN network.

[0184] According to the multi-scale features, the image features are obtained.

[0185] Figure 6 FIG. 1 is a schematic diagram of the structure of a training device for a perception model provided by an embodiment of the present disclosure. Figure 6 As shown, the device 600 at least includes:

[0186] A second feature extraction module 601 is used to obtain sample features from a sample image using a feature extraction network of a second perception model; wherein the sample image is obtained according to image acquisition devices at multiple different angles;

[0187] A second encoding module 602 is used to determine a bird's-eye view feature map according to the sample features and the bird's-eye view query vector using an encoding network of a second perception model;

[0188] A prediction module 603 is used to respectively use multiple head networks of the second perception model to determine the prediction results of the perception tasks corresponding to each head network according to the bird's-eye view feature map; wherein the multiple head networks include a first head network for performing the target detection task and a second head network for performing the grid occupation task;

[0189] A loss determination module 604, configured to determine loss values ​​of the plurality of perception tasks according to the prediction results of the plurality of perception tasks and the corresponding labeling results;

[0190] The parameter updating module 605 is used to update the parameters of the second perception model according to the loss values ​​of multiple perception tasks and the corresponding weight coefficients to obtain a trained target perception model.

[0191] In a possible implementation, the loss determination module 604 is used to:

[0192] Determine a first loss value for the difference between the prediction result of the target detection task and the corresponding annotation result;

[0193] For the difference between the prediction result of the occupied grid task and the corresponding annotation result, the second loss value is determined by the focal loss function.

[0194] In a possible implementation, the parameter updating module 605 is used to:

[0195] According to the first loss value and the first weight coefficient, a target detection task loss value is obtained;

[0196] According to the second loss value and the second weight coefficient, the grid occupation task loss value is obtained; wherein the second weight coefficient is more than 5 times the first weight coefficient;

[0197] According to the target detection task loss value and the grid occupation task loss value, the total loss value is obtained;

[0198] The parameters of the second perception model are updated according to the total loss value to obtain the trained target perception model.

[0199] In a possible implementation, the device of the embodiment of the present disclosure further includes:

[0200] An additional loss module, used to determine a third loss value for the difference between the prediction result of the occupied grid task and the corresponding annotation result through an affinity loss function; and / or

[0201] The fourth loss value is determined by the Lovász-softmax Loss function;

[0202] The loss determination module 604 is further configured to:

[0203] The grid occupation task loss value is obtained according to the product of the second loss value and the second weight coefficient, the product of the third loss value and the third weight coefficient, and / or the product of the fourth loss value and the fourth weight coefficient.

[0204] In a possible implementation, the prediction module 603 is used to:

[0205] Determine a prediction result of the target detection task according to the bird's-eye view feature map using a first head network of the second perception model for performing the target detection task;

[0206] A second head network of the second perception model for performing the grid occupation task is used to determine a prediction result of the grid occupation task according to the bird's-eye view feature map.

[0207] In a possible implementation, the prediction module 603 is used to:

[0208] Using a second head network of the second perception model, upsampling the bird's-eye view feature map to obtain a first feature map of a preset size; wherein the preset size is larger than an original size of the bird's-eye view feature map;

[0209] According to the first feature map, the prediction result of the grid occupation task is obtained.

[0210] In a possible implementation, the prediction module 603 is used to:

[0211] Expanding the height direction features of the first feature map to obtain a second feature map;

[0212] The second feature map is input into the linear layer of the second head network to obtain the prediction result of the occupancy grid task.

[0213] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0214] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0215] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0216] Figure 7 A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0217] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0218] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0219] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as a road environment perception method and a perception model training method. For example, in some embodiments, the road environment perception method and the perception model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the road environment perception method and the perception model training method described above may be executed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the road environment perception method and the perception model training method in any other appropriate manner (for example, by means of firmware).

[0220] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0221] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0222] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0223] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0224] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0225] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0226] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0227] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A road environment perception method, comprising: Using a feature extraction network of the first perception model, image features are obtained from a surround view image of the vehicle; wherein the surround view image is obtained based on image acquisition devices at multiple different angles; Determine a bird's-eye view feature map according to the image features and the bird's-eye view query vector using the encoding network of the first perception model; The multiple head networks of the first perception model are used to respectively determine output results of multiple perception tasks according to the bird's-eye view feature map.

2. The method according to claim 1, wherein: Using the multiple head networks of the first perception model, and according to the bird's-eye view feature map, output results of multiple perception tasks are determined respectively, including: Determine a detection result of the target detection task according to the bird's-eye view feature map using a first head network of the first perception model for performing the target detection task; A second head network of the first perception model for performing an occupation grid task is used to determine a detection result of the occupation grid task according to the bird's-eye view feature map.

3. The method according to claim 2, wherein: Determining a detection result of the grid occupation task according to the bird's-eye view feature map using the second head network of the first perception model for performing the grid occupation task includes: Using the second head network of the first perception model, upsampling the bird's-eye view feature map to obtain a first feature map of a preset size; wherein the preset size is larger than the original size of the bird's-eye view feature map; According to the first feature map, a detection result of the grid occupation task is obtained.

4. The method according to claim 3, wherein: According to the first feature map, a detection result of the grid occupation task is obtained, including: Expanding the height direction features of the first feature map to obtain a second feature map; The second feature map is input into the linear layer of the second head network to obtain a detection result of the occupied grid task.

5. The method according to claim 1, wherein: Determining a bird's-eye view feature map using the encoding network of the first perception model according to the image features and the bird's-eye view query vector includes: Initialize the bird's-eye view query vector; Using at least one encoder layer of the first perception model, a bird's-eye view feature map is obtained according to the bird's-eye view query vector, the historical frame feature map, and the image features of the target frame surround image; wherein the historical frame feature map is obtained according to the bird's-eye view feature map of the previous surround image frame; In the case that the encoder layer further includes a next encoder layer, the bird's-eye view feature map is input as a bird's-eye view query vector into the next encoder layer for updating to obtain an updated bird's-eye view feature map.

6. The method according to claim 5, wherein: Using at least one encoder layer of the first perception model, obtaining a bird's-eye view feature map according to the bird's-eye view query vector, the historical frame feature map, and the image features of the target frame surround image, including: Using the temporal self-attention module of the encoder layer, the first fused feature map is obtained based on the bird's-eye view query vector and the historical frame feature map; Using the spatial cross attention module of the encoder layer, obtaining a second fused feature map according to the first fused feature map and the image features of the current frame; A bird's-eye view feature map is obtained according to the second fused feature map.

7. The method according to claim 2, wherein: The first head network includes a Group Detection Transformation Group-DETR network.

8. The method according to claim 1, wherein: Using the feature extraction network of the first perception model, image features are obtained from the vehicle's surround view image, including: Determine a first image feature based on a surround view image of the vehicle using a torso portion of the first perception model; wherein the torso portion includes a deep layer aggregation DLA-34 network; Using the neck part of the first perception model, and according to the first image feature, a multi-scale feature is obtained; wherein the neck part includes a secondary feature pyramid SecondFPN network; Image features are obtained according to the multi-scale features.

9. A method for training a perception model, comprising: Using a feature extraction network of a second perception model, sample features are obtained from a sample image; wherein the sample image is obtained according to image acquisition devices at multiple different angles; Determine a bird's-eye view feature map according to the sample features and the bird's-eye view query vector using the encoding network of the second perception model; Using the multiple head networks of the second perception model respectively, and determining the prediction results of the perception tasks corresponding to the respective head networks according to the bird's-eye view feature map; wherein the multiple head networks include a first head network for performing the target detection task and a second head network for performing the grid occupation task; Determine the loss values ​​of multiple perception tasks according to the prediction results of multiple perception tasks and the corresponding annotation results; According to the loss values ​​and corresponding weight coefficients of the multiple perception tasks, the parameters of the second perception model are updated to obtain a trained target perception model.

10. The method according to claim 9, wherein: According to the prediction results of multiple perception tasks and the corresponding annotation results, the loss values ​​of multiple perception tasks are determined, including: Determine a first loss value for the difference between the prediction result of the target detection task and the corresponding annotation result; For the difference between the prediction result of the occupied grid task and the corresponding annotation result, the second loss value is determined by the focal loss FocalLoss function.

11. The method according to claim 10, wherein: Updating the parameters of the second perception model according to the loss values ​​of the plurality of perception tasks and the corresponding weight coefficients to obtain a trained target perception model includes: Obtaining a target detection task loss value according to the first loss value and a first weight coefficient; Obtaining a grid occupation task loss value according to the second loss value and a second weight coefficient; wherein the second weight coefficient is more than 5 times the first weight coefficient; Obtaining a total loss value according to the target detection task loss value and the grid occupation task loss value; Update the parameters of the second perception model according to the total loss value to obtain a trained target perception model.

12. The method according to claim 11, further comprising: For the difference between the prediction result of the occupied grid task and the corresponding annotation result, a third loss value is determined by an affinity loss AffinityLoss function; and / or The fourth loss value is determined by the Lovász-softmax Loss function; The step of obtaining the grid occupation task loss value according to the second loss value and the second weight coefficient includes: The grid occupation task loss value is obtained according to the product of the second loss value and the second weight coefficient, the product of the third loss value and the third weight coefficient and / or the product of the fourth loss value and the fourth weight coefficient.

13. The method according to claim 9, wherein: Using the multiple head networks of the second perception model respectively, and determining the prediction results of the perception tasks corresponding to the head networks according to the bird's-eye view feature map, includes: Determine a prediction result of the target detection task according to the bird's-eye view feature map using a first head network of the second perception model for performing the target detection task; A second head network of the second perception model for performing the occupancy grid task is used to determine a prediction result of the occupancy grid task according to the bird's-eye view feature map.

14. The method according to claim 13, wherein: Determining a prediction result of the occupancy grid task according to the bird's-eye view feature map using the second head network of the second perception model for performing the occupancy grid task includes: Using the second head network of the second perception model, upsampling the bird's-eye view feature map to obtain a first feature map of a preset size; wherein the preset size is larger than the original size of the bird's-eye view feature map; According to the first feature map, a prediction result of the grid occupation task is obtained.

15. The method according to claim 14, wherein: According to the first feature map, a prediction result of the grid occupation task is obtained, including: Expanding the height direction features of the first feature map to obtain a second feature map; The second feature map is input into the linear layer of the second head network to obtain a prediction result of the occupied grid task.

16. A road environment perception device, comprising: A first feature extraction module is used to obtain image features from a surround view image of the vehicle using a feature extraction network of a first perception model; wherein the surround view image is obtained based on image acquisition devices at multiple different angles; A first encoding module, configured to determine a bird's-eye view feature map according to the image features and the bird's-eye view query vector by using an encoding network of the first perception model; An inference module is used to use multiple head networks of the first perception model to determine output results of multiple perception tasks according to the bird's-eye view feature map.

17. The device according to claim 16, wherein: The reasoning module is used to: Determine a detection result of the target detection task according to the bird's-eye view feature map using a first head network of the first perception model for performing the target detection task; A second head network of the first perception model for performing an occupation grid task is used to determine a detection result of the occupation grid task according to the bird's-eye view feature map.

18. The device according to claim 17, wherein: The reasoning module is used to: Using the second head network of the first perception model, upsampling the bird's-eye view feature map to obtain a first feature map of a preset size; wherein the preset size is larger than the original size of the bird's-eye view feature map; According to the first feature map, a detection result of the grid occupation task is obtained.

19. The device according to claim 18, wherein: The reasoning module is also used to: Expanding the height direction features of the first feature map to obtain a second feature map; The second feature map is input into the linear layer of the second head network to obtain a detection result of the occupied grid task.

20. The device according to claim 16, wherein: The first encoding module is used for: Initialize the bird's-eye view query vector; Using at least one encoder layer of the first perception model, a bird's-eye view feature map is obtained according to the bird's-eye view query vector, the historical frame feature map, and the image features of the target frame surround image; wherein the historical frame feature map is obtained according to the bird's-eye view feature map of the previous surround image frame; In the case that the encoder layer further includes a next encoder layer, the bird's-eye view feature map is input as a bird's-eye view query vector into the next encoder layer for updating to obtain an updated bird's-eye view feature map.

21. The device according to claim 20, wherein: The first encoding module is used for: Using the temporal self-attention module of the encoder layer, the first fused feature map is obtained based on the bird's-eye view query vector and the historical frame feature map; Using the spatial cross attention module of the encoder layer, obtaining a second fused feature map according to the first fused feature map and the image features of the current frame; A bird's-eye view feature map is obtained according to the second fused feature map.

22. The device according to claim 17, wherein: The first head network includes a Group Detection Transformation Group-DETR network.

23. The device according to claim 16, wherein: The first feature extraction module is used for: Determine a first image feature based on a surround view image of the vehicle using a torso portion of the first perception model; wherein the torso portion includes a deep layer aggregation DLA-34 network; Using the neck part of the first perception model, and according to the first image feature, a multi-scale feature is obtained; wherein the neck part includes a secondary feature pyramid SecondFPN network; Image features are obtained according to the multi-scale features.

24. A training device for a perception model, comprising: A second feature extraction module is used to obtain sample features from a sample image using a feature extraction network of a second perception model; wherein the sample image is obtained according to image acquisition devices at multiple different angles; A second encoding module, configured to determine a bird's-eye view feature map according to the sample features and the bird's-eye view query vector by using an encoding network of the second perception model; A prediction module, used to respectively use multiple head networks of the second perception model and determine the prediction results of the perception tasks corresponding to each head network according to the bird's-eye view feature map; wherein the multiple head networks include a first head network for performing the target detection task and a second head network for performing the grid occupation task; A loss determination module, used to determine the loss values ​​of multiple perception tasks according to the prediction results of the multiple perception tasks and the corresponding annotation results; A parameter updating module is used to update the parameters of the second perception model according to the loss values ​​and corresponding weight coefficients of the multiple perception tasks to obtain a trained target perception model.

25. The device according to claim 24, wherein: The loss determination module is used to: Determine a first loss value for the difference between the prediction result of the target detection task and the corresponding annotation result; For the difference between the prediction result of the occupied grid task and the corresponding annotation result, the second loss value is determined by the focal loss FocalLoss function.

26. The device according to claim 25, wherein The parameter updating module is used for: Obtaining a target detection task loss value according to the first loss value and a first weight coefficient; Obtaining a grid occupation task loss value according to the second loss value and a second weight coefficient; wherein the second weight coefficient is more than 5 times the first weight coefficient; Obtaining a total loss value according to the target detection task loss value and the grid occupation task loss value; Update the parameters of the second perception model according to the total loss value to obtain a trained target perception model.

27. The apparatus according to claim 26, further comprising: An additional loss module, used to determine a third loss value for the difference between the prediction result of the occupied grid task and the corresponding annotation result through an affinity loss function; and / or The fourth loss value is determined by using the Lovász-softmax Loss function; The loss determination module is also used for: The grid occupation task loss value is obtained according to the product of the second loss value and the second weight coefficient, the product of the third loss value and the third weight coefficient and / or the product of the fourth loss value and the fourth weight coefficient.

28. The device according to claim 24, wherein: The prediction module is used to: Determine a prediction result of the target detection task according to the bird's-eye view feature map using a first head network of the second perception model for performing the target detection task; A second head network of the second perception model for performing the occupancy grid task is used to determine a prediction result of the occupancy grid task according to the bird's-eye view feature map.

29. The device according to claim 28, wherein The prediction module is used to: Using the second head network of the second perception model, upsampling the bird's-eye view feature map to obtain a first feature map of a preset size; wherein the preset size is larger than the original size of the bird's-eye view feature map; According to the first feature map, a prediction result of the grid occupation task is obtained.

30. The device according to claim 29, wherein: The prediction module is used to: Expanding the height direction features of the first feature map to obtain a second feature map; The second feature map is input into the linear layer of the second head network to obtain a prediction result of the occupied grid task.

31. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 15.

32. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-15.

33. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images