An automatic driving visual perception feature extraction method and device

Abstract

The application provides an automatic driving visual perception feature extraction method and device, raw images are collected based on multiple camera sensors around a vehicle body, and the surrounding space is divided into multiple voxel units with the vehicle body as the center; a raw image feature map of raw image features extracted by a backbone network is constructed, a preset attention module containing a time sequence attention module and a space attention module is constructed, each voxel is taken as a basic prediction unit, past multi-time features are fused through the time sequence attention module, features of the raw image feature map mapped on voxels in a three-dimensional space are mined through the space attention module, features in the 3D space are recovered from the 2D image, memories of multiple time points are fused, and spatial features with better representation ability are mined to meet learning requirements of downstream tasks of automatic driving.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method and apparatus for extracting visual perception features for autonomous driving. Background Technology

[0002] Autonomous driving is a currently popular technology field. To achieve autonomous driving and assisted driving functions, it is essential for the vehicle to perceive and extract features from its surrounding environment. Existing technologies for perceiving the vehicle's surroundings mainly include radar sensing and video acquisition.

[0003] Currently, many autonomous driving solutions rely heavily on radar to acquire depth information. While radar can directly obtain depth information of the surrounding environment, it also has some insurmountable drawbacks due to the inherent characteristics of radar sensors. These include: high manufacturing costs and short lifespan; and due to the limitations of radar sensor detection principles, the depth information it can acquire becomes sparser with distance. This can lead to inconsistencies in radar depth detection of distant objects. Furthermore, lidar detection in the height dimension is also relatively sparse. Increasing the detection density in this dimension requires increasing the number of radar lines, which undoubtedly increases hardware and computational costs. Because radar only relies on the reflection of radar waves or laser beams to calculate the distance between itself and objects, it can only acquire depth information of the surrounding environment and cannot obtain richer information such as color, texture, and light.

[0004] While camera-based video acquisition and perception solutions can compensate for the limitations of radar, they typically project the 3D real world onto 2D images. They then extract features from these images to perform downstream tasks such as object detection, drivable area detection, and lane detection. Due to the limitations of image imaging principles, they cannot explicitly obtain the depth of the surrounding environment like radar, resulting in the loss of much information. Furthermore, the perception range of a single camera is limited. Although the camera outputs a video stream, each frame does not contain information seen at previous moments; in other words, the camera has no memory and cannot perceive whether objects in the surrounding environment are static or dynamic. Existing visual perception solutions use deep neural networks to select and identify objects, but they cannot effectively perceive objects with significantly protruding parts. Moreover, the visual perception algorithm of autonomous vehicles may fail when encountering situations not included in the dataset.

[0005] To address the aforementioned issues, a new scheme for extracting visual perception features for autonomous driving is urgently needed. Summary of the Invention

[0006] In view of this, embodiments of the present invention provide a method and apparatus for extracting visual perception features for autonomous driving, so as to eliminate or improve one or more defects existing in the prior art, and solve the problems that existing autonomous driving visual perception schemes have narrow recognition range, cannot perceive spatial position and have no memory ability.

[0007] The technical solution of the present invention is as follows:

[0008] On one hand, the present invention provides a method for extracting visual perception features for autonomous driving, comprising:

[0009] Acquire raw images from multiple camera sensors around the vehicle body, and divide the three-dimensional space around the vehicle body into H×W×D cubic voxels along the length, width and height, with the vehicle body as the center and reference frame.

[0010] The raw images provided by each camera sensor are input into the backbone network for image feature extraction, and a first set number of raw image feature maps of different scales are output for each raw image.

[0011] A preset attention module is obtained, comprising a temporal attention module and a spatial attention module. At each time step, the temporal attention module takes a first query matrix and the previous time step's visual perception feature map output by the preset attention module as input. The temporal attention module concatenates the previous time step's visual perception feature map and the first query matrix to obtain a 2×(H×W×D)×C dimensional second query matrix. The second query matrix is ​​mapped through a linear layer to obtain a 2×(H×W×D)×C dimensional first value matrix. The second query matrix is ​​used to query the first value matrix and calculate attention based on a deformable multi-head attention mechanism, updating the second query matrix. The updated second query matrix is ​​then fused into a 1×(H×W×D)×C dimensional matrix. The third query matrix is ​​then joined with the first query matrix using a residual connection and regularization to obtain a 1×(H×W×D)×C dimensional fourth query matrix. The spatial attention module takes the fourth query matrix and each original image feature map as input, obtains the projection coordinates of the voxel corresponding to each query vector in the fourth query matrix on each original image, uses each original image feature map as the second value matrix, and uses the fourth query matrix to query the projection coordinate positions on the corresponding original image feature map through a multi-head attention mechanism. Attention is then calculated to obtain the fifth query matrix. The fifth query matrix and the fourth query matrix are then joined with a residual connection and regularized to obtain a 1×(H×W×D)×C dimensional visual perception feature map at the current time, which is then applied to execute downstream tasks of autonomous driving.

[0012] In the initial state, the first query matrix is ​​obtained by adding bev_embedding and pose_embedding, both of which have a dimension of 1×(H×W×D)×C. The bev_embedding is obtained through random initialization, and the pose_embedding is obtained by mapping the coordinates of each voxel through a linear layer. In the first frame, since there is no historical visual perception feature map, the second query matrix is ​​formed by concatenating two identical first query matrices with a dimension of 1×(H×W×D)×C, resulting in a dimension of 2×(H×W×D)×C. In non-first frames, the second query matrix is ​​formed by concatenating a historical visual perception feature map with a dimension of 1×(H×W×D)×C and a first query matrix with a dimension of 1×(H×W×D)×C, resulting in a dimension of 2×(H×W×D)×C. The first value matrix is ​​obtained by passing the second query matrix through a linear connection layer, and its dimension is 2×(H×W×D)×C. The first (H×W×D)×C part of the second query matrix is ​​sampled from the first (H×W×D)×C part of the first value matrix after adding the vehicle body offset from the current time to the previous time. The last (H×W×D)×C part of the second query matrix is ​​sampled from the last (H×W×D)×C part of the first value matrix.

[0013] In some embodiments, the method further includes upsampling the visual perception feature map at the current moment to improve resolution.

[0014] In some embodiments, the method includes:

[0015] During the sampling process of each query vector in the second query matrix, the voxel coordinates corresponding to each query vector in the three-dimensional space are obtained. The positions of a second set number of voxels around the corresponding voxel coordinates of each query vector in the first value matrix at the previous time are sampled, queried, and fused.

[0016] In some embodiments, each query vector samples and fuses the positions of a second predetermined number of voxels surrounding its corresponding voxel coordinates in the first value matrix, including:

[0017] A three-dimensional offset is added to the coordinates of a second set number of surrounding voxels, and the three-dimensional offset is optimized and updated in conjunction with the downstream autonomous driving task.

[0018] In some embodiments, the three-dimensional offset is obtained by mapping the corresponding query vector through a linear layer.

[0019] In some embodiments, the projection coordinates on the corresponding original image feature map are sampled and queried using the fourth query matrix through a multi-head attention mechanism, including:

[0020] The fourth query matrix is ​​used to find the projection coordinates of the voxels corresponding to the query vectors in the field of view of each camera sensor, and the projection coordinates are used to find the original image and its corresponding original image feature map.

[0021] Obtain the projection coordinates of each query vector on the original image feature map where the projection hits, and sample and query the surrounding third set number of coordinate positions and fuse them.

[0022] In some embodiments, the projected coordinates of each query vector on the original image feature map of the projected hit are obtained, and a third predetermined number of coordinate positions in the surrounding area are sampled, queried, and fused, including:

[0023] A two-dimensional offset is added to each of the predetermined number of coordinate positions in the surrounding area, and the two-dimensional offsets are optimized and updated in conjunction with the downstream tasks of the autonomous driving system.

[0024] In some embodiments, the backbone network consists of ResNet and BiFPN.

[0025] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0026] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0027] The beneficial effects of the present invention are at least as follows:

[0028] The autonomous driving visual perception feature extraction method and apparatus of the present invention acquires original images based on multiple camera sensors around the vehicle body, and divides the surrounding space into multiple voxel units with the vehicle body as the center; extracts original image feature maps of original image features in the backbone network, constructs a preset attention module including a temporal attention module and a spatial attention module, uses each voxel as the basic unit of prediction, fuses features from multiple past moments through the temporal attention module, and mines features of the original image feature maps mapped to each voxel in three-dimensional space through the spatial attention module, realizes the recovery of features in 3D space from 2D images, and fuses the memory of multiple moments to mine more representative spatial features to meet the learning needs of downstream tasks of autonomous driving.

[0029] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the text, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the written description, claims, and drawings.

[0030] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0031] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0032] Figure 1 This is a flowchart illustrating the autonomous driving visual perception feature extraction method according to an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the process for establishing a query matrix in the autonomous driving visual perception feature extraction method according to an embodiment of the present invention.

[0034] Figure 3 This is a schematic diagram of the workflow of the temporal attention module in the autonomous driving visual perception feature extraction method according to an embodiment of the present invention.

[0035] Figure 4 This is a schematic diagram of the workflow of the spatial attention module in the autonomous driving visual perception feature extraction method according to an embodiment of the present invention.

[0036] Figure 5 This is a flowchart illustrating the autonomous driving visual perception feature extraction method according to another embodiment of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0038] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0039] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0040] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0041] Solutions employing computer vision to achieve autonomous driving and related functions, such as Tesla's approach, have built a comprehensive hardware and software architecture for autonomous driving across all levels, including data, algorithms, and computing power, encompassing perception, control, and execution. Pure vision-based perception solutions mimic the principles of the human visual system, completely abandoning non-camera sensors such as LiDAR and millimeter-wave radar, relying solely on cameras for perception. These solutions utilize a shared, multi-task neural network architecture based on video stream data, possessing object depth recognition and short-term memory capabilities.

[0042] However, existing visual perception solutions can only acquire two-dimensional information for feature extraction, while the real world is three-dimensional, resulting in the loss of a lot of information when performing autonomous driving tasks.

[0043] Existing technologies typically involve mounting multiple camera probes around the vehicle body and processing them separately, without fusing all images to consider the overall picture of the environment around the vehicle. For example, an object may appear in multiple cameras; or, a single camera may not be able to capture the full picture of an object, and information from multiple cameras needs to be fused to see the entire object.

[0044] Existing visual perception solutions lack memory and cannot keep track of the continuous states of objects in the real environment. For example, they cannot identify whether objects appearing in the surrounding environment are static or dynamic. For dynamic objects, it is necessary to predict their trajectory, which is very important. For another example, if a camera sensor sees a car one moment and then the car is obscured by some obstacle the next moment, the car's position will be lost. This is very dangerous for autonomous driving, because the car may suddenly appear at some point in the future, threatening the driving safety of the autonomous vehicle.

[0045] In existing technologies, vision-based autonomous driving solutions primarily rely on object detection to perceive surrounding obstacles. These solutions typically process the raw image using a deep neural network to identify targets of interest using bounding boxes. However, if the detected target has noticeably protruding parts, such as a crane boom or a hoist arm, the neural network often excludes these protrusions to ensure accurate bounding box regression. This poses a significant safety hazard for the vehicle. The accuracy of autonomous driving perception tasks using deep learning neural networks heavily depends on the richness of the dataset. For the algorithm to be effective in all situations, the dataset must be the complete set of the real world, which is clearly impossible. Therefore, when the autonomous vehicle encounters situations not included in the dataset, the perception algorithm may fail.

[0046] Therefore, the shortcomings of existing visual perception technologies can be summarized as follows: the provided structure is 2D, while the real world is 3D; the perception tasks of multiple cameras are independent, making information sharing impossible; they cannot perceive the position of objects or other vehicles through occlusions; they cannot distinguish between static and dynamic objects, and therefore cannot obtain kinematic information of dynamic objects; they cannot accurately obtain the boundaries of highly irregularly shaped objects; and the perception algorithms of autonomous vehicles may fail when encountering situations not included in the dataset. These deficiencies severely affect downstream tasks such as autonomous driving planning and control, hinder the improvement of autonomous vehicle safety, and are problems that urgently need to be overcome in the field of autonomous driving.

[0047] In view of this, this application proposes an autonomous driving visual perception feature extraction method, which divides the 3D space around the vehicle into a series of cubic grids, similar to the concept of pixels in a 2D plane, and calls them voxels. Based on each voxel, features are further extracted from the 2D image features to predict whether there are obstacles in each voxel. The vehicle can then know which areas are drivable and which areas are dangerous.

[0048] Furthermore, predicting the kinematic properties of each voxel, such as the velocity vector and acceleration vector, can yield richer information, which will bring many benefits to autonomous driving tasks. For example, it can clearly identify which objects are static and which are dynamic, and can further predict the motion trajectory of dynamic objects.

[0049] Since it only predicts whether each voxel is filled with obstacles, it is no longer constrained by the shape of the predicted target. At the same time, even if it encounters a target not included in the dataset, although the autonomous vehicle may not be able to identify the category of the target, it can still know that there is an obstacle at this location, so the vehicle will not crash into it because it has never seen this situation before.

[0050] To enable vehicles to "remember," the solution proposed in this invention also considers temporal information. It combines the surrounding world seen at the current moment with the surrounding world seen at multiple previous moments. Thus, even if a previously seen vehicle is temporarily obscured, it can still remember its previous position and roughly predict where it has moved to.

[0051] Furthermore, to prevent multiple cameras from working independently and instead enable information sharing, the image data from multiple cameras is fused together to form a global image from a BEV (bird's-eye view) perspective. This method allows for better utilization of information from multiple cameras.

[0052] Specifically, this invention provides a method for extracting visual perception features for autonomous driving, such as... Figure 1 As shown, steps S101 to S103 are included:

[0053] Step S101: Acquire raw images provided by multiple camera sensors around the vehicle body. Using the vehicle body as the center and reference frame, divide the three-dimensional space around the vehicle body into H×W×D cubic voxels of unit volume.

[0054] Step S102: Input the raw images provided by each camera sensor into the backbone network for image feature extraction, and output a first set number of raw image feature maps of different scales for each raw image.

[0055] Step S103: Obtain a preset attention module, which includes a temporal attention module and a spatial attention module. At each time step, the temporal attention module takes the first query matrix and the visual perception feature map of the previous time step output by the preset attention module as input. The temporal attention module concatenates the visual perception feature map of the previous time step and the first query matrix to obtain a 2×(H×W×D)×C dimensional second query matrix. The second query matrix is ​​mapped through a linear layer to obtain a 2×(H×W×D)×C dimensional first value matrix. The second query matrix is ​​used to query the first value matrix and calculate the attention based on a deformable multi-head attention mechanism, and the second query matrix is ​​updated. Its dimension is still 2×(H×W×D)×C. The updated second query matrix is ​​then merged into a 1×(H×W×D)×C matrix. The third query matrix is ​​1×(H×W×D)×C dimensional. The third query matrix is ​​residually concatenated with the first query matrix and regularized to obtain the fourth query matrix of 1×(H×W×D)×C dimensional. The spatial attention module takes the fourth query matrix and each original image feature map as input, obtains the projection coordinates of the voxel corresponding to each query vector in the fourth query matrix on each original image, uses each original image feature map as the second value matrix, and uses the fourth query matrix to query the projection coordinate position on the corresponding original image feature map through a multi-head attention mechanism, calculates the attention to obtain the fifth query matrix, and performs residual concatenation and regularization on the fifth query matrix and the fourth query matrix to obtain the 1×(H×W×D)×C dimensional visual perception feature map at the current time, which is applied to perform downstream tasks of autonomous driving.

[0056] In the initial state, the first query matrix is ​​obtained by adding bev_embedding and pose_embedding, both of which have a dimension of 1×(H×W×D)×C. bev_embedding is obtained through random initialization, while pose_embedding is obtained by mapping the voxel coordinates through a linear layer. In the first frame, since there is no historical visual perception feature map, the second query matrix is ​​formed by concatenating two identical first query matrices with a dimension of 1×(H×W×D)×C, resulting in a dimension of 2×(H×W×D)×C. In non-first frames, the second query matrix is ​​formed by concatenating a historical visual perception feature map with a dimension of 1×(H×W×D)×C and a first query matrix with a dimension of 1×(H×W×D)×C, resulting in a dimension of 2×(H×W×D)×C. The first value matrix is ​​obtained by passing the second query matrix through a linear connection layer, with a dimension of 2×(H×W×D)×C.

[0057] In step S101, camera sensors are pre-installed around the vehicle body, the relative positions of each camera sensor are fixed, and the viewing angles of each camera sensor can be stitched together to form a bird's-eye view (BEV). For example, six camera sensors can be set, one each on the middle left and right sides of the front of the vehicle, and one each on the middle left and right sides of the rear of the vehicle.

[0058] Simultaneously, the space surrounding the vehicle body is divided into multiple cubic voxels, similar to pixels in a two-dimensional image, and feature mining is performed. Specifically, the three-dimensional space around the vehicle body is divided into H×W×D voxels according to length, width, and height. The side length of the voxels is set according to specific needs. For example, in real space, a BEV space with a length, width, and height of 100m, 100m, and 10m is constructed, and H, W, and D are set to 200, 200, and 20 respectively. Then, the side length of the voxel is 0.5m, and the vehicle body is the center of this BEV space. Since the relative positions of each camera sensor and the vehicle body are fixed, each voxel can be associated with each original image.

[0059] In step S102, images are acquired by each camera sensor, with the sampling times of each camera sensor aligned. Original images from each viewpoint are acquired at each time and input into the backbone network for feature extraction. In some embodiments, the backbone network consists of ResNet and BiFPN. Each original image is processed by the backbone network to extract feature maps at multiple scales. For example, four feature maps at different scales can be extracted from each image, so the six original images from the six camera sensors at each time step can generate 6×4 feature maps.

[0060] In step S103, the input for the temporal attention module to extract features through the self-attention mechanism includes two parts: one is the final output of the pre-defined attention module at historical time points, which is the visual perception feature map, i.e., the BEV feature map. The BEV feature map at time t is denoted as B. t On the other hand, there is the query matrix Q. At time t, B is queried through the query matrix Q. t-1 The purpose of the query is to integrate the current query with historical information. The input to the spatial attention module is the original image feature maps at multiple scales extracted from each original image by the backbone network. The spatial attention module uses a cross-attention mechanism to fuse the query matrix Q, which incorporates historical information, with the input original image feature maps at multiple scales, thereby allowing the query matrix Q to obtain the information collected by each camera sensor at the current moment.

[0061] like Figure 2As shown, the query matrix Q is a predefined set of grid-type learnable parameters with dimensions H×W×D×C. H, W, and D represent the number of voxels into which the 3D BEV space is divided along the x, y, and z axes, respectively. After being learned by two attention modules, the query matrix Q can fuse temporal and spatial information. The BEV query matrix Q after fusing this information becomes the BEV feature map, which is the output of the entire network model, and is then used for other downstream tasks. Each voxel corresponds to a query vector q of length C, recording voxel information. The query matrix Q input to the temporal attention module is denoted as the first query matrix. In the initial state, the first query matrix is ​​obtained by adding bev_embedding and pose_embedding. Both bev_embedding and pose_embedding have dimensions of 1×(H×W×D)×C. bev_embedding is obtained through random initialization, and pose_embedding is obtained by mapping the coordinates of each voxel through a linear layer. Both bev_embedding and pose_embedding are learnable.

[0062] like Figure 3 As shown, in the temporal attention module, the temporal attention module will process the visual perception feature map B from the previous time step. t-1 Concatenate with the first query matrix Q to obtain a 2×(H×W×D)×C dimensional second query matrix Q. ′ Q at this time ′ =[B t-1 [Q]; The second query matrix Q ′ The first value matrix, Value, is obtained by performing a linear mapping through the Linear layer. Value = [Linear(B t-1 ,Q)], the dimension is 2×(H×W×D)×C, where, Value[0]=Linear(B t-1 Value[1] = Linear(Q). For the initial state, the first value matrix Value = [Linear(Q,Q)], and the second query matrix Q... ′ =[Q, Q]. Query matrix Q ′ The Value is queried, and attention is calculated. The updated second query matrix is ​​then merged into a third query matrix of 1×(H×W×D)×C dimensions.

[0063] Since the vehicle body shifts and deflects at times t-1 and t, the positions of the voxels in the query matrix at time t change compared to time t-1, requiring correction. This is done using the query matrix Q. ′ During the query for Value, Q in the second query matrix ′In the [0] part, with the vehicle body offset from the current time to the previous time added, the Value[0] in the first value matrix, which is mapped from the visual perception feature map of the previous time through a linear layer, is sampled; Q in the second query matrix ′ [1] In this part, the Value[1] in the first value matrix, which is mapped from the first query matrix through a linear layer, is sampled. Here, the query matrix Q ′ In practice, the visual perception feature map output by the preset attention module at time t-1 and the query Q at time t are queried simultaneously. To ensure that all positions in the real space at times t-1 and t are aligned during the query process, the sampling position of the query vector corresponding to each voxel at times t-1 and t needs to be corrected during sampling. The real position of each voxel in space is different at times t-1 and t. The coordinates (x, y, z) of each voxel in the 3DBEV space are denoted as Ref_3D. The Ref_3D at time t is calculated by combining the Ref_3D at time t-1 with the information of the vehicle body rotation angle and offset. The change in the vehicle body from time t to time t-1 is denoted as Shift. The coordinates of each voxel at times t-1 and t are recorded using Ref_point. For the query matrix Q at time t... ′ The coordinates of the sampling point relative to the real space are actually Ref_point = [Ref_3D + Shift, Ref_3D]. In the initial state, since the first value matrix Value = [Linear(Q,Q)] and the second query matrix Q... ′ = [Q, Q], then the actual query position in the real space is Ref_point = [Ref_3D, Ref_3D].

[0064] However, in reality, during the deflection process, in addition to the displacement and deflection of the vehicle body, the positions of dynamic objects in the surrounding environment also change. Therefore, during the sampling process of each query vector in the second query matrix, the voxel coordinates corresponding to each query vector in three-dimensional space are obtained. Each query vector samples and queries the positions of a second predetermined number of voxels surrounding its corresponding voxel coordinates in the visual perception feature map at the previous moment and then fuses them. Furthermore, a three-dimensional offset is added to the coordinates of the second predetermined number of surrounding voxels, and the three-dimensional offset is optimized and updated in conjunction with the downstream tasks of autonomous driving. In some embodiments, the three-dimensional offset is obtained by mapping the corresponding query vector through a linear layer.

[0065] Finally, the updated second query matrix is ​​fused into a 1×(H×W×D)×C dimensional third query matrix; the third query matrix is ​​then residually joined with the first query matrix and regularized to obtain a 1×(H×W×D)×C dimensional fourth query matrix, denoted as Q1, which is used as the output of the temporal attention module and input into the spatial attention module.

[0066] like Figure 4 As shown, in the spatial attention module, the 2D original image feature map needs to be queried through the fourth query matrix. Therefore, the voxel corresponding to each query vector in the fourth query matrix is ​​projected onto the original images acquired by each camera sensor to obtain the projection coordinates of the corresponding voxel on each original image. However, usually, each voxel will only be projected onto one or two original images. Then, the feature maps of the original images at different scales can be queried according to the corresponding projection coordinates and scaled. If two original images are hit, one can be selected, or features can be extracted from the two images separately and fused. For example, based on 6 camera sensors, the projection coordinates of a voxel in each original image plane are Projection_2D_1...Projection_2D_6. Usually, only 1 or 2 of these 6 projection points (denoted as Ref_2D) can fall within the image of the corresponding camera. Therefore, the positions near Ref_2D are sampled. Then, according to the scaling ratio of the feature map at different scales relative to the original image, the corresponding position of Ref_2D is found on the feature map at each scale.

[0067] In some embodiments, a multi-head attention mechanism is used to sample and query the projected coordinate positions on the corresponding original image feature maps using the fourth query matrix. This includes: obtaining the projected coordinates of the voxels corresponding to the query vectors in the fourth query matrix within the field of view of each camera sensor, and finding the projected coordinates to the matched original image and its corresponding original image feature map; obtaining the projected coordinate positions of each query vector on the matched original image feature map, and sampling and querying a third predetermined number of surrounding coordinate positions and fusing them. In some embodiments, a two-dimensional offset is added to each of the third predetermined number of surrounding coordinate positions, and the two-dimensional offsets are optimized and updated in conjunction with downstream autonomous driving tasks.

[0068] Finally, the fifth query matrix is ​​updated by querying and calculating attention. The fifth query matrix and the fourth query matrix are then joined by residuals and regularized to obtain a 1×(H×W×D)×C dimensional visual perception feature map at the current time, which is then applied to perform downstream tasks of autonomous driving.

[0069] In some embodiments, after step S103, the method further includes: upsampling the visual perception feature map at the current time to improve the resolution.

[0070] On the other hand, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0071] On the other hand, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0072] The present invention will now be described in conjunction with specific embodiments:

[0073] This embodiment provides a method for extracting visual perception features for autonomous driving, such as... Figure 5 As shown, on the left side, six camera sensors are used to provide the raw image input. These six cameras are distributed around the vehicle body, located at the front, right front, right rear, rear, left rear, and left front positions, respectively, to capture surround view images of the vehicle body. The collaborative working process of these six cameras is as follows: First, each of the six cameras generates an image frame. Then, after time synchronization, these images are input into the network proposed in this embodiment. After the network processes these six images, they are transformed from the original surround view images into BEV feature maps from the BEV's perspective, i.e., visual perception feature maps.

[0074] The six images captured by the camera are fed into six backbone networks composed of ResNet and BiFPN for feature extraction. Each image yields four feature maps at different scales, resulting in a total of 6×4 feature maps for the six images.

[0075] Next, we proceed to the preset attention module, which consists of two parts: a temporal attention module and a spatial attention module. The temporal attention module primarily receives two inputs: one is the BEV feature map from historical time points (e.g., if the BEV feature map at time t is denoted as B...). t Therefore, the BEV feature map at a historical moment is the BEV feature map at time t-1, denoted as B. t-1 The other part is the BEV query matrix Q. In the temporal attention module, the query matrix Q will use a self-attention mechanism to process the BEV feature map B at time t-1. t-1 The significance of performing this query lies in integrating the current query matrix Q with historical information. The spatial attention module's main inputs are the multi-scale feature maps extracted from the backbone network and the output of the temporal attention module. In this module, the network uses a cross-attention mechanism to fuse the temporally fused query matrix Q with the input multi-scale feature maps, thus allowing the query matrix Q to incorporate information captured by each camera at the current moment.

[0076] A predefined set of learnable grid-like parameters Q is used, with dimensions H×W×D×C. H, W, and D represent the number of grids into which the 3D BEV space is divided along the x, y, and z directions, respectively. Each grid is called a voxel. Each voxel contains a query vector q to record its information, and the length of this query vector q is C. Each voxel in the 3D BEV space corresponds to s meters in the real world. If we want to construct a BEV space with a length, width, and height of 100m, 100m, and 10m respectively, and H, W, and D are set to 200, 200, and 20 respectively, then s would be 100m / 200, which equals 0.5m. The center position of the BEV query matrix Q corresponds to the position of the vehicle by default.

[0077] Before being input into the temporal attention module for the first time, the BEV query matrix Q needs to be pre-initialized. It consists of two parts: a randomly initialized bev_embedding with dimensions H×W×D×C, and a pose_embedding, i.e., position encoding, which is obtained by mapping the 3D coordinates of each voxel in the entire BEV space to a C-dimensional vector through a linear layer, also with dimensions H×W×D×C. Then, bev_embedding and pose_embedding are added together to obtain the initialized BEV query matrix Q. It should be noted that both bev_embedding and pose_embedding that make up the BEV query matrix Q are learnable. Therefore, after being learned by the two attention modules, the BEV query matrix Q can fuse temporal and spatial information. The BEV query matrix Q after fusing this information becomes the BEV feature map, which is the output of the entire network model, and is then used for other downstream tasks.

[0078] To enable the model to "memorize," a temporal attention module is introduced. This module receives the previous time step's BEV feature map and an initialized BEV query matrix Q as input. Although it only inputs the BEV feature map at time t-1, it borrows the idea of ​​an RNN (Recurrent Neural Network), so this feature map theoretically contains information from all historical time steps. Furthermore, considering that information from one second ago and information from thirty seconds ago have different impacts on the current time step, the weights of the BEV feature maps at different time steps decay as time progresses.

[0079] Having clarified the two inputs to the temporal attention component, the main functions of this structure will be explained in detail below: ① How the BEV query matrix Q fuses the prior information of the BEV feature map at time t-1. ② How the BEV query matrix Q interacts with itself through a self-attention mechanism.

[0080] First, define a few variables: Ref_3D: the coordinates (x, y, z) of each voxel in the 3D BEV space at time t.

[0081] Shift: The body changes from time t to time t-1, calculated using information about the body rotation angle and offset from the CANbus.

[0082] Ref_point: The coordinates (x, y, z) of a voxel in 3D BEV space. This coordinate has two parts: one part corresponds to the coordinate at time t-1, and the other part corresponds to the coordinate at time t.

[0083] Value: In the transformer structure, this specifically refers to the feature value at a certain location in the feature map. In this embodiment, it is the feature queried by the query matrix Q.

[0084] First, the BEV feature map at time t-1 is... t-1 Align the vehicle's own motion parameters (such as information on body rotation angle and offset from CANbus data) with the BEV feature map at time t, i.e., the query matrix Q, so that features in the same voxel correspond to the same positions in the real world. The aligned BEV feature map at time t-1 is still denoted as B. t-1 The query matrix can also be corrected during subsequent queries. Since precise alignment is not possible during this process, it needs to be corrected through the network's own learning. The specific steps are as follows:

[0085] For all frames except the first frame (the first frame does not have the BEV feature map from the previous time step as input), firstly B... t-1 The query matrix Q is concatenated with the query matrix Q to obtain a new query matrix, denoted as Q. ′ Q at this time ′ =[B t-1 [Q], with dimensions H×W×D×2C. Then add B... t-1 After concatenating Q with it, the result goes through a Linear layer to obtain Value. At this point, Value = [Linear(B...]. t-1 [,Q)], with dimensions 2×(H×W×D)×C, where Linear(x) represents a linear mapping of x through a Linear layer, which does not change B. t-1And the dimensions of Q. Then find the aligned position Ref_point of each BEV query vector q, where Ref_point = [Ref_3D, Ref_3D + Shift], and the dimensions are H×W×D×6. For the first frame image, Value = [Linear(Q,Q)], Ref_point = [Ref_3D, Ref_3D], Q ′ =[Q, Q]. Then query matrix Q. ′ The system queries the Values ​​and computes attention. Unlike traditional global attention mechanisms, the deformable attention mechanism does not compute attention between each query vector q and all Values, as this would lead to unacceptable computational complexity and slow training convergence. In this deformable attention mechanism, each query matrix Q... ′ Each query vector q in the algorithm will have K sampling points sample_points around its corresponding Ref_point. For each query vector q of length C, it will have a corresponding coordinate in the Ref_point. ′ Each q in [0] will find the Ref_3D+Shift position in Value[0] and sample the K sample_points around it; similarly, Q ′Each q in [1] will also find the Ref_3D position in Value[1] and sample the K sample_points around it. However, the initially selected K sample_points are not necessarily the points most relevant to the current query vector q, so a 3D offset sample_offset will be predicted for each of the K sample_points. This sample_offset is a learnable parameter that will be updated during the backpropagation process of the network. It is obtained by passing each q through a Linear layer. Each updated sample_offset will be applied to its corresponding sample_point to help each q shift its attention to the position most relevant to it. In addition, for each query vector q, in order to realize the function of using multiple convolutional kernels in CNN and reduce the number of parameters, we use a multi-head attention mechanism. In each single-head attention mechanism, we sample at K sample_point positions. Assuming there are M heads, for each query vector q, it needs to fuse K×M values. To more effectively fuse these features, a weighted sum is used for these K×M values. By passing each query vector q through a Linear layer, the weights of these K×M values ​​can be predicted, and these weights are also learnable. Each query vector q updates itself with the weighted sum of these K×M values ​​after passing through a temporal attention module. This operation is performed on each q, resulting in the entire query matrix Q. ′ After passing through a temporal attention module, historical BEV features are incorporated and interacted with the module's own information. Finally, this updated 2×(H×W×D)×C dimensional query matrix Q is... ′ The average value is calculated along the first dimension to integrate historical and current information, resulting in an updated 1×(H×W×D)×C dimensional query matrix Q. This updated Q is then residually concatenated with the query matrix Q before it was updated by the temporal attention module. Finally, after passing through a regularization layer, the final output of the temporal attention module is obtained: a 1×(H×W×D)×C dimensional query matrix, denoted as Q1. This generated Q1 can then be fed into the subsequent spatial attention module.

[0086] In the spatial attention module, the Q1 output from the temporal attention is used to query the multi-scale look-around feature maps extracted by the backbone network, generating a BEV feature map in the BEV space. In fact, the spatial attention part has many similarities to the temporal attention part described above. However, its difference lies in the fact that it does not need to use historical BEV features B. t-1 .

[0087] The temporal attention module uses a self-attention mechanism, where the value is obtained through a linear mapping of Q. The spatial attention module, however, uses a cross-attention mechanism, where the value is a feature map extracted from the loop image by the backbone network. Finally, the methods for obtaining Q differ. Since Q is not generated before entering the first temporal attention module, it is obtained by randomly initializing a vector and adding positional encoding. In the spatial attention module, the output Q from the temporal attention module is already available, so further initialization of Q is unnecessary.

[0088] The specific process of the spatial attention module will be explained below:

[0089] This section uses a deformable cross-head attention mechanism. Analogous to the temporal attention module above, the Value here is the multi-scale feature map of the 6 images captured by the camera. The query matrix Q is Q1, the output of the temporal attention module. First, for each voxel containing the query vector q in the query matrix Q, its 3D coordinates in the BEV space are still denoted as Ref_3D. Then, using the intrinsic parameters of each camera and the extrinsic parameters of the camera coordinate system relative to the vehicle coordinate system, the coordinates of this Ref_3D projected onto each image coordinate system can be obtained. Here, we assume there are 6 cameras, so we can obtain 6 different image coordinate systems, denoted as Projection_2D_1...Projection_2D_6. Usually, only 1 to 2 of these 6 projection points (denoted as Ref_2D) can fall within the image of the corresponding camera. The camera views with these projection points are called hit views. Therefore, only the positions near these Ref_2D are sampled. Then, based on the scaling ratio of the feature map at different scales relative to the original image, the corresponding position of this Ref_2D is found on the feature map at each scale. This determines where each query vector q should be sampled in the feature maps at different scales. Similar to the temporal attention module, for each query vector q, K sample_points are sampled near the Ref_2D position on the image feature map at each scale. The number of sample_points for each q is then K×M×L×n, where M is the number of heads in the multi-head attention mechanism, L is the number of levels in the multi-scale feature maps, and n is the number of Ref_2Ds. Then, a sample_offset is predicted for each sample_point, resulting in a sample_offset of K×M×L×n×2 dimensions for each q, where 2 represents the coordinate offset in 2D space. This sample_offset is learnable and applied to the corresponding sample_point. The values ​​at all sample_point positions are then weighted and summed to update the value of the corresponding query vector q. The weights here are also learnable. By performing the above operations on each query vector q, the entire query matrix Q can fuse the multi-scale features of the panoramic image into the 3D BEV space. Then, similar to the temporal attention module, the 1×(H×W×D)×C dimension query matrix Q after spatial fusion is joined with the input query matrix Q without spatial fusion using a residual concatenation. The result of the residual operation is then passed through a regularization layer to obtain the final output.

[0090] The overall workflow of the attention module is built by repeatedly stacking the temporal and spatial attention modules described above. Connecting a temporal attention module to a spatial attention module yields an output matrix, denoted as Q. Passing this Q through a feedforward neural network yields another output, denoted as Q'. ′ Then put Q and Q ′ Residual connections are performed, and the results after residual connections are then passed through a regularization layer, thus completing one complete attention module. Repeating this stacking structure multiple times (the number of stacks can be balanced between accuracy and speed) yields the BEV feature map B at time t. t .

[0091] The resolution of the 3D BEV feature map obtained after the spatiotemporal fusion attention module is relatively low. Therefore, this feature map is upsampled to obtain a higher resolution 3D feature map. The resolution of this feature map should basically meet the accuracy requirements of autonomous driving perception tasks. Based on this high-resolution feature map, it is possible to predict whether there is an object in each voxel in 3D space, as well as the kinematic characteristics of the object in each voxel.

[0092] The autonomous driving perception scheme proposed in this embodiment uses only a camera as the pure visual input of the perception module. Compared with the common autonomous driving perception schemes on the market that use cameras plus radar and other pure vision solutions, this scheme has certain advantages.

[0093] Compared to cameras, the various automotive radars used in autonomous driving are much more expensive. The solution proposed in this embodiment is based on pure vision, using only a camera as the sole perception sensor, thus eliminating the hardware costs of radar and achieving greater economic efficiency.

[0094] Currently, most autonomous driving solutions employ a camera-radar fusion approach because cameras alone struggle to obtain depth information about the surrounding environment, a deficiency that radar effectively addresses. However, the solution in this embodiment can accomplish tasks previously requiring radar using only a camera. Furthermore, due to the inherent limitations of radar's operating principle, it cannot obtain dense depth information about distant objects, while cameras can capture richer features. Therefore, this embodiment, using only a less expensive camera, not only effectively obtains depth information about the surrounding environment but also, to some extent, compensates for the limitations of radar.

[0095] Compared to other purely vision-based solutions, the solution in this embodiment largely overcomes the drawbacks of using only images as input. Even with only 2D images, this embodiment can recover the 3D real world from 2D images. Furthermore, through a spatial fusion module, this embodiment effectively solves the problem of information sharing between different cameras. Abandoning the original detection method based on the entire target, this embodiment uses each tiny voxel as the basic unit of prediction. By determining whether an object exists within this voxel, this embodiment effectively solves the problem that the predicted bounding box can only be rectangular. Simultaneously, since this embodiment focuses on "object presence" and "object absence," the model remains effective even when encountering situations where the object does not exist in the dataset. Moreover, by incorporating temporal information, this allows autonomous vehicles to not only distinguish between static and dynamic objects but also possess a certain "memory" capability, remembering previously seen information for a period of time.

[0096] This invention proposes a deep learning neural network based on the presence of objects within voxels, which can be used as a pure visual perception solution for autonomous driving. Building upon this, to fuse information from multiple cameras and give the network "memory" capabilities, spatial and temporal fusion modules based on attention mechanisms are added. Ultimately, we obtain a 3D feature matrix storing whether an object exists in every tiny voxel around the vehicle, and the kinematic characteristics (velocity vector, acceleration vector, etc.) of the object in each voxel. The key innovations of this invention are: 1) Mesh segmentation of the 3D world, with each segmented unit called a voxel, and then predicting the presence of an object in each voxel. 2) Employing a spatial attention module to fuse images from multiple cameras to achieve information sharing between cameras and the construction of the entire 3D scene. 3) Employing a temporal attention module to give the model "memory" capabilities and the ability to distinguish between dynamic and static objects.

[0097] In summary, the autonomous driving visual perception feature extraction method and apparatus of the present invention acquires original images based on multiple camera sensors around the vehicle body, and divides the surrounding space into multiple voxel units with the vehicle body as the center; extracts original image feature maps of original image features in the backbone network, constructs a preset attention module including a temporal attention module and a spatial attention module, uses each voxel as the basic unit of prediction, fuses features from multiple past moments through the temporal attention module, and mines features of the original image feature maps mapped onto each voxel in three-dimensional space through the spatial attention module, thereby realizing the recovery of features in 3D space from 2D images, and fusing memories from multiple moments to mine more representative spatial features to meet the learning needs of downstream autonomous driving tasks.

[0098] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0099] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0100] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0101] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for extracting visual perception features for autonomous driving, characterized in that, include: Acquire raw images from multiple camera sensors around the vehicle body, and divide the three-dimensional space around the vehicle body into H×W×D cubic voxels along the length, width and height, with the vehicle body as the center and reference frame. The raw images provided by each camera sensor are input into the backbone network for image feature extraction, and a first set number of raw image feature maps of different scales are output for each raw image. A preset attention module is obtained, which includes a temporal attention module and a spatial attention module. At each time step, the temporal attention module takes a first query matrix and the visual perception feature map of the previous time step output by the preset attention module as input. The temporal attention module concatenates the visual perception feature map of the previous time step and the first query matrix to obtain a 2×(H×W×D)×C dimensional second query matrix. The second query matrix is ​​mapped through a linear layer to obtain a 2×(H×W×D)×C dimensional first value matrix. The second query matrix is ​​used to query the first value matrix and calculate the attention based on a deformable multi-head attention mechanism, and the second query matrix is ​​updated. The updated 2×(H×W×D)×C dimensional second query matrix is ​​averaged in the first dimension, and then merged to generate a 1×(H×W×D)×C dimensional third query matrix; The third query matrix is ​​residually joined with the first query matrix and regularized to obtain a 1×(H×W×D)×C dimensional fourth query matrix. The spatial attention module takes the fourth query matrix and each original image feature map as input, obtains the projection coordinates of the voxel corresponding to each query vector in the fourth query matrix on each original image, uses each original image feature map as the second value matrix, and uses the fourth query matrix to query the projection coordinate position on the corresponding original image feature map through a multi-head attention mechanism, calculates the attention to obtain the fifth query matrix, and performs residual joining and regularization on the fifth query matrix and the fourth query matrix to obtain a 1×(H×W×D)×C dimensional visual perception feature map at the current time, which is then applied to perform downstream tasks of autonomous driving. In the initial state, the first query matrix is ​​obtained by adding bev_embedding and pose_embedding, both of which have a dimension of 1×(H×W×D)×C. bev_embedding is obtained through random initialization, and pose_embedding is obtained by mapping the coordinates of each voxel through a linear layer. In the first frame, since there is no historical visual perception feature map, the second query matrix is ​​formed by concatenating the two identical first query matrices with a dimension of 1×(H×W×D)×C, resulting in a dimension of 2×(H×W×D)×C. However, in non-first frames... The second query matrix is ​​formed by concatenating a historical visual perception feature map with a dimension of 1×(H×W×D)×C and a first query matrix with a dimension of 1×(H×W×D)×C. After concatenation, its dimension remains 2×(H×W×D)×C. The first value matrix is ​​obtained by passing the second query matrix through a linear connection layer, and its dimension is 2×(H×W×D)×C. The first (H×W×D)×C part of the second query matrix is ​​sampled from the first (H×W×D)×C part of the first value matrix after adding the vehicle body offset from the current time to the previous time. The last (H×W×D)×C part of the second query matrix is ​​sampled from the last (H×W×D)×C part of the first value matrix.

2. The method for extracting visual perception features for autonomous driving according to claim 1, characterized in that, The method further includes: The visual perception feature map at the current moment is upsampled to improve resolution.

3. The method for extracting visual perception features for autonomous driving according to claim 1, characterized in that, In the method: During the sampling process of each query vector in the second query matrix, the voxel coordinates corresponding to each query vector in three-dimensional space are obtained. The positions of a second set number of voxels around the corresponding voxel coordinates of each query vector in the first value matrix are sampled and fused.

4. The method for extracting visual perception features for autonomous driving according to claim 3, characterized in that, Each query vector samples and merges the positions of a second predetermined number of voxels surrounding its corresponding voxel coordinates in the first value matrix, including: A three-dimensional offset is added to the coordinates of a second set number of surrounding voxels, and the three-dimensional offset is optimized and updated in conjunction with the downstream autonomous driving task.

5. The method for extracting visual perception features for autonomous driving according to claim 4, characterized in that, The three-dimensional offset is obtained by mapping the corresponding query vector through a linear layer.

6. The method for extracting visual perception features for autonomous driving according to claim 1, characterized in that, The multi-head attention mechanism utilizes the fourth query matrix to sample and query the projected coordinate positions on the corresponding original image feature map, including: The fourth query matrix is ​​used to find the projection coordinates of the voxels corresponding to the query vectors in the field of view of each camera sensor, and the projection coordinates are used to find the original image and its corresponding original image feature map. Obtain the projection coordinates of each query vector on the original image feature map where the projection hits, and sample and query the surrounding third set number of coordinate positions and fuse them.

7. The method for extracting visual perception features for autonomous driving according to claim 6, characterized in that, Obtain the projected coordinates of each query vector on the original image feature map where the projection hits, and sample and fuse the surrounding third set number of coordinate positions, including: A two-dimensional offset is added to each of the three predetermined coordinate positions in the surrounding area, and the two-dimensional offset is optimized and updated in conjunction with the downstream tasks of the autonomous driving system.

8. The method for extracting visual perception features for autonomous driving according to claim 1, characterized in that, The backbone network consists of ResNet and BiFPN.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 8.

Images