Target recognition method and device based on BEV perspective, storage medium and equipment

Abstract

The application discloses a target recognition method and device based on a BEV perspective, a storage medium and equipment, and belongs to the technical field of unmanned driving. The method comprises the following steps: acquiring a multi-path ring view, wherein the multi-path ring view is composed of images obtained by simultaneously photographing a same scene by multiple cameras with different horizontal photographing perspectives; extracting a feature map from the multi-path ring view, generating a feature vector according to the feature map; calculating a feature coefficient vector according to the feature vector and an activation function, and calculating a fused feature map according to the feature map and the feature coefficient vector, wherein the feature coefficient vector represents the importance of different channels in the feature map within the global range of multiple perspectives; generating depth information and a target feature information map according to the fused feature map; creating a BEV feature map under the BEV perspective according to the depth information and the target feature information map, and performing target recognition according to the BEV feature map. After the feature vectors of the multi-path ring view are fused, the target recognition accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, and in particular to a target recognition method, device, storage medium and equipment based on the BEV perspective. Background Technology

[0002] In the field of autonomous driving, the commonly used sensors in mainstream perception solutions are LiDAR and cameras. LiDAR offers reliable 3D detection capabilities, but perception methods based on multi-view cameras have received significant attention due to their lower cost. Furthermore, after long-term iteration, relatively mature technical solutions for using cameras have been developed. For example... Figure 1 As shown, the perception task takes the above multi-view images as input to obtain information about the target obstacle, including the coordinates, size, orientation, and other information of the target's surrounding 3D bounding box.

[0003] Commonly used image-based 3D target recognition algorithms include BEVDet, BEVDepth, and BEVFormer. These algorithms all take multi-view images as input, extract feature vectors and depth information from the multi-view images, and combine the feature vectors and depth information to recognize target information in a unified BEV (Bird's-Eye View) perspective.

[0004] However, the relevant technologies directly concatenate the feature vectors of the multi-view loop into the feature vectors of the BEV view, which cannot achieve feature fusion and results in low target recognition accuracy. Summary of the Invention

[0005] This application provides a target recognition method, apparatus, storage medium, and device based on the BEV (Browser Active Vehicle) perspective, to address the problem of low target recognition accuracy in related technologies. The technical solution is as follows:

[0006] According to a first aspect of this application, a target recognition method based on a BEV (Browser-Electronic Vehicle) perspective is provided, the method comprising:

[0007] A multi-view loop is obtained, which consists of images captured simultaneously by multiple cameras with different horizontal shooting angles of the same scene.

[0008] Feature maps are extracted from the multi-path loop view, and feature vectors are generated based on the feature maps;

[0009] The feature coefficient vector is calculated based on the feature vector and the activation function, and the fused feature map is calculated based on the feature map and the feature coefficient vector. The feature coefficient vector represents the importance of different channels in the feature map in the global scope of multiple views.

[0010] A BEV feature map is created from the depth information and the target feature information map, and target recognition is performed based on the BEV feature map.

[0011] In one possible implementation, calculating the feature coefficient vector based on the feature vector and the activation function includes:

[0012] The feature vectors of the multi-path loop view are fused to obtain an intermediate feature vector;

[0013] The intermediate feature vector is calculated based on the activation function to obtain the feature coefficient vector.

[0014] In one possible implementation, fusing the feature vectors of the multi-loop view to obtain an intermediate feature vector includes:

[0015] The first feature vector is obtained by globally averaging the feature vectors of each loop view.

[0016] The second feature vector is obtained by concatenating all the first feature vectors of the multi-path loop view.

[0017] The second feature vector is fused using a fully linear link to obtain an intermediate feature vector.

[0018] In one possible implementation, the step of calculating the intermediate feature vector according to the activation function to obtain the feature coefficient vector includes:

[0019] The intermediate feature vector is input as a parameter into the activation function, and the output of the activation function is determined as the feature coefficient vector.

[0020] In one possible implementation, calculating the fused feature map based on the feature vector and the feature coefficient vector includes:

[0021] Multiply the feature map by the feature coefficient vector;

[0022] The resulting product is added to the feature map to obtain the fused feature map.

[0023] In one possible implementation, extracting the feature map from the multi-loop view includes:

[0024] The multi-path loop view is subjected to deep semantic extraction using an encoding neural network to obtain a feature map.

[0025] According to a second aspect of this application, a target recognition device based on a BEV (Browser-Electric Vehicle) perspective is provided, the device comprising:

[0026] The acquisition module is used to acquire a multi-view loop, which consists of images obtained by multiple cameras with different horizontal shooting angles simultaneously capturing the same scene.

[0027] The extraction module is used to extract feature maps from the multi-path loop view and generate feature vectors based on the feature maps;

[0028] The fusion module is used to calculate a feature coefficient vector based on the feature vector and the activation function, and to calculate a fused feature map based on the feature map and the feature coefficient vector, wherein the feature coefficient vector represents the importance of different channels in the feature map in the global scope of multiple views;

[0029] The recognition module is used to create a BEV feature map from the BEV perspective based on the depth information and the target feature information map, and to perform target recognition based on the BEV feature map.

[0030] In one possible implementation, the fusion module is further configured to:

[0031] The feature vectors of the multi-path loop view are fused to obtain an intermediate feature vector;

[0032] The intermediate feature vector is calculated based on the activation function to obtain the feature coefficient vector.

[0033] According to a third aspect of this application, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the target recognition method based on the BEV perspective as described above.

[0034] According to a fourth aspect of this application, a computer device is provided, the computer device including a target recognition device based on a BEV perspective.

[0035] The beneficial effects of the technical solution provided in this application include at least the following:

[0036] After extracting the feature maps and feature vectors from the multi-view loop, feature coefficient vectors are first calculated based on the feature vectors and activation functions. Then, a fused feature map is calculated based on the feature maps and feature coefficient vectors. Next, depth information and target feature information maps are generated from the fused feature map. Finally, a BEV feature map is created from the depth information and target feature information map, enabling target recognition based on the BEV feature map. Since the depth information is generated from the fused feature map, its accuracy is improved. Furthermore, the feature coefficient vectors represent the importance of different channels in the feature map within the global scope of multiple views. Therefore, fusing the feature vectors from the multi-view loop can improve target recognition accuracy. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of a multi-view image;

[0039] Figure 2 This is a flowchart of a target recognition method based on the BEV perspective provided in one embodiment of this application;

[0040] Figure 3 This is a flowchart of a target recognition method based on the BEV perspective provided in one embodiment of this application;

[0041] Figure 4 This is a schematic diagram of global average sampling provided in one embodiment of this application;

[0042] Figure 5 This is a schematic diagram of feature fusion provided in one embodiment of this application;

[0043] Figure 6 This is a structural block diagram of a target recognition device based on the BEV perspective provided in one embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0045] like Figure 2 The diagram illustrates a flowchart of a target recognition method based on a BEV (Battery Electric Vehicle) perspective according to an embodiment of this application. This BEV-based target recognition method can be applied to computer devices. The BEV-based target recognition method may include:

[0046] Step 201: Obtain a multi-view loop view, which consists of images captured simultaneously by multiple cameras with different horizontal shooting angles of the same scene.

[0047] For a given scene, multiple cameras with different horizontal shooting angles can be installed around it. The number of cameras depends on their installation locations, shooting angles, and the total required shooting angle for the scene. For example, if a panoramic view of the scene is needed, and assuming each camera has a horizontal shooting angle of 60°, then six cameras are required. The first camera would have a shooting angle of 0-60°, the second 60°-120°, the third 120°-180°, the fourth 180°-240°, the fifth 240°-300°, and the sixth 300°-360° to achieve a panoramic view. If a panoramic view is not required, the number of cameras needed can be calculated based on the shooting angles.

[0048] The image captured by each camera can be called a single loop view, and the images captured by multiple cameras simultaneously form a multi-loop view.

[0049] Specifically, computer equipment can acquire video streams captured by multiple cameras over a period of time, sample each video stream at the same sampling frequency, and obtain a multi-channel loop view.

[0050] Step 202: Extract feature maps from the multi-path loop view and generate feature vectors based on the feature maps.

[0051] Computer equipment can use various extraction methods to extract features from multi-view loops. First, a feature map is obtained, and then a feature vector is generated based on the feature map. The feature vector can be denoted as F[B, N, C, W, H], where B stands for Batch, representing the number of images captured by a single camera in a set of multi-view loops, N represents the number of cameras, C represents the number of channels of the extracted features, and W and H represent the width and height of the extracted feature map, respectively.

[0052] Step 203: Calculate the feature coefficient vector based on the feature vector and activation function, and calculate the fused feature map based on the feature map and feature coefficient vector. The feature coefficient vector represents the importance of different channels in the feature map in the global scope of multiple views.

[0053] The activation function can be the sigmoid function.

[0054] The feature coefficient vector can be a coefficient vector representing the importance of different channels in the feature map. The calculation method is described in detail below and will not be repeated here.

[0055] Step 204: Generate depth information and target feature information map based on the fused feature map.

[0056] Because the calculation method for fused feature maps is relatively simple, it reduces the difficulty of extracting depth information and target feature information maps. Furthermore, since the fused feature maps are obtained based on multi-path loop views, the accuracy of depth information and target feature information maps can be improved.

[0057] Step 205: Create a BEV feature map from the BEV perspective based on the depth information and the target feature information map, and perform target recognition based on the BEV feature map.

[0058] Computer devices can model objects from a BEV perspective and perform target recognition based on depth information and target feature information maps.

[0059] In summary, the target recognition method based on the BEV perspective provided in this application extracts the feature maps and feature vectors of the multi-view loop view. First, it calculates feature coefficient vectors based on the feature vectors and activation functions, then calculates a fused feature map based on the feature maps and feature coefficient vectors. Next, it generates depth information and a target feature information map based on the fused feature map. Finally, it creates a BEV feature map from the BEV perspective based on the depth information and the target feature information map, thereby performing target recognition based on the BEV feature map. Since the depth information is generated from the fused feature map, the accuracy of the depth information is improved. Furthermore, the feature coefficient vectors represent the importance of different channels in the feature map within the global scope of multiple views. Therefore, fusing the feature vectors of the multi-view loop view can improve the target recognition accuracy.

[0060] like Figure 3 The diagram illustrates a flowchart of a target recognition method based on a BEV (Battery Electric Vehicle) perspective according to an embodiment of this application. This BEV-based target recognition method can be applied to computer devices. The BEV-based target recognition method may include:

[0061] Step 301: Obtain a multi-view loop view, which consists of images captured simultaneously by multiple cameras with different horizontal shooting angles of the same scene.

[0062] For a given scene, multiple cameras with different horizontal shooting angles can be installed around it. The number of cameras depends on their installation locations, shooting angles, and the total required shooting angle for the scene. For example, if a panoramic view of the scene is needed, and assuming each camera has a horizontal shooting angle of 60°, then six cameras are required. The first camera would have a shooting angle of 0-60°, the second 60°-120°, the third 120°-180°, the fourth 180°-240°, the fifth 240°-300°, and the sixth 300°-360° to achieve a panoramic view. If a panoramic view is not required, the number of cameras needed can be calculated based on the shooting angles.

[0063] The image captured by each camera can be called a single loop view, and the images captured by multiple cameras simultaneously form a multi-loop view.

[0064] Specifically, computer equipment can acquire video streams captured by multiple cameras over a period of time, sample each video stream at the same sampling frequency, and obtain a multi-channel loop view.

[0065] Step 302: Extract feature maps from the multi-path loop view and generate feature vectors based on the feature maps.

[0066] Computer equipment can use various extraction methods to extract features from multi-view loops. First, a feature map is obtained, and then a feature vector is generated based on the feature map. The feature vector can be denoted as F[B, N, C, W, H], where B stands for Batch, representing the number of images captured by a single camera in a set of multi-view loops, N represents the number of cameras, C represents the number of channels of the extracted features, and W and H represent the width and height of the extracted feature map, respectively.

[0067] Specifically, feature map extraction from a multi-way loop view can include: using an encoding neural network to perform deep semantic extraction on the multi-way loop view to obtain the feature map. The encoding neural network can be MobileNet, ResNet, or VGGNet.

[0068] Step 303: Fuse the feature vectors of the multi-path loop view to obtain the intermediate feature vector; calculate the feature coefficient vector based on the activation function.

[0069] Specifically, fusing the feature vectors of the multi-way loop view to obtain the intermediate feature vector can include the following sub-steps:

[0070] (1) Perform global average sampling on the feature vector of each loop view to obtain the first feature vector.

[0071] To reduce computational load, the computer device needs to perform global sampling of the feature vectors at a two-dimensional level. In this embodiment, the sampling strategy is global average sampling (also known as global average pooling), such as... Figure 4 As shown, the first feature vector obtained is F[B, N, C, 1, 1].

[0072] (2) Concatenate all the first feature vectors of the multi-path loop view to obtain the second feature vector.

[0073] Each camera's corresponding panorama view can yield a first feature vector of length C. To fuse the feature maps of multiple panoramas, the computer device can concatenate N feature vectors of length C to obtain a second feature vector.

[0074] (3) The second feature vector is fused using a fully linear link to obtain an intermediate feature vector.

[0075] After concatenation, a second feature vector of length N×C can be obtained. Computer equipment can then use a fully linear link to fuse this second feature vector, resulting in an intermediate feature vector of length C, such as... Figure 5 As shown.

[0076] Specifically, the intermediate feature vector is calculated based on the activation function to obtain the feature coefficient vector. This can include: inputting the intermediate feature vector as a parameter into the activation function, and determining the output of the activation function as the feature coefficient vector.

[0077] The activation function can be the sigmoid function. The formula for the activation function can be... e is a natural constant. That is, the computer takes the intermediate feature vector F as the x parameter and inputs it into the sigmoid function to obtain the feature coefficient vector F. sigmoid .

[0078] Step 304: Calculate the fused feature map based on the feature map and the feature coefficient vector. The feature coefficient vector represents the importance of different channels in the feature map in the global scope of multiple views.

[0079] Specifically, calculating the fused feature map based on the feature map and the feature coefficient vector can include: multiplying the feature map by the feature coefficient vector; and adding the resulting product to the feature map to obtain the fused feature map.

[0080] The formula for calculating the fused feature map is: F out =F[B, N, C, W, H] + F sigmoid *F[B, N, C, W, H].

[0081] Step 305: Generate depth information and target feature information map based on the fused feature map.

[0082] Because the calculation method for fused feature maps is relatively simple, it reduces the difficulty of extracting depth information and target feature information maps. Furthermore, since the fused feature maps are obtained based on multi-path loop views, the accuracy of depth information and target feature information maps can be improved.

[0083] Step 306: Create a BEV feature map from the BEV perspective based on the depth information and the target feature information map, and perform target recognition based on the BEV feature map.

[0084] Computer devices can model objects from a BEV perspective and perform target recognition based on depth information and target feature information maps.

[0085] In related technologies, when feature fusion is required for N-way panoramic views, connections need to be established between the N cameras, meaning the data volume is N*N. However, in this embodiment, the fused data volume is 1. Therefore, the data volume of this application is only the original data volume. This approach provides a better pre-processing solution for modeling with minimal computational cost. Furthermore, while related technologies require convolution operations for fusion, this embodiment only requires addition and multiplication, significantly reducing computational complexity.

[0086] In summary, the target recognition method based on the BEV perspective provided in this application extracts the feature maps and feature vectors of the multi-view loop view. First, it calculates feature coefficient vectors based on the feature vectors and activation functions, then calculates a fused feature map based on the feature maps and feature coefficient vectors. Next, it generates depth information and a target feature information map based on the fused feature map. Finally, it creates a BEV feature map from the BEV perspective based on the depth information and the target feature information map, thereby performing target recognition based on the BEV feature map. Since the depth information is generated from the fused feature map, the accuracy of the depth information is improved. Furthermore, the feature coefficient vectors represent the importance of different channels in the feature map within the global scope of multiple views. Therefore, fusing the feature vectors of the multi-view loop view can improve the target recognition accuracy.

[0087] like Figure 6 As shown, it illustrates a structural block diagram of a target recognition device based on the BEV perspective provided in an embodiment of this application. This target recognition device based on the BEV perspective can be applied to computer devices.

[0088] The target recognition device based on the BEV perspective may include:

[0089] The acquisition module 610 is used to acquire a multi-view loop, which consists of images captured simultaneously by multiple cameras with different horizontal shooting angles of the same scene.

[0090] Extraction module 620 is used to extract feature maps from multi-path loop view and generate feature vectors based on the feature maps;

[0091] The fusion module 630 is used to calculate the feature coefficient vector based on the feature vector and the activation function, and to calculate the fused feature map based on the feature map and the feature coefficient vector. The feature coefficient vector represents the importance of different channels in the feature map in the global scope of multiple views.

[0092] The generation module 640 is used to generate depth information and target feature information maps based on the fused feature map;

[0093] The recognition module 650 is used to create a BEV feature map from the BEV perspective based on depth information and target feature information map, and to perform target recognition based on the BEV feature map.

[0094] In an optional embodiment, the fusion module 630 is further configured to:

[0095] The feature vectors of the multi-way loop view are fused to obtain the intermediate feature vector;

[0096] The feature coefficient vector is obtained by calculating the intermediate feature vector based on the activation function.

[0097] In summary, the target recognition device based on the BEV perspective provided in this application embodiment,

[0098] In an optional embodiment, the fusion module 630 is further configured to:

[0099] The first feature vector is obtained by globally averaging the feature vectors of each loop view.

[0100] The second feature vector is obtained by concatenating all the first feature vectors of the multi-path loop view.

[0101] The second feature vector is fused using a fully linear link to obtain an intermediate feature vector.

[0102] In an optional embodiment, the fusion module 630 is further configured to:

[0103] The intermediate feature vector is input as a parameter into the activation function, and the output of the activation function is determined as the feature coefficient vector.

[0104] In an optional embodiment, the fusion module 630 is further configured to:

[0105] Multiply the feature map by the feature coefficient vector;

[0106] The resulting product is added to the feature map to obtain the fused feature map.

[0107] In an optional embodiment, the extraction module 620 is configured to:

[0108] A deep semantic extraction of the multi-path loop view is performed using an encoding neural network to obtain a feature map.

[0109] In summary, the target recognition device based on the BEV perspective provided in this application extracts the feature maps and feature vectors of multiple loop views. First, it calculates feature coefficient vectors based on the feature vectors and activation functions, then calculates a fused feature map based on the feature maps and feature coefficient vectors. Next, it generates depth information and a target feature information map based on the fused feature map. Finally, it creates a BEV feature map from the BEV perspective based on the depth information and the target feature information map, thereby performing target recognition based on the BEV feature map. Since the depth information is generated from the fused feature map, the accuracy of the depth information is improved. Furthermore, the feature coefficient vectors represent the importance of different channels in the feature map within the global scope of multiple perspectives. Therefore, fusing the feature vectors of the multiple loop views can improve the target recognition accuracy.

[0110] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the target recognition method based on the BEV perspective as described above.

[0111] One embodiment of this application provides a computer device, which includes any of the above-described target recognition devices based on the BEV perspective.

[0112] It should be noted that the target recognition device based on the BEV perspective provided in the above embodiments is only illustrated by the division of the above functional modules when performing target recognition based on the BEV perspective. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the target recognition device based on the BEV perspective can be divided into different functional modules to complete all or part of the functions described above. In addition, the target recognition device based on the BEV perspective provided in the above embodiments and the target recognition method embodiments based on the BEV perspective belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0113] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0114] The above description is not intended to limit the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A target recognition method based on BEV perspective, characterized in that, The method includes: A multi-view loop is obtained, which consists of images captured simultaneously by multiple cameras with different horizontal shooting angles of the same scene. Feature maps are extracted from the multi-path loop view, and feature vectors are generated based on the feature maps; The feature coefficient vector is calculated based on the feature vector and activation function, and the fused feature map is calculated based on the feature map and the feature coefficient vector. The feature coefficient vector is a coefficient vector representing the importance of different channels in the feature map in a global context of multiple views. Generate depth information and target feature information maps based on the fused feature map; A BEV feature map is created from the depth information and the target feature information map, and target recognition is performed based on the BEV feature map. The step of calculating the feature coefficient vector based on the feature vector and the activation function includes: fusing the feature vectors of the multi-loop view to obtain an intermediate feature vector; and calculating the feature coefficient vector based on the intermediate feature vector according to the activation function. The step of fusing the feature vectors of the multi-view loop to obtain an intermediate feature vector includes: performing global average sampling on the feature vectors of each loop to obtain a first feature vector of length C, where C represents the number of channels of the extracted features; concatenating all the first feature vectors of the multi-view loop to obtain a second feature vector of length N×C, where N represents the number of cameras; and fusing the second feature vector using a fully linear link to obtain an intermediate feature vector of length C. The step of calculating the intermediate feature vector according to the activation function to obtain the feature coefficient vector includes: inputting the intermediate feature vector as a parameter into the activation function, and determining the output of the activation function as the feature coefficient vector.

2. The target recognition method based on BEV perspective according to claim 1, characterized in that, The step of calculating the fused feature map based on the feature vector and the feature coefficient vector includes: Multiply the feature map by the feature coefficient vector; The resulting product is added to the feature map to obtain the fused feature map.

3. The target recognition method based on BEV perspective according to claim 1 or 2, characterized in that, The step of extracting feature maps from the multi-path loop view includes: The multi-path loop view is subjected to deep semantic extraction using an encoding neural network to obtain a feature map.

4. A target recognition device based on BEV perspective, characterized in that, The device includes: The acquisition module is used to acquire a multi-view loop, which consists of images obtained by multiple cameras with different horizontal shooting angles simultaneously capturing the same scene. The extraction module is used to extract feature maps from the multi-path loop view and generate feature vectors based on the feature maps; The fusion module is used to calculate a feature coefficient vector based on the feature vector and the activation function, and to calculate a fused feature map based on the feature map and the feature coefficient vector. The feature coefficient vector is a coefficient vector representing the importance of different channels in the feature map in a global context of multiple views. The generation module is used to generate depth information and target feature information maps based on the fused feature map; The recognition module is used to create a BEV feature map from the BEV perspective based on the depth information and the target feature information map, and to perform target recognition based on the BEV feature map; The fusion module is further configured to: fuse the feature vectors of the multi-loop view to obtain an intermediate feature vector; and calculate the intermediate feature vector according to the activation function to obtain a feature coefficient vector; The fusion module is further configured to: perform global average sampling on the feature vector of each loop view to obtain a first feature vector of length C, where C represents the number of channels of the extracted features; concatenate all the first feature vectors of the multi-loop views to obtain a second feature vector of length N×C, where N represents the number of cameras; and fuse the second feature vector using a fully linear link to obtain an intermediate feature vector of length C. The fusion module is further configured to: input the intermediate feature vector as a parameter into the activation function, and determine the output of the activation function as a feature coefficient vector.

5. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction, which is loaded and executed by a processor to implement the target recognition method based on the BEV perspective as described in any one of claims 1 to 3.

6. A computer device, characterized in that, The computer device includes: the target recognition device based on the BEV perspective as described in claim 4.

Images