A BEV space target detection method and device based on multi-perspective images

Abstract

The present invention discloses a BEV spatial target detection method and device for multi-perspective images. The method inputs the multi-perspective images into a preset BEV target detector, generates a corresponding perspective image feature map through an image feature extractor, inputs the perspective image feature map into a built-in depth feature extractor and foreground and background mask extractor, generates a depth feature map through the depth feature extractor, and generates a foreground and background mask feature map through the foreground and background mask extractor. The features of foreground pixels are then projected based on the perspective image feature map, the foreground and background mask feature map, and the depth feature map to generate a BEV feature map. Based on the BEV feature map, the position and category information of the target to be detected are detected. The implementation of the embodiments of the present invention can reduce the amount of computation and improve the efficiency and accuracy of target detection.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and autonomous driving, and in particular to a BEV space target detection method and device based on multi-perspective images. Background Art

[0002] In the autonomous driving field, the vehicle needs to accurately perceive obstacles, cars, pedestrians, and other targets in its surroundings. Although image-based 2D object detectors are mature, the image processing process suffers from perspective effects, making it difficult to determine the position and distance of the target relative to the vehicle based solely on the image detection results. Without the need for expensive equipment such as lidar, the BEV object detector can directly detect targets in the BEV space by simply inputting perspective image information collected by multiple cameras installed around the vehicle. BEV object detection already represents multiple perspective images in a unified BEV space, inherently solving the problem of target association.

[0003] However, the existing BEV target detector projects all the features of the image into the BEV space where the imaging line of sight is located, which results in high computational complexity and low target detection efficiency. Summary of the Invention

[0004] An embodiment of the present invention provides a BEV spatial target detection method and device for multi-perspective images, which can separate the foreground and background of the image during target detection, and only deliver foreground features when delivering features, thereby reducing the amount of calculation and improving target detection efficiency. At the same time, the BEV feature map delivered to the BEV eliminates background information, which is more conducive to the training and learning of foreground targets, thereby improving the accuracy of target detection.

[0005] An embodiment of the present invention provides a BEV space target detection method using multi-eye perspective images, comprising: acquiring a multi-eye perspective image of a target to be detected;

[0006] Inputting the multi-perspective image into a preset BEV object detector so that the BEV object detector recognizes the position and category information of the target to be detected;

[0007] The BEV target detector identifies the location and category information of the target to be detected, including:

[0008] Performing feature extraction on the multi-view perspective image by an image feature extractor in a BEV object detector to generate a perspective image feature map;

[0009] Inputting the perspective image feature map into a depth feature extractor in a BEV object detector, so that the depth feature extractor extracts the depth features of each pixel in the perspective image feature map to generate a depth feature map;

[0010] Inputting the perspective image feature map into a foreground-background mask extractor in a BEV object detector, so that the foreground-background mask extractor extracts mask features of foreground pixels and background pixels in the perspective image feature map to generate a foreground-background mask feature map;

[0011] Generating a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels according to the foreground-background mask feature map;

[0012] Performing topK processing on each depth category corresponding to each pixel in the depth feature map to obtain a topK probability feature map and a topK index feature map; calculating a probability weighted feature map based on the topK probability feature map and the perspective image feature map; calculating a topK depth feature map based on the topK index feature map; generating a BEV mapping coordinate of each pixel in the BEV coordinate system based on the depth value of each pixel in the topK depth feature map and the pixel coordinates of each pixel;

[0013] According to the positions of the foreground pixels in the binary mask feature map, the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map are extracted, and then foreground features are projected to generate a BEV feature map;

[0014] According to the BEV feature map, the position and category information of the target to be detected are detected.

[0015] Furthermore, the depth feature extractor extracts the depth feature of each pixel in the perspective image feature map to generate a depth feature map, including:

[0016] Determining the probability value of each depth category corresponding to each pixel point of the perspective image feature map, and generating the depth feature map according to the probability values ​​of each depth category corresponding to all pixels points;

[0017] The generation of the depth categories corresponding to each pixel includes: dividing the preset depth range corresponding to the pixel according to preset intervals to generate a plurality of depth categories corresponding to the pixel.

[0018] Furthermore, generating a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels based on the foreground-background mask feature map includes:

[0019] Extracting the mask value of each pixel in the foreground and background mask feature map;

[0020] Compare the mask value of each pixel with the preset threshold;

[0021] The mask values ​​of pixels whose mask values ​​are greater than a preset threshold are updated to the preset foreground mask value, and the mask values ​​of pixels whose mask values ​​are less than the preset threshold are updated to the preset background mask value, thereby generating a background mask feature map before binarization.

[0022] Furthermore, performing topK processing on each depth category corresponding to each pixel point in the depth feature map to obtain a topK probability feature map and a topK index feature map includes:

[0023] For each pixel, sort the probability values ​​of each depth category corresponding to the pixel in descending order, and then obtain the top k probability values;

[0024] Generate a topK probability feature map based on the topK probability values ​​of all pixels and the depth categories corresponding to the topK probability values;

[0025] For each pixel point, the index value of the top K probability values ​​of the pixel point is determined according to the index position of the top K probability values ​​of the pixel point before sorting; and the topK index feature map is generated according to the index values ​​of the top K probability values ​​corresponding to all pixels.

[0026] Furthermore, the calculating of the probability weighted feature map based on the topK probability feature map and the perspective image feature map includes:

[0027] The K probability values ​​of each pixel point on the topK probability feature map are multiplied by the feature values ​​of each channel of the corresponding pixel point in the perspective image feature map to generate a probability weighted feature map.

[0028] Furthermore, the step of calculating the topK depth feature map based on the topK index feature map includes:

[0029] For each pixel in the topK index feature map, determine the depth category corresponding to each index value based on the index value of the first K probability values ​​of the pixel; extract the center point of the depth range corresponding to each depth category to generate the k depth values ​​corresponding to the pixel;

[0030] Generate topK depth feature maps based on all depth values ​​of all pixels.

[0031] Furthermore, generating the BEV mapping coordinates of each pixel in the BEV coordinate system according to the depth value of each pixel in the topK depth feature map and the pixel coordinates of each pixel includes:

[0032] Combine the depth values ​​of all pixels in the topK depth feature map with the pixel coordinates of the corresponding pixels to generate three-dimensional space coordinates;

[0033] Performing depth inverse normalization processing on the three-dimensional space coordinates to generate inverse normalized coordinates;

[0034] Mapping the inverse normalized three-dimensional coordinates to a camera coordinate system of a corresponding camera to generate the camera three-dimensional coordinates in the camera coordinate system;

[0035] The camera 3D coordinates are mapped to the radar coordinate system to generate the BEV mapping coordinates of each pixel point in the BEV coordinate system.

[0036] Furthermore, according to the position of the foreground pixel in the binary mask feature map, the BEV mapping coordinates of the foreground pixel in all BEV mapping coordinates and the probability weighted features of the foreground pixel in the probability weighted feature map are extracted, and then the foreground features are projected to generate the BEV feature map, including:

[0037] Extracting the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map according to the position of each foreground mask value in the binary mask feature map, thereby obtaining the BEV mapping coordinates of the foreground pixels and the probability weighted feature map of the foreground pixels, respectively;

[0038] Rasterizing the preset BEV space to obtain a number of BEV grids, filtering out the BEV mapping coordinates of foreground pixels that are not within the BEV grids, and updating the BEV mapping coordinates of the foreground pixels and the probability weighted feature map of the foreground pixels based on the filtered BEV mapping coordinates;

[0039] Generate the coordinates of the valid BEV mapping point and the features of the valid BEV mapping point according to the updated BEV mapping coordinates of the foreground pixel point and the updated probability weighted feature map of the foreground pixel point;

[0040] According to the coordinates and features of the valid BEV mapping points, foreground features are projected to generate a BEV feature map.

[0041] Furthermore, detecting the position and category information of the target to be detected based on the BEV feature map includes:

[0042] Inputting the BEV feature map into a BEV encoder in a BEV target detector so that the BEV encoder encodes the BEV feature map to generate a BEV encoded feature map;

[0043] Inputting the BEV encoding feature map into a BEV decoder in a BEV target detector so that the BEV decoder decodes the BEV encoding feature map to generate a BEV decoding feature map;

[0044] According to the BEV decoding feature map, the position and category information of the target to be detected are generated.

[0045] Based on the above method embodiment, the present invention provides a corresponding device embodiment;

[0046] An embodiment of the present invention provides a BEV space target detection device for multi-perspective images, comprising: a multi-perspective image acquisition module and a target recognition module; the target recognition module comprises: a perspective image feature map generation unit, a depth feature map generation unit, a foreground and background mask feature map generation unit, a binarized foreground and background mask feature map generation unit, a BEV mapping coordinate generation unit, a BEV feature map generation unit, and a detection unit;

[0047] The multi-eye perspective image acquisition module is used to acquire a multi-eye perspective image of the target to be detected;

[0048] The target recognition module is used to input the multi-perspective image into a preset BEV target detector so that the BEV target detector recognizes the position and category information of the target to be detected;

[0049] The perspective image feature map generating unit is configured to extract features from the multi-view perspective image using an image feature extractor within a BEV object detector to generate a perspective image feature map;

[0050] The depth feature map generating unit is configured to input the perspective image feature map into a depth feature extractor in a BEV object detector, so that the depth feature extractor extracts the depth features of each pixel in the perspective image feature map to generate a depth feature map;

[0051] The foreground-background mask feature map generating unit is used to input the perspective image feature map into the foreground-background mask extractor in the BEV object detector, so that the foreground-background mask extractor extracts the mask features of the foreground pixels and the background pixels in the perspective image feature map to generate the foreground-background mask feature map;

[0052] The binary foreground-background mask feature map generating unit is used to generate a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels based on the foreground-background mask feature map;

[0053] The BEV mapping coordinate generation unit is used to perform topK processing on each depth category corresponding to each pixel point in the depth feature map to obtain a topK probability feature map and a topK index feature map; calculate a probability weighted feature map based on the topK probability feature map and the perspective image feature map; calculate a topK depth feature map based on the topK index feature map; generate a BEV mapping coordinate of each pixel point in the BEV coordinate system based on the depth value of each pixel point in the topK depth feature map and the pixel coordinate of each pixel point;

[0054] The BEV feature map generating unit is used to extract the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map according to the positions of the foreground pixels in the binary mask feature map, and then perform foreground feature projection to generate a BEV feature map;

[0055] The detection unit is used to detect the position and category information of the target to be detected based on the BEV feature map.

[0056] The following beneficial effects are achieved by implementing the present invention:

[0057] An embodiment of the present invention provides a BEV spatial target detection method and device for multi-view perspective images. The method inputs the multi-view perspective image into a preset BEV target detector. The BEV target detector generates a corresponding perspective image feature map through a built-in image feature extractor, and then inputs the perspective image feature map into a built-in depth feature extractor and foreground and background mask extractor. The depth feature extractor generates a depth feature map, and the foreground and background mask extractor generates a foreground and background mask feature map. Then, based on the foreground and background mask feature map, a binary foreground and background mask feature map for distinguishing foreground pixels from background pixels is generated. A corresponding probability weighted feature map and BEV mapping coordinates of each pixel are generated based on the depth feature map. Then, based on the position of the foreground pixel in the binary foreground and background mask feature map, the BEV mapping coordinates of the foreground pixel in all BEV mapping coordinates and the probability weighted features of the foreground pixel in the probability weighted feature map are extracted, and then foreground features are projected to generate a BEV feature map. Finally, based on the BEV feature map, the position and category information of the target to be detected are detected. Compared with the existing technology, during the detection process, the present invention only projects the foreground features to generate a BEV feature map, and does not need to project the background features of the image, thereby reducing the amount of calculation and improving the efficiency of target detection. At the same time, the BEV feature map projected into the BEV eliminates background information, which is more conducive to the training and learning of foreground targets, thereby improving the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 11 is a flow chart of a BEV space target detection method using multi-perspective images provided by an embodiment of the present invention;

[0059] Figure 2 1 is a flow chart of a BEV target detector according to an embodiment of the present invention identifying the position and category information of a target to be detected;

[0060] Figure 3 The figure is a schematic structural diagram of a BEV space target detection device for multi-perspective images provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0062] like Figure 1 As shown, an embodiment of the present invention provides a BEV space target detection method for multi-perspective images, which includes at least the following steps:

[0063] S1: Obtain a multi-perspective image of the target to be detected.

[0064] In one embodiment of the present invention, pictures taken by cameras installed around the vehicle body may be collected to obtain a multi-perspective image of the target to be detected.

[0065] Schematically, the multi-perspective image has a total of 6 images, which are taken by 6 cameras that perceive the front, left front, right front, rear left, rear, and rear right of the vehicle.

[0066] S2: Input the multi-perspective image into a preset BEV target detector so that the BEV target detector can identify the position and category information of the target to be detected.

[0067] In one embodiment of the present invention, the multi-perspective image can be input into a trained BEV object detector. The input feature map corresponding to the multi-perspective image is a high-dimensional matrix of 6x3xHxW, where 6 represents the six images, 3 represents the RGB values ​​of the three image channels, and H and W represent the image height and width. After receiving the input multi-perspective image, the BEV object detector identifies the location and category of the target object.

[0068] like Figure 2 As shown, in one embodiment of the present invention, the BEV object detector identifies the position and category information of the target to be detected, which may include the following steps:

[0069] S21: Perform feature extraction on the multi-view perspective image through the image feature extractor in the BEV target detector to generate a perspective image feature map.

[0070] In one embodiment of the present invention, the BEV object detector is equipped with an image feature extractor. The input multi-view image is passed through the image feature extractor to extract a perspective image feature map 6xCx(H / / 8)x(W / / 8). The feature extraction process is performed by running the CNN forward operation. / / 8 represents 8x downsampling by the CNN, meaning the resolution of the feature map is reduced by 8x. C is the number of feature map channels, i.e., the feature vector dimension.

[0071] S22: Input the perspective image feature map into the depth feature extractor in the BEV object detector, so that the depth feature extractor extracts the depth features of each pixel in the perspective image feature map to generate a depth feature map.

[0072] In one embodiment of the present invention, the BEV object detector is further provided with a depth feature extractor, which extracts the depth features of each pixel point in the perspective image feature map to generate a depth feature map;

[0073] In a preferred embodiment, the depth feature extractor extracts the depth features of each pixel in the perspective image feature map to generate a depth feature map, including: determining the probability value of each depth category corresponding to each pixel in the perspective image feature map, and generating the depth feature map based on the probability values ​​of each depth category corresponding to all pixels; wherein, the generation of each depth category corresponding to each pixel includes: dividing the preset depth range corresponding to the pixel according to a preset interval to generate a plurality of depth categories corresponding to the pixel.

[0074] Specifically, the depth feature extractor is composed of a common convolutional neural network, in which the depth estimation is performed using a classification method, that is, a preset depth range is pre-assumed for each pixel, such as 2-52 meters, and then the preset depth range is divided according to a preset interval, such as 0.5, to generate (52-2) / 0.5=100 depth categories. Therefore, D=100 in the depth feature map, and the D values ​​corresponding to each pixel of each camera represent the probability values ​​of the 100 depth categories corresponding to the pixel position. According to the probability values ​​of the 100 depth categories corresponding to all pixel positions, the above-mentioned depth feature map 6xDx(H / / 8)x(W / / 8) is generated.

[0075] S23: Input the perspective image feature map into the foreground and background mask extractor in the BEV target detector, so that the foreground and background mask extractor extracts the mask features of the foreground pixels and background pixels in the perspective image feature map to generate the foreground and background mask feature map.

[0076] In one embodiment of the present invention, a foreground-background mask extractor is further provided in the BEV target detector, which extracts the mask features of the foreground pixels and background pixels in the perspective image feature map to generate a foreground-background mask feature map 6x1x(H / / 8)x(W / / 8); the preferred foreground-background mask extractor is composed of a common convolutional neural network.

[0077] S24: Based on the foreground-background mask feature map, a binary foreground-background mask feature map is generated for distinguishing foreground pixels from background pixels.

[0078] After generating the foreground and background mask feature map, in order to distinguish the foreground and background, the foreground and background mask feature map is binarized to generate a binary foreground and background mask feature map.

[0079] In a preferred embodiment, generating a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels based on the foreground-background mask feature map includes:

[0080] Extract the mask value of each pixel in the foreground and background mask feature map; compare the mask value of each pixel with a preset threshold; update the mask value of the pixel whose mask value is greater than the preset threshold to the preset foreground mask value, and update the mask value of the pixel whose mask value is less than the preset threshold to the preset background mask value, thereby generating a binary foreground and background mask feature map.

[0081] Specifically, in the present invention, the above-mentioned preset threshold is set to 0.5, the above-mentioned preset foreground mask value is set to 1, and the above-mentioned background mask value is set to 0. Then, during the binarization processing, the mask value of the pixel point with a mask value less than 0.5 in the background mask feature map is updated to 0, and the mask value of the pixel point greater than or equal to 0.5 is updated to 1; thereby generating a background mask feature map before binarization. In the background mask feature map before binarization, the pixel point with a mask value of 1 is the foreground pixel point, and the pixel point with a mask value of 0 is the background pixel point.

[0082] S25: Perform topK processing on each depth category corresponding to each pixel point in the depth feature map to obtain a topK probability feature map and a topK index feature map; calculate a probability weighted feature map based on the topK probability feature map and the perspective image feature map; calculate a topK depth feature map based on the topK index feature map; generate the BEV mapping coordinates of each pixel point in the BEV coordinate system based on the depth value of each pixel point in the topK depth feature map and the pixel coordinates of each pixel point.

[0083] In a preferred embodiment, performing topK processing on each depth category corresponding to each pixel point in the depth feature map to obtain a topK probability feature map and a topK index feature map includes:

[0084] For each pixel, sort the probability values ​​of each depth category corresponding to the pixel in descending order, and then obtain the top K probability values; generate a topK probability feature map based on the top K probability values ​​of all pixels and the depth categories corresponding to the top K probability values; for each pixel, determine the index value of the top K probability value of the pixel based on the index position of the top K probability value of the pixel before sorting; generate a topK index feature map based on the index value of the top K probability value corresponding to all pixels.

[0085] In the above example, each pixel corresponds to 100 depth categories and therefore has 100 corresponding probability values. The 100 probability values ​​corresponding to the pixels are sorted in descending order, and the top k probability values ​​are taken. Based on the K probability values ​​taken and the depth categories corresponding to each of the K probability values, a topK probability feature map of 6xKx(H / / 8)x(W / / 8) is generated. K can be set to 5; by obtaining the top k probability values ​​for calculation, the computational complexity can be reduced. This is because subsequent probability values ​​will participate in a series of operations. After obtaining the topK, only 5 values ​​need to be involved, instead of 100 values, which greatly reduces the computational complexity.

[0086] At the same time, according to the index positions of the k probability values ​​before sorting, the corresponding index values ​​are determined. According to the determined index values ​​and the depth categories corresponding to the K probability values, a topK index feature map of 6xKx(H / / 8)x(W / / 8) is generated.

[0087] The topK probability feature map describes the top K probability values ​​obtained from the D probability values ​​after the topK process. The topK index feature map describes the index position of the top K probability values ​​obtained after the topK process in the original D probability values, that is, the index position of the D values.

[0088] In a preferred embodiment, the calculating of the probability weighted feature map based on the topK probability feature map and the perspective image feature map includes:

[0089] The K probability values ​​of each pixel point on the topK probability feature map are multiplied by the feature values ​​of each channel of the corresponding pixel point in the perspective image feature map to generate a probability weighted feature map.

[0090] Specifically, multiply the K probability values ​​of each pixel of the topK probability feature map (6xKx(H / / 8)x(W / / 8)) by the feature values ​​of the C feature map channels of each pixel of the perspective image feature map (6xCx(H / / 8)x(W / / 8)) to obtain a probability weighted feature map of [6xCxKx(H / / 8)x(W / / 8)].

[0091] In a preferred embodiment, the calculating of the topK depth feature map according to the topK index feature map includes:

[0092] For each pixel in the topK index feature map, the depth category corresponding to each index value is determined based on the index value of the first K probability values ​​of the pixel; the center point of the depth range corresponding to each depth category is extracted to generate the k depth values ​​corresponding to the pixel; and the topK depth feature map is generated based on all the depth values ​​of all the pixels.

[0093] Referring to the above example, the K index values ​​corresponding to each pixel of each camera in the topK index feature map are used to find the corresponding depth category in the pre-defined 100 depth categories, and then the center point of the depth range represented by the depth category is taken as the depth value: for example, 2-2.5 represents the depth range of a depth category, and its center point is 2.25. At this time, 2.25 is used as the depth value. Finally, based on all the depth values ​​of all pixels, a 6xKx(H / / 8)x(W / / 8)x1 topK depth feature map can be generated.

[0094] In a preferred embodiment, generating the BEV mapping coordinates of each pixel in the BEV coordinate system according to the depth value of each pixel in the topK depth feature map and the pixel coordinates of each pixel includes:

[0095] The depth values ​​of all pixels in the topK depth feature map are combined with the pixel coordinates of the corresponding pixels to generate three-dimensional space coordinates; the three-dimensional space coordinates are depth-denormalized to generate inverse normalized coordinates; the inverse normalized three-dimensional coordinates are mapped to the camera coordinate system of the corresponding camera to generate the camera three-dimensional coordinates in the camera coordinate system; the camera three-dimensional coordinates are mapped to the radar coordinate system to generate the BEV mapping coordinates of each pixel in the BEV coordinate system.

[0096] Specifically, each pixel of each camera in the topK depth feature map corresponds to K depth values. Each depth value is combined with its pixel coordinate (u, v) to obtain a three-dimensional space coordinate of [6xKx(H / / 8)x(W / / 8)x3]. The first and second values ​​of the last dimension are the pixel coordinates (u, v), and the third value is the depth value z. Then, the depth is inverse normalized, that is, the first and second values ​​are multiplied by the depth value to obtain (uz, vz, z), thereby obtaining a new inverse normalized coordinate of [6xKx(H / / 8)x(W / / 8)x3].

[0097] Then, the inverse normalized coordinates are mapped to the camera coordinate system using matrix multiplication using the pre-calibrated intrinsic parameter matrix of each camera to obtain the three-dimensional coordinates in the camera coordinate system of [6xKx(H / / 8)x(W / / 8)x3];

[0098] Finally, the camera's 3D coordinates are mapped to the radar coordinate system, i.e., the BEV coordinate system, or the vehicle body coordinate system (the three coordinate systems can be integrated), using matrix multiplication using the pre-calibrated extrinsic matrix. This yields a BEV mapping coordinate of [6xKx(H / / 8)x(W / / 8)x3], which is the BEV mapping coordinate of the pixel point in the BEV coordinate system.

[0099] S26: According to the positions of the foreground pixels in the binary mask feature map, the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map are extracted, and then the foreground features are projected to generate a BEV feature map.

[0100] In a preferred embodiment, the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map are extracted according to the positions of the foreground pixels in the binary mask feature map, and then the foreground features are projected to generate the BEV feature map, including:

[0101] Extracting the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map according to the position of each foreground mask value in the binary mask feature map, thereby obtaining the BEV mapping coordinates of the foreground pixels and the probability weighted feature map of the foreground pixels, respectively;

[0102] Rasterizing the preset BEV space to obtain a number of BEV grids, filtering out the BEV mapping coordinates of foreground pixels that are not within the BEV grids, and updating the BEV mapping coordinates of the foreground pixels and the probability weighted feature map of the foreground pixels based on the filtered BEV mapping coordinates;

[0103] Generate the coordinates of the valid BEV mapping point and the features of the valid BEV mapping point according to the updated BEV mapping coordinates of the foreground pixel point and the updated probability weighted feature map of the foreground pixel point;

[0104] According to the coordinates and features of the valid BEV mapping points, foreground features are projected to generate a BEV feature map.

[0105] Specifically, the probability weighted feature map and the BEV mapping coordinates are filtered using the foreground pixel positions in the binary mask feature map, i.e., the 1 value, to obtain the foreground probability weighted feature map (i.e., foreground probability weighted feature map [6xCxKxM]) and the BEV mapping coordinates of the foreground pixels (i.e., foreground BEV mapping coordinates [6xKxMx3]). M is the number of foreground pixels in the (H / / 8)x(W / / 8) pixel coordinates, and each M has a one-to-one correspondence.

[0106] The BEV space is then rasterized, retaining the mapped points within the BEV range. Multiple perspective images contain the ego vehicle's surround view information. Using the ego vehicle's body coordinate system, a three-dimensional cube is constructed with a radius of 50 meters in front, 46 meters behind, 32 meters to the left and right, 5 meters above, and 3 meters below to create the pre-defined BEV space. The xy plane is parallel to the ground, with the z axis centered above and below. The three-dimensional space is rasterized in increments of 0.5 meters in the xy direction and 8 meters in the z direction, resulting in a 192x128x1 BEV grid of [96 / 0.5, 64 / 0.5, 8 / 8], with each grid cell corresponding to the spatial extent of the three-dimensional space. Further filter out points that fall outside the BEV grid and retain valid points. That is, based on the 3D coordinates of the 6xKxM points (BEV space uniformly represents the six images) in the foreground BEV mapping coordinates [6xKxMx3], determine whether they are within the BEV grid. If they are, retain them, and if they are not, discard them. Here, both retention and discarding process the foreground probability weighted feature map and the foreground BEV mapping coordinates simultaneously. Obtain an Nx3 valid BEV mapping point and a CxN valid BEV mapping point feature (the foreground probability weighted feature map is considered as a feature of 6xKxM points, with each feature vector dimension of C), where N is the number of valid points after filtering the 6xKxM points.

[0107] Traverse N and project the corresponding features onto the BEV grid based on their 3D coordinates. If multiple mapping points correspond to the same grid, the C-dimensional feature vectors are summed. For grids without feature projections, their C-dimensional features are initialized to 0. Finally, a BEV feature map of size Cx192x128 is obtained, where 192 = HBEV and 128 = WBEV. The generated BEV feature map only contains the features of foreground pixels, significantly reducing the amount of computation. At the same time, the BEV feature map projected onto the BEV removes background information, making it more conducive to training and learning for foreground objects, thereby improving the accuracy of object detection.

[0108] S27: Detect the position and category information of the target to be detected based on the BEV feature map.

[0109] In a preferred embodiment, detecting the position and category information of the target to be detected based on the BEV feature map includes:

[0110] The BEV feature map is input into a BEV encoder in a BEV target detector so that the BEV encoder encodes the BEV feature map to generate a BEV encoding feature map; the BEV encoding feature map is input into a BEV decoder in a BEV target detector so that the BEV decoder decodes the BEV encoding feature map to generate a BEV decoding feature map; and the position and category information of the target to be detected is generated based on the BEV decoding feature map.

[0111] In one embodiment of the present invention, the BEV target detector is further provided with a BEV encoder and a BEV decoder; the BEV encoder and the BEV decoder are both CNN networks; the BEV feature map is encoded by the BEV encoder to generate a BEV encoded feature map, the BEV encoded feature map is decoded by the BEV decoder, and finally the BEV decoded feature map is post-processed to obtain the position and category information of the detected target.

[0112] To better illustrate the present invention, the training process of the BEV target detector is described below from the perspective of model training. The training process of the BEV target detector includes the following steps:

[0113] 1. Prepare training data and the corresponding BEV annotations, and perform annotation processing: The BEV annotations of the training data include the target position information and category information of the BEV space. In actual applications, the target position information and category information of the BEV space can be manually annotated by the radar point cloud installed on the roof, or the point cloud detection model can predict the target position information and category information from the roof radar point cloud data.

[0114] The annotation processing includes processing the training data annotation to obtain the foreground and background mask annotation, and processing the point cloud data to obtain the sparse depth annotation corresponding to each image.

[0115] The radar coordinate system is the unified coordinate system of the BEV space. The mapping relationship between the radar coordinate system and the pixel coordinate system of each camera image can be obtained by pre-calibration.

[0116] When annotating training data, specifically, the target position coordinates in the radar coordinate system can be projected to the camera coordinate system using the extrinsic parameter matrix. Then, the intrinsic parameter matrix is ​​used to project the camera coordinate system to the corresponding pixel coordinate system of the image. This determines the region corresponding to each target in the BEV space in each image. By taking the circumscribed rectangle of this region, we can obtain the ROI region of interest (ROI) in each image. Initialize a 6x1xHxW all-zero mask annotation matrix. Set the mask matrix to 1 for all target ROIs in the six images, indicating the presence of a foreground target at that pixel.

[0117] When processing point cloud data, all point cloud data coordinates (which are already in the radar coordinate system) are projected into the camera coordinate system using an extrinsic parameter matrix, and then projected into the pixel coordinate system of each image using an intrinsic parameter matrix. This gives us the depth of each point in the camera coordinate system and the pixel position in the pixel coordinate system. Combining the two gives us the depth of each pixel in each image. If a pixel has no corresponding radar point, the depth of that pixel is 0, which results in a 6x1xHxW depth annotation matrix. The depth is the distance between each point in the real world and the camera's optical center; the camera imaging process itself loses depth information.

[0118] 2. Input multi-camera perspective image samples installed around the vehicle body (e.g., 6 images, sensing the front, left front, right front, rear left, rear, and rear right, respectively, with overlapping sensing areas of adjacent cameras).

[0119] 3. The input multi-view perspective image samples are passed through the image feature extractor to extract the perspective image feature map corresponding to the training sample.

[0120] 4. The perspective image feature map corresponding to the training sample is sent to the depth feature extractor and the foreground and background mask extractor respectively to obtain the depth feature map and foreground and background mask feature map corresponding to the training sample.

[0121] 5. Use the mask annotation matrix to supervise the foreground and background mask feature maps corresponding to the training samples and calculate the mask loss.

[0122] 6. Use the depth annotation matrix to supervise the depth feature map corresponding to the training sample and calculate the depth loss.

[0123] 7. Use the depth feature map, perspective image feature map, and background mask feature map corresponding to the training sample to convert the features into the BEV space to obtain the BEV feature map corresponding to the training sample.

[0124] 8. Use the BEV encoder to obtain the BEV encoded feature map corresponding to the training sample.

[0125] 9. Use the BEV decoder to obtain the BEV decoding feature map corresponding to the training sample.

[0126] 10. Use the BEV annotations corresponding to the training data to supervise the BEV decoding feature map and calculate the target detection loss.

[0127] 11. Use mask loss + depth loss + target detection loss to update and train all CNN model parameters using the backpropagation algorithm.

[0128] 12. Repeat the above process until the network converges, and a trained BEV target detector is obtained.

[0129] Preferably, the BEV space target detection method for multi-perspective images provided in the above-mentioned embodiments of the present invention performs target detection based on the BEV target detector trained in the above-mentioned steps 1 to 12. It can be understood that the specific processing flow of steps 2 to 9 in the above-mentioned model training process is consistent with the processing flow of the corresponding steps in the model application process during target detection. In order to avoid redundancy, the specific processing flow of steps 2 to 9 will not be elaborated.

[0130] Based on the above method embodiment, the present invention provides a corresponding device embodiment:

[0131] like Figure 3 As shown, an embodiment of the present invention provides a BEV space target detection device for multi-perspective images, comprising: a multi-perspective image acquisition module and a target recognition module; the target recognition module comprises: a perspective image feature map generation unit, a depth feature map generation unit, a foreground and background mask feature map generation unit, a binarized foreground and background mask feature map generation unit, a BEV mapping coordinate generation unit, a BEV feature map generation unit, and a detection unit;

[0132] The multi-eye perspective image acquisition module is used to acquire a multi-eye perspective image of the target to be detected;

[0133] The target recognition module is used to input the multi-perspective image into a preset BEV target detector so that the BEV target detector recognizes the position and category information of the target to be detected;

[0134] The perspective image feature map generating unit is configured to extract features from the multi-view perspective image using an image feature extractor within a BEV object detector to generate a perspective image feature map;

[0135] The depth feature map generating unit is configured to input the perspective image feature map into a depth feature extractor in a BEV object detector, so that the depth feature extractor extracts the depth features of each pixel in the perspective image feature map to generate a depth feature map;

[0136] The foreground-background mask feature map generating unit is used to input the perspective image feature map into the foreground-background mask extractor in the BEV object detector, so that the foreground-background mask extractor extracts the mask features of the foreground pixels and the background pixels in the perspective image feature map to generate the foreground-background mask feature map;

[0137] The binary foreground-background mask feature map generating unit is used to generate a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels based on the foreground-background mask feature map;

[0138] The BEV mapping coordinate generation unit is used to perform topK processing on each depth category corresponding to each pixel point in the depth feature map to obtain a topK probability feature map and a topK index feature map; calculate a probability weighted feature map based on the topK probability feature map and the perspective image feature map; calculate a topK depth feature map based on the topK index feature map; generate a BEV mapping coordinate of each pixel point in the BEV coordinate system based on the depth value of each pixel point in the topK depth feature map and the pixel coordinate of each pixel point;

[0139] The BEV feature map generating unit is used to extract the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map according to the positions of the foreground pixels in the binary mask feature map, and then perform foreground feature projection to generate a BEV feature map;

[0140] The detection unit is used to detect the position and category information of the target to be detected based on the BEV feature map.

[0141] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0142] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A BEV space target detection method based on multi-perspective images, characterized in that: include: Acquire a multi-perspective image of the target to be detected; Inputting the multi-perspective image into a preset BEV object detector so that the BEV object detector recognizes the position and category information of the target to be detected; The BEV target detector identifies the location and category information of the target to be detected, including: Performing feature extraction on the multi-view perspective image by an image feature extractor in a BEV object detector to generate a perspective image feature map; Inputting the perspective image feature map into a depth feature extractor in a BEV object detector, so that the depth feature extractor extracts the depth features of each pixel in the perspective image feature map to generate a depth feature map; Inputting the perspective image feature map into a foreground-background mask extractor in a BEV object detector, so that the foreground-background mask extractor extracts mask features of foreground pixels and background pixels in the perspective image feature map to generate a foreground-background mask feature map; Generating a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels according to the foreground-background mask feature map; Performing topK processing on each depth category corresponding to each pixel in the depth feature map to obtain a topK probability feature map and a topK index feature map; calculating a probability weighted feature map based on the topK probability feature map and the perspective image feature map; calculating a topK depth feature map based on the topK index feature map; generating a BEV mapping coordinate of each pixel in the BEV coordinate system based on the depth value of each pixel in the topK depth feature map and the pixel coordinates of each pixel; According to the positions of the foreground pixels in the binary mask feature map, the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map are extracted, and then foreground features are projected to generate a BEV feature map; According to the BEV feature map, the position and category information of the target to be detected are detected.

2. The BEV space target detection method of multi-perspective images according to claim 1, characterized in that: The depth feature extractor extracts the depth feature of each pixel in the perspective image feature map to generate a depth feature map, including: Determine the probability value of each depth category corresponding to each pixel point of the perspective image feature map, and generate the depth feature map according to the probability values ​​of each depth category corresponding to all pixels points; The generation of the depth categories corresponding to each pixel includes: dividing the preset depth range corresponding to the pixel according to preset intervals to generate a plurality of depth categories corresponding to the pixel.

3. The BEV space target detection method of multi-perspective images according to claim 2, characterized in that: The step of generating a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels based on the foreground-background mask feature map includes: Extracting the mask value of each pixel in the foreground and background mask feature map; Compare the mask value of each pixel with the preset threshold; The mask values ​​of pixels whose mask values ​​are greater than a preset threshold are updated to the preset foreground mask value, and the mask values ​​of pixels whose mask values ​​are less than the preset threshold are updated to the preset background mask value, thereby generating a background mask feature map before binarization.

4. The BEV space target detection method of multi-perspective images according to claim 3, characterized in that: The performing topK processing on each depth category corresponding to each pixel point in the depth feature map to obtain a topK probability feature map and a topK index feature map includes: For each pixel, sort the probability values ​​of each depth category corresponding to the pixel in descending order, and then obtain the top k probability values; Generate a topK probability feature map based on the topK probability values ​​of all pixels and the depth categories corresponding to the topK probability values; For each pixel point, the index value of the top K probability values ​​of the pixel point is determined according to the index position of the top K probability values ​​of the pixel point before sorting; and the topK index feature map is generated according to the index values ​​of the top K probability values ​​corresponding to all pixels.

5. The BEV space target detection method of multi-perspective images according to claim 4, characterized in that: The calculating of the probability weighted feature map according to the topK probability feature map and the perspective image feature map includes: The K probability values ​​of each pixel point on the topK probability feature map are multiplied by the feature values ​​of each channel of the corresponding pixel point in the perspective image feature map to generate a probability weighted feature map.

6. The BEV space target detection method of multi-perspective images according to claim 5, characterized in that: The calculating of the topK depth feature map according to the topK index feature map includes: For each pixel in the topK index feature map, determine the depth category corresponding to each index value based on the index value of the first K probability values ​​of the pixel; extract the center point of the depth range corresponding to each depth category to generate the k depth values ​​corresponding to the pixel; Generate topK depth feature maps based on all depth values ​​of all pixels.

7. The BEV space target detection method of multi-perspective images according to claim 6, characterized in that: The generating of the BEV mapping coordinates of each pixel in the BEV coordinate system according to the depth value of each pixel in the topK depth feature map and the pixel coordinates of each pixel includes: Combine the depth values ​​of all pixels in the topK depth feature map with the pixel coordinates of the corresponding pixels to generate three-dimensional space coordinates; Performing depth inverse normalization processing on the three-dimensional space coordinates to generate inverse normalized coordinates; Mapping the inverse normalized three-dimensional coordinates to a camera coordinate system of a corresponding camera to generate the camera three-dimensional coordinates in the camera coordinate system; The camera 3D coordinates are mapped to the radar coordinate system to generate the BEV mapping coordinates of each pixel point in the BEV coordinate system.

8. The BEV space target detection method of multi-perspective images according to claim 7, characterized in that: The method extracts the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map according to the positions of the foreground pixels in the binary mask feature map, and then performs foreground feature projection to generate a BEV feature map, including: Extracting the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map according to the position of each foreground mask value in the binary mask feature map, thereby obtaining the BEV mapping coordinates of the foreground pixels and the probability weighted feature map of the foreground pixels, respectively; Rasterizing the preset BEV space to obtain a number of BEV grids, filtering out the BEV mapping coordinates of foreground pixels that are not within the BEV grids, and updating the BEV mapping coordinates of the foreground pixels and the probability weighted feature map of the foreground pixels based on the filtered BEV mapping coordinates; Generate the coordinates of the valid BEV mapping points and the features of the valid BEV mapping points according to the updated BEV mapping coordinates of the foreground pixel points and the updated probability weighted feature map of the foreground; According to the coordinates and features of the valid BEV mapping points, foreground features are projected to generate a BEV feature map.

9. The BEV space target detection method of multi-perspective images according to claim 8, characterized in that: Detecting the position and category information of the target to be detected based on the BEV feature map includes: Inputting the BEV feature map into a BEV encoder in a BEV target detector so that the BEV encoder encodes the BEV feature map to generate a BEV encoded feature map; Inputting the BEV encoding feature map into a BEV decoder in a BEV target detector so that the BEV decoder decodes the BEV encoding feature map to generate a BEV decoding feature map; According to the BEV decoding feature map, the position and category information of the target to be detected are generated.

10. A BEV space target detection device based on multi-perspective images, characterized in that: include: Multi-perspective image acquisition module and target recognition module; The target recognition module includes: a perspective image feature map generation unit, a depth feature map generation unit, a foreground and background mask feature map generation unit, a binarized foreground and background mask feature map generation unit, a BEV mapping coordinate generation unit, a BEV feature map generation unit, and a detection unit; The multi-eye perspective image acquisition module is used to acquire a multi-eye perspective image of the target to be detected; The target recognition module is used to input the multi-perspective image into a preset BEV target detector so that the BEV target detector recognizes the position and category information of the target to be detected; The perspective image feature map generating unit is configured to extract features from the multi-view perspective image using an image feature extractor within a BEV object detector to generate a perspective image feature map; The depth feature map generating unit is configured to input the perspective image feature map into a depth feature extractor in a BEV object detector, so that the depth feature extractor extracts the depth features of each pixel in the perspective image feature map to generate a depth feature map; The foreground-background mask feature map generating unit is used to input the perspective image feature map into the foreground-background mask extractor in the BEV object detector, so that the foreground-background mask extractor extracts the mask features of the foreground pixels and the background pixels in the perspective image feature map to generate the foreground-background mask feature map; The binary foreground-background mask feature map generating unit is used to generate a binary foreground-background mask feature map for distinguishing foreground pixels from background pixels based on the foreground-background mask feature map; The BEV mapping coordinate generation unit is used to perform topK processing on each depth category corresponding to each pixel point in the depth feature map to obtain a topK probability feature map and a topK index feature map; calculate a probability weighted feature map based on the topK probability feature map and the perspective image feature map; calculate a topK depth feature map based on the topK index feature map; generate a BEV mapping coordinate of each pixel point in the BEV coordinate system based on the depth value of each pixel point in the topK depth feature map and the pixel coordinate of each pixel point; The BEV feature map generating unit is used to extract the BEV mapping coordinates of the foreground pixels in all BEV mapping coordinates and the probability weighted features of the foreground pixels in the probability weighted feature map according to the positions of the foreground pixels in the binary mask feature map, and then perform foreground feature projection to generate a BEV feature map; The detection unit is used to detect the position and category information of the target to be detected based on the BEV feature map.

Images