Three-dimensional target detection method and device, electronic equipment and storage medium
By acquiring pseudo-point cloud features in three-dimensional object detection and separating depth and semantic weights, combining cascade operation and encoder convolution processing, the problem of insufficient accuracy of object detection in the prior art is solved, and the prospect and background features are realized The adaptive fusion of the detection results improves the accuracy of the detection results.
Patent Information
- Application Number
- CN202411296171.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-12-27
AI Technical Summary
In the feature conversion between perspective view angle to bird's-eye view angle, it is difficult to ensure the accuracy of object detection, especially due to the immaturity of semantic information extraction in the BEV space and incomplete filtering of background information.
By obtaining the pseudo-point cloud feature set of the image to be detected, using depth and semantic weights for separation, the foreground and background feature sets are obtained, and cascade operations are performed on the feature channel dimension, and convolution is combined with the trained encoder to perform convolution processing, so as to achieve Adaptive fusion of foreground and background features.
It effectively improves the accuracy of object detection, avoids feature loss, and ensures the accuracy of detection results and information integrity.
Smart Images
Figure CN119206177A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of target detection technology, and in particular to a three-dimensional target detection method and device, electronic equipment, and storage medium. Background Art
[0002] The most cutting-edge three-dimensional target detection method in autonomous driving scenarios is to obtain the features of the perspective image features in the bird's eye view (BEV) space, namely the BEV features, and then perform target detection based on the BEV features.
[0003] When performing target detection based on BEV features, one way is to directly convert the image features of the perspective view into BEV features, so that all the information in the scene can be retained. When performing target detection later, target detection is achieved by performing scene semantic information recognition on the BEV features. Another way is to perform semantic recognition on the image features of the perspective view in advance, and convert the image features of the perspective view into BEV features based on the semantic recognition, thereby filtering out the features of the background that are semantically recognized, so that the BEV features can be directly used for target detection later.
[0004] However, for the first method, since most of the current work on extracting semantic information from image modalities is done in perspective space, the method of extracting semantic information in BEV space is not mature yet, so the accuracy of target detection cannot be guaranteed. For the second method, it is easy to fail to completely filter out all background information, or foreground information is incorrectly filtered out, which will also affect the accuracy of target detection. Summary of the invention
[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a three-dimensional target detection method and device, electronic device, and storage medium to solve the problem that the prior art cannot effectively guarantee the accuracy of target detection.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] The first aspect of the present application provides a three-dimensional object detection method, comprising:
[0008] Get the current image to be detected;
[0009] Calculate a pseudo point cloud feature set of the current image to be detected; wherein the pseudo point cloud feature set includes each pseudo point cloud feature obtained by weighting the image feature elements at multiple depths using the depth weights of each image feature element on the image to be detected;
[0010] Based on each of the image feature elements, predicting a semantic weight corresponding to each of the pseudo point cloud features;
[0011] By comparing the weight corresponding to each of the pseudo point cloud features with a preset weight threshold, each of the pseudo point cloud features in the pseudo point cloud feature set is separated to obtain a foreground feature set belonging to a foreground area and a background feature set belonging to a background area; wherein the weight corresponding to the pseudo point cloud features includes the semantic weight and the depth weight;
[0012] Converting the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively;
[0013] In the feature channel dimension, the foreground bird's-eye view feature set and the background bird's-eye view feature set are cascaded to obtain a cascaded bird's-eye view feature set of the current image to be detected;
[0014] Performing convolution processing on the cascaded bird's-eye view feature set of the current image to be detected by using a trained encoder to obtain a fused bird's-eye view feature set;
[0015] The fused bird's-eye view feature set is used to perform target detection to obtain a current target detection result.
[0016] Optionally, in the above three-dimensional object detection method, the convolution process is performed on the cascaded bird's-eye view feature set of the current image to be detected by the trained encoder to obtain a fused bird's-eye view feature set, including:
[0017] Through the first-order module of the trained bird's-eye view encoder, the cascaded bird's-eye view feature set of the current image to be detected is convolved according to the dimension of the foreground bird's-eye view feature set to obtain the fused bird's-eye view feature set; wherein the first-order module of the bird's-eye view encoder is iteratively trained in advance based on the target detection error.
[0018] Optionally, in the above three-dimensional object detection method, the method further includes:
[0019] Based on the error between the current target detection result and the current true result, the current loss gradient is detected, and the current loss gradient is transferred to the first-order module of the bird's-eye view encoder through the back propagation algorithm to optimize the parameters of the first-order module of the bird's-eye view encoder.
[0020] Optionally, in the above three-dimensional object detection method, the step of calculating a pseudo point cloud feature set of the current image to be detected includes:
[0021] For each of the image feature elements of the current image to be detected, predicting a depth weight of the image feature element at a plurality of discretized depth values based on the image feature element;
[0022] Respectively weighting the depth weight of the image feature element at each discretized depth value and the image feature element to obtain each pseudo point cloud feature corresponding to the image feature element;
[0023] The pseudo point cloud features corresponding to the image feature elements are combined into a pseudo point cloud feature set of the current image to be detected.
[0024] Optionally, in the above-mentioned three-dimensional object detection method, the pseudo point cloud features in the pseudo point cloud feature set are separated by comparing the weights corresponding to the pseudo point cloud features with a preset weight threshold to obtain a foreground feature set belonging to the foreground area and a background feature set belonging to the background area, including:
[0025] Filtering out from the pseudo point cloud feature set the pseudo point cloud features belonging to the foreground area whose semantic weight is not less than the foreground semantic threshold in the preset weight threshold and whose depth weight is not less than the foreground depth threshold in the preset weight threshold, to obtain the foreground feature set belonging to the foreground area;
[0026] The pseudo point cloud features in the pseudo point cloud feature set retained after filtering are combined into the background feature set belonging to the background area.
[0027] Optionally, in the above three-dimensional object detection method, converting the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively, comprises:
[0028] Calculating the coordinates of each pseudo point corresponding to each image feature element according to each of the depth values and the internal and external parameters of the camera that collects the current image to be detected;
[0029] According to the set perception range and resolution, the three-dimensional space image composed of each of the pseudo points is divided into a plurality of square columns;
[0030] Adding the features in the foreground feature set corresponding to each of the pseudo points whose coordinates are located in the same square column to obtain the foreground bird's-eye view features corresponding to each of the direction columns, and adding the features in the background feature set corresponding to each of the pseudo points whose coordinates are located in the same square column to obtain the background bird's-eye view features corresponding to each of the direction columns;
[0031] The foreground bird's-eye view features corresponding to the direction columns are combined into the foreground bird's-eye view feature set, and the background bird's-eye view features corresponding to the direction columns are combined into the background bird's-eye view feature set.
[0032] A second aspect of the present application provides a three-dimensional object detection device, comprising:
[0033] An image acquisition unit, used to acquire the current image to be detected;
[0034] A pseudo point cloud computing unit is used to calculate a pseudo point cloud feature set of the current image to be detected; wherein the pseudo point cloud feature set includes each pseudo point cloud feature obtained by weighting the image feature elements on the image to be detected using the depth weights of the image feature elements at multiple depths;
[0035] A first prediction unit, configured to predict, based on each of the image feature elements, a semantic weight corresponding to each of the pseudo point cloud features;
[0036] A separation unit, configured to separate each of the pseudo point cloud features in the pseudo point cloud feature set by comparing the weight corresponding to each of the pseudo point cloud features with a preset weight threshold, so as to obtain a foreground feature set belonging to a foreground area and a background feature set belonging to a background area; wherein the weight corresponding to the pseudo point cloud features includes the semantic weight and the depth weight;
[0037] A conversion unit, used to convert the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively;
[0038] A cascade operation unit, configured to perform a cascade operation on the foreground bird's-eye view feature set and the background bird's-eye view feature set in a feature channel dimension to obtain a cascaded bird's-eye view feature set of the current image to be detected;
[0039] A fusion unit, configured to perform convolution processing on the cascaded bird's-eye view feature set of the current image to be detected through a trained encoder to obtain a fused bird's-eye view feature set;
[0040] The detection unit is used to perform target detection using the fused bird's-eye view feature set to obtain a current target detection result.
[0041] Optionally, in the above three-dimensional object detection device, the fusion unit includes:
[0042] The fusion subunit is used to perform a convolution operation on the cascaded bird's-eye view feature set of the current image to be detected according to the dimension of the foreground bird's-eye view feature set through the first-order module of the trained bird's-eye view encoder, so as to obtain the fused bird's-eye view feature set; wherein the first-order module of the bird's-eye view encoder is iteratively trained in advance based on the target detection error.
[0043] Optionally, in the above three-dimensional object detection device, it also includes:
[0044] An optimization unit is used to detect a current loss gradient based on an error between the current target detection result and the current true result, and to transfer the current loss gradient to the first-order module of the bird's-eye view encoder through a back-propagation algorithm to optimize the parameters of the first-order module of the bird's-eye view encoder.
[0045] Optionally, in the above-mentioned three-dimensional object detection device, the pseudo-point cloud computing unit includes:
[0046] A second prediction unit is used to predict, for each of the image feature elements of the current image to be detected, a depth weight of the image feature element at a plurality of discretized depth values based on the image feature element;
[0047] A weighting unit, configured to weight the depth weight of the image feature element at each discretized depth value and the image feature element, respectively, to obtain each pseudo point cloud feature corresponding to the image feature element;
[0048] The first combining unit is used to combine the pseudo point cloud features corresponding to the image feature elements into a pseudo point cloud feature set of the current image to be detected.
[0049] Optionally, in the above three-dimensional object detection device, the separation unit includes:
[0050] A filtering unit, configured to filter out from the pseudo point cloud feature set each of the pseudo point cloud features whose semantic weight belonging to the foreground area is not less than the foreground semantic threshold in the preset weight threshold and whose depth weight is not less than the foreground depth threshold in the preset weight threshold, to obtain the foreground feature set belonging to the foreground area;
[0051] The composition unit is used to combine the pseudo point cloud features in the pseudo point cloud feature set retained after filtering into the background feature set belonging to the background area.
[0052] Optionally, in the above three-dimensional object detection device, the conversion unit includes:
[0053] A coordinate calculation unit, used to calculate the coordinates of each pseudo point corresponding to each of the image feature elements according to each of the depth values and the internal and external parameters of the camera that collects the current image to be detected;
[0054] A space division unit, used to divide the three-dimensional space image composed of each of the pseudo points into a plurality of square columns according to a set perception range and resolution;
[0055] a summing unit, configured to add the features in the foreground feature set corresponding to each of the pseudo points whose coordinates are located in the same square column to obtain the foreground bird's-eye view features corresponding to each of the direction columns, and to add the features in the background feature set corresponding to each of the pseudo points whose coordinates are located in the same square column to obtain the background bird's-eye view features corresponding to each of the direction columns;
[0056] The second combining unit is used to combine the foreground bird's-eye view features corresponding to the direction columns into the foreground bird's-eye view feature set, and to combine the background bird's-eye view features corresponding to the direction columns into the background bird's-eye view feature set.
[0057] A third aspect of the present application provides an electronic device, including:
[0058] Memory and processor;
[0059] Wherein, the memory is used to store programs;
[0060] The processor is used to execute the program, and when the program is executed, it is specifically used to implement the three-dimensional target detection method as described in any one of the above.
[0061] A fourth aspect of the present application provides a computer storage medium for storing a computer program, which, when executed, is used to implement a three-dimensional target detection method as described in any one of the above.
[0062] The embodiment of the present application provides a three-dimensional target detection method. After obtaining the current image to be detected, the pseudo point cloud feature set of the current image to be detected is first calculated, and based on each image feature element, the semantic weight corresponding to each pseudo point cloud feature is predicted, so that the semantics are extracted under the perspective view to ensure the accuracy of the extracted semantic information. Among them, the pseudo point cloud feature set includes the image feature elements weighted by the depth weights of each image feature element on the image to be detected at multiple depths, and the pseudo point cloud features obtained. Then, by comparing the weights corresponding to each pseudo point cloud feature with the preset weight threshold, the pseudo point cloud features in the pseudo point cloud feature set are separated to obtain a foreground feature set belonging to the foreground area and a background feature set belonging to the background area. Among them, the weights corresponding to the pseudo point cloud features include semantic weights and depth weights, thereby respectively segmenting the features belonging to the foreground area and the background area from the semantics and depth, not only extracting the features belonging to the foreground area. Then, the foreground feature set and the background feature set are converted into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively. Then, in the feature channel dimension, the foreground bird's-eye view feature set and the background bird's-eye view feature set are cascaded to obtain the cascaded bird's-eye view feature set of the current image to be detected, and the cascaded bird's-eye view feature set of the current image to be detected is convolved by the trained encoder to obtain the fused bird's-eye view feature set, that is, to achieve adaptive fusion of the features of the two regions to ensure that the features required for target detection are obtained. Finally, the fused bird's-eye view feature set is used for target detection to obtain the current target detection result. Since the features of the foreground and background regions are retained, there is no feature loss, and the information required for detection can still be obtained through adaptive fusion, thus effectively ensuring the accuracy of target detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0064] Figure 1 A flowchart of a three-dimensional target detection method provided in an embodiment of the present application;
[0065] Figure 2 A flowchart of a method for calculating a pseudo point cloud feature set provided in an embodiment of the present application;
[0066] Figure 3 A flowchart of a method for segmenting foreground features and background features provided in an embodiment of the present application;
[0067] Figure 4 A flowchart of a method for converting a feature set into a feature set from a bird's-eye view provided in an embodiment of the present application;
[0068] Figure 5 A schematic diagram of the architecture of a three-dimensional object detection device provided in an embodiment of the present application;
[0069] Figure 6 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0071] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0072] The present application embodiment provides a three-dimensional target detection method, such as Figure 1 As shown, the following steps are included:
[0073] S101, obtaining the current image to be detected.
[0074] The current image to be detected refers to an image that needs to be detected currently, specifically an image acquired under a perspective view, and also an image transformed into a bird's-eye view.
[0075] S102: Calculate a pseudo point cloud feature set of the current image to be detected.
[0076] The pseudo point cloud feature set includes various pseudo point cloud features obtained by weighting image feature elements using depth weights of various image feature elements at multiple depths on the image to be detected.
[0077] It should be noted that for a scene in an image, it can be divided into background and foreground. Since the detected object is usually in the foreground, and the background and foreground in the image can be distinguished by depth information, it is necessary to extract the corresponding semantic feature information so that the object detection can be performed based on it in the subsequent process.
[0078] Therefore, in an embodiment of the present application, each image feature element of the current image to be detected is specifically determined first. Among them, the image feature element refers to a vector corresponding to a width and height coordinate on the image feature. Then, the depth weights of each image feature element at multiple depths can be predicted based on the image feature elements through the model backbone network, which is equivalent to the probability that the image feature element may belong to each depth. Then, the predicted image feature elements are weighted to obtain weighted features. Compared with the image features obtained directly, the weighted features are called pseudo-point cloud features. That is, for an image feature element, multiple corresponding pseudo-point cloud features consistent with the number of depth values will be calculated. The set of each pseudo-point cloud feature is the pseudo-point cloud feature set of the current image to be detected.
[0079] Specifically, for an image feature i, its depth weight at a depth value is W D , then the corresponding pseudo point cloud feature p is calculated as:
[0080] p=W D i
[0081] Optionally, in another embodiment of the present application, a specific implementation of step S102 is as follows: Figure 2 As shown, the following steps are included:
[0082] S201 . For each image feature element of the current image to be detected, predict a depth weight of the image feature element at a plurality of discretized depth values based on the image feature element.
[0083] Specifically, in the embodiment of the present application, the entire depth value range is discretized into a plurality of depth values, and then the depth weight of the image feature element at each depth value is predicted.
[0084] S202 , respectively weighting the depth weights of the image feature elements at each discretized depth value and the image feature elements to obtain each pseudo point cloud feature corresponding to the image feature element.
[0085] S203 , composing the pseudo point cloud features corresponding to the image feature elements into a pseudo point cloud feature set of the current image to be detected.
[0086] S103: Based on each image feature element, predict the semantic weight corresponding to each pseudo point cloud feature.
[0087] In order to perform semantic extraction and semantically segment features of different scenes, in an embodiment of the present application, not only is the depth weight predicted based on the image feature elements, but additional channels can also be added to the existing depth scores to predict the semantic weights of the image feature elements.
[0088] Optionally, since only the features belonging to the foreground area or the background area need to be screened out, the features of the other scene can also be obtained accordingly, so only the semantic weight belonging to one of the scene areas can be predicted. Therefore, optionally, step S103 can specifically be to only predict the semantic weight of each pseudo point cloud feature belonging to the foreground area, so that the features belonging to the foreground area can be screened out based on the semantic weight, and then the features belonging to the background area can be obtained.
[0089] S104 , separating each pseudo point cloud feature in the pseudo point cloud feature set by comparing the weight corresponding to each pseudo point cloud feature with a preset weight threshold, to obtain a foreground feature set belonging to the foreground area and a background feature set belonging to the background area.
[0090] The weight corresponding to the pseudo point cloud feature includes a semantic weight and a depth weight. The depth weight corresponding to the pseudo point cloud feature is the depth weight of the pseudo point cloud feature obtained by weighting the image feature element.
[0091] It should be noted that, since the depth weight and semantic weight corresponding to the feature are used to determine whether it belongs to the foreground area or the background area in terms of depth and semantics. Therefore, in the embodiment of the present application, a weight threshold for dividing the foreground area and the background area is pre-set. Among them, the preset weight threshold includes a preset semantic threshold and a depth threshold.
[0092] Therefore, by comparing the semantic weights corresponding to the pseudo point cloud features with the semantic threshold, the pseudo point cloud features belonging to the foreground area and the background area are semantically segmented, and by comparing the semantic weights corresponding to the pseudo point cloud features with the semantic threshold, the pseudo point cloud features belonging to the foreground area and the background area are segmented from the depth information. At the same time, the pseudo point cloud features that meet the requirements in terms of semantic information and depth information constitute the foreground feature set belonging to the foreground area and the background feature set belonging to the background area.
[0093] Optionally, the weight threshold can be obtained through experiments under the constraint that the accuracy of the detection result meets the requirements.
[0094] Optionally, since the features of one scene can be divided, the features of another scene can be obtained, so the preset weight threshold can only include the semantic threshold and depth threshold belonging to the foreground area, that is, the foreground semantic threshold and the foreground depth threshold. Of course, the semantic threshold and weight threshold belonging to the foreground area or the background area can be set according to the predicted semantic weight of the scene area.
[0095] Optionally, in another embodiment of the present application, a specific implementation of step S104 is as follows: Figure 3 As shown, the following steps are included:
[0096] S301. Filter out from the pseudo point cloud feature set the pseudo point cloud features whose semantic weight belonging to the foreground area is not less than the foreground semantic threshold in the preset weight threshold and whose depth weight is not less than the foreground depth threshold in the preset weight threshold, to obtain a foreground feature set belonging to the foreground area.
[0097] Among them, in the embodiment of the present application, what is predicted is the semantic weight and depth weight of the pseudo point cloud feature belonging to the foreground area, so the corresponding preset weight threshold is set to the foreground semantic threshold and foreground weight threshold belonging to the foreground.
[0098] Optionally, feature filtering can be achieved by setting a filter function. The specific filter function is as follows:
[0099]
[0100] Therefore, the background is filtered out accordingly, and only the pseudo point cloud features belonging to the foreground area can be specifically expressed as:
[0101] p F =F(W F ,T F )F(W D ,T D )·p
[0102] Among them, W D and W F are the depth weight and semantic weight of the feature belonging to the foreground area; T D and T F is the foreground depth threshold and foreground semantic threshold. F is the set filter function.
[0103] S302 , each pseudo point cloud feature in the pseudo point cloud feature set retained after filtering is combined into a background feature set belonging to the background area.
[0104] After filtering out the foreground, what is retained is the pseudo point cloud features belonging to the background area, so the pseudo point cloud features belonging to the background area can be specifically expressed as:
[0105]
[0106] S105 . Convert the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively.
[0107] Since what is ultimately needed is the features from a bird's-eye view, the foreground feature set and the background feature set are converted into feature sets from a bird's-eye view, respectively, to obtain a foreground bird's-eye view feature set and a background bird's-eye view feature set. The foreground bird's-eye view feature set is the feature set of the foreground feature set from a bird's-eye view. The background bird's-eye view feature set is the feature set of the background feature set from a bird's-eye view.
[0108] Optionally, in another embodiment of the present application, a specific implementation of step S105 is as follows: Figure 4 As shown, the following steps are included:
[0109] S401 , calculating the coordinates of each pseudo point corresponding to each image feature element according to each depth value and the internal and external parameters of the camera that collects the current image to be detected.
[0110] Specifically, the internal and external parameters of the camera that collects the current image to be detected can determine the coordinates corresponding to the image feature elements, and then further combined with each depth value, the coordinates of each pseudo point corresponding to each image feature element in the three-dimensional space can be obtained.
[0111] S402: Divide the three-dimensional space image composed of the pseudo points into a plurality of square columns according to the set perception range and resolution.
[0112] It should be noted that the perception range and resolution of the vehicle are the requirements for imaging, so it is necessary to divide the three-dimensional space composed of each pseudo point into multiple square columns according to the perception range and resolution.
[0113] S403, adding the features in the foreground feature set corresponding to each pseudo point whose coordinates are located in the same square column to obtain the foreground bird's-eye view features corresponding to the columns in each direction, and adding the features in the background feature set corresponding to each pseudo point whose coordinates are located in the same square column to obtain the background bird's-eye view features corresponding to the columns in each direction.
[0114] The feature corresponding to the pseudo point is a feature obtained by weighting the depth weight predicted by the depth value corresponding to the pseudo point and the image feature element corresponding to the pseudo point.
[0115] Since the foreground feature set and the background feature set need to be converted separately, the positions of the foreground feature set and the background feature set need to be judged and calculated separately.
[0116] S404: The foreground bird's-eye view features corresponding to the columns in each direction are combined into a foreground bird's-eye view feature set, and the background bird's-eye view features corresponding to the columns in each direction are combined into a background bird's-eye view feature set.
[0117] S106. In the feature channel dimension, a foreground bird's-eye view feature set and a background bird's-eye view feature set are cascaded to obtain a cascaded bird's-eye view feature set of the current image to be detected.
[0118] In order to fuse the foreground bird's-eye view feature set and the background bird's-eye view feature set as a whole, in the embodiment of the present application, the foreground bird's-eye view feature set and the background bird's-eye view feature set are first cascaded in the feature channel dimension to obtain a cascaded bird's-eye view feature set of the current image to be detected, which can be specifically expressed as:
[0119] B C =Concat(B F ,B B )
[0120] Among them, Concat represents a cascade operation; B F and B B They represent the foreground BEV feature set and the background BEV feature set, respectively, and the dimensions of both are N*C*H*W. N represents the sample data; C represents the feature channel, and H and W represent the height and width of the feature, respectively. Therefore, the dimension of the cascade feature set obtained after the cascade operation is N*2C*H*W.
[0121] S107 . Perform convolution processing on the cascaded bird's-eye view feature set of the current image to be detected through the trained encoder to obtain a fused bird's-eye view feature set.
[0122] Since the feature set after the simple cascade combination has not yet merged the features in the two feature sets into the features that can be used later, it is necessary to further perform a convolution operation to map the current features to the features that can be used later. In this process, the foreground features and background features will be merged.
[0123] It should be noted that in order to ensure that the fused features are more conducive to target detection and effectively improve the accuracy of target detection, in the embodiment of the present application, the encoder performs adaptive fusion of features. It can continuously iterate and update parameters according to the error of the final target detection.
[0124] Optionally, in another embodiment of the present application, a specific implementation of step S107 includes the following steps:
[0125] Through the first-order module of the trained bird's-eye view encoder, the cascaded bird's-eye view feature set of the current image to be detected is convolved according to the dimension of the foreground bird's-eye view feature set to obtain a fused bird's-eye view feature set.
[0126] Among them, the first-order module of the bird's-eye view encoder is pre-trained by iterative training based on the target detection error.
[0127] It should be noted that after the cascade operation, the number of channels is doubled, and increasing the number of channels will lead to an increase in computational costs. In order to ensure the lightweight of the model, in an embodiment of the present application, during the convolution operation, the first-order module of the BEV encoder is used to reduce the dimension of the cascaded BEV feature set, reducing its channel number to the original number of channels C.
[0128] Specifically, the convolution process can be expressed as:
[0129]
[0130] in, Represents the first order module of the BEV encoder.
[0131] The scene information represented by the adaptive BEV feature set is uncertain, so in the embodiment of the present application, the following is further performed:
[0132] Based on the error between the current target detection result and the current true result, the current loss gradient is detected, and the current loss gradient is transferred to the first-order module of the bird's-eye view encoder through the back propagation algorithm to optimize the parameters of the first-order module of the bird's-eye view encoder.
[0133] That is, according to the neural network back-propagation algorithm, the loss gradient obtained by calculating the target detection loss is passed to the first-order module of the BEV encoder, and its parameters are continuously updated, so that it can continuously adapt to the quality of the BEV feature set, thereby obtaining features that are more suitable for three-dimensional target detection in autonomous driving scenarios and improving the model's perception ability.
[0134] S108. Utilize the fused bird's-eye view feature set to perform target detection and obtain a current target detection result.
[0135] It should be noted that since BE features can be applied not only to target detection but also to other scenarios, the method for obtaining the fused bird's-eye view feature set provided in the embodiment of the present application can also be applied to other scenarios, such as BEVDet, BEVStereo, etc.
[0136] The embodiment of the present application provides a three-dimensional target detection method. After obtaining the current image to be detected, the pseudo point cloud feature set of the current image to be detected is first calculated, and based on each image feature element, the semantic weight corresponding to each pseudo point cloud feature is predicted, so that the semantics are extracted under the perspective view to ensure the accuracy of the extracted semantic information. Among them, the pseudo point cloud feature set includes the image feature elements weighted by the depth weights of each image feature element on the image to be detected at multiple depths, and the pseudo point cloud features obtained. Then, by comparing the weights corresponding to each pseudo point cloud feature with the preset weight threshold, the pseudo point cloud features in the pseudo point cloud feature set are separated to obtain a foreground feature set belonging to the foreground area and a background feature set belonging to the background area. Among them, the weights corresponding to the pseudo point cloud features include semantic weights and depth weights, thereby respectively segmenting the features belonging to the foreground area and the background area from the semantics and depth, not only extracting the features belonging to the foreground area. Then, the foreground feature set and the background feature set are converted into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively. Then, in the feature channel dimension, the foreground bird's-eye view feature set and the background bird's-eye view feature set are cascaded to obtain the cascaded bird's-eye view feature set of the current image to be detected, and the cascaded bird's-eye view feature set of the current image to be detected is convolved with the trained encoder to obtain a fused bird's-eye view feature set to achieve adaptive fusion of features of the two areas. Finally, the fused bird's-eye view feature set is used for target detection to obtain the current target detection result. Since the features of the foreground area and the background area are retained, there is no feature loss, and the information required for detection can still be obtained through adaptive fusion, thereby effectively ensuring the accuracy of target detection.
[0137] In order to verify the effectiveness of the method provided in the embodiments of the present application, further experiments were conducted.
[0138] The nuScenes dataset is a relatively common autonomous driving dataset, and provides annotations and related evaluation indicators for 3D object detection. In order to demonstrate the effectiveness of the adaptive fusion of foreground and background information in the autonomous driving scene through experiments, the present invention was evaluated on the nuScenes dataset, and the evaluation indicators were compared with the method without semantic extraction and the method using only foreground information.
[0139] Among them, the main evaluation indicators used are:
[0140] mAP (meanAverage Precision): Different from the commonly used mAP calculation method, it first matches the target value and the predicted value by checking whether the distance between the center points is within a specific threshold. This can prevent the size difference of different targets from causing deviations in the evaluation of perception ability.
[0141] NDS (nuScenes Detection Score): Comprehensively evaluates the model’s ability to predict different attributes of the target. The specific calculation formula is:
[0142]
[0143] Among them, mTP includes the average translation error, average scale error, average orientation error, average speed error, and average attribute error, and only calculates the target prediction value error that can match the target true value. These five errors are weighted with mAP to obtain NDS, so it can be used to measure the overall detection ability of the model.
[0144] The comparison results of specific experimental data are shown in Table 1 below:
[0145] Table 1
[0146] method mAP NDS Undivided 0.3363 0.4530 Use only foreground information 0.3458 0.4561 Adaptive fusion of foreground and background 0.3474 0.4608
[0147] Table 1 shows the accuracy improvement that the present invention can bring after adaptively fusing the foreground and background information of the autonomous driving scene. Specifically, it can be observed that after semantic segmentation is performed on the image features and background information is filtered out, the mAP of the model increased by 0.95%, indicating that extracting semantic information in the perspective space and using it to generate BEV features can effectively improve the perception ability of the model. After adaptive fusion of the foreground and background, the mAP of the model further increased by 0.16%, and the NDS increased by 0.47%, indicating that further adaptive fusion of the foreground and background information of the autonomous driving scene on the basis of extracting semantic information can screen out more appropriate key information for improving the accuracy of three-dimensional target detection, confirming the effectiveness of the present invention.
[0148] Another embodiment of the present application provides a three-dimensional object detection device, such as Figure 5 As shown, it includes the following units:
[0149] The image acquisition unit 501 is used to acquire the current image to be detected.
[0150] The pseudo point cloud computing unit 502 is used to calculate a pseudo point cloud feature set of the current image to be detected, wherein the pseudo point cloud feature set includes each pseudo point cloud feature obtained by weighting the image feature elements at multiple depths using the depth weights of each image feature element on the image to be detected.
[0151] The first prediction unit 503 is used to predict the semantic weight corresponding to each pseudo point cloud feature based on each image feature element.
[0152] The separation unit 504 is used to separate each pseudo point cloud feature in the pseudo point cloud feature set by comparing the weight corresponding to each pseudo point cloud feature with a preset weight threshold, so as to obtain a foreground feature set belonging to the foreground area and a background feature set belonging to the background area. The weight corresponding to the pseudo point cloud feature includes a semantic weight and a depth weight.
[0153] The conversion unit 505 is used to convert the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively.
[0154] The cascade operation unit 506 is used to perform a cascade operation on the foreground bird's-eye view feature set and the background bird's-eye view feature set in the feature channel dimension to obtain a cascaded bird's-eye view feature set of the current image to be detected.
[0155] The fusion unit 507 is used to perform convolution processing on the cascaded bird's-eye view feature set of the current image to be detected through the trained encoder to obtain a fused bird's-eye view feature set.
[0156] The detection unit 508 is used to perform target detection using the fused bird's-eye view feature set to obtain a current target detection result.
[0157] Optionally, in a three-dimensional object detection device provided in another embodiment of the present application, the fusion unit includes:
[0158] The fusion subunit is used to perform a convolution operation on the cascaded bird's-eye view feature set of the current image to be detected according to the dimension of the foreground bird's-eye view feature set through the first-order module of the trained bird's-eye view encoder to obtain a fused bird's-eye view feature set. The first-order module of the bird's-eye view encoder is pre-trained iteratively based on the target detection error.
[0159] Optionally, in another embodiment of the present application, the three-dimensional object detection device further includes:
[0160] The optimization unit is used to detect the current loss gradient based on the error between the current target detection result and the current true result, and transfer the current loss gradient to the first-order module of the bird's-eye view encoder through the back propagation algorithm to optimize the parameters of the first-order module of the bird's-eye view encoder.
[0161] Optionally, in the three-dimensional object detection device provided in another embodiment of the present application, the pseudo-point cloud computing unit includes:
[0162] The second prediction unit is used to predict, for each image feature element of the current image to be detected, a depth weight of the image feature element at a plurality of discretized depth values based on the image feature element.
[0163] The weighting unit is used to weight the depth weight of the image feature element at each discretized depth value and the image feature element respectively to obtain each pseudo point cloud feature corresponding to the image feature element.
[0164] The first combining unit is used to combine the pseudo point cloud features corresponding to the image feature elements into a pseudo point cloud feature set of the current image to be detected.
[0165] Optionally, in a three-dimensional object detection device provided in another embodiment of the present application, the separation unit includes:
[0166] The filtering unit is used to filter out from the pseudo point cloud feature set each pseudo point cloud feature whose semantic weight belonging to the foreground area is not less than the foreground semantic threshold in the preset weight threshold and whose depth weight is not less than the foreground depth threshold in the preset weight threshold, to obtain a foreground feature set belonging to the foreground area.
[0167] The composition unit is used to form a background feature set belonging to the background area by combining the pseudo point cloud features in the pseudo point cloud feature set retained after filtering.
[0168] Optionally, in a three-dimensional object detection device provided in another embodiment of the present application, the conversion unit includes:
[0169] The coordinate calculation unit is used to calculate the coordinates of each pseudo point corresponding to each image feature element according to each depth value and the internal and external parameters of the camera that collects the current image to be detected.
[0170] The space division unit is used to divide the three-dimensional space picture composed of various pseudo points into a plurality of square columns according to the set perception range and resolution.
[0171] The summing unit is used to add the features in the foreground feature set corresponding to each pseudo point whose coordinates are located in the same square column to obtain the foreground bird's-eye view features corresponding to the columns in each direction, and to add the features in the background feature set corresponding to each pseudo point whose coordinates are located in the same square column to obtain the background bird's-eye view features corresponding to the columns in each direction.
[0172] The second combining unit is used to combine the foreground bird's-eye view features corresponding to the columns in each direction into a foreground bird's-eye view feature set, and to combine the background bird's-eye view features corresponding to the columns in each direction into a background bird's-eye view feature set.
[0173] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.
[0174] Another embodiment of the present application provides an electronic device, such as Figure 6As shown, including:
[0175] Memory 601 and processor 602 .
[0176] The memory 601 is used to store programs.
[0177] The processor 602 is used to execute the program stored in the memory 601. When the program is executed, it is specifically used to implement the three-dimensional object detection method provided by any one of the above embodiments.
[0178] Another embodiment of the present application provides a computer storage medium for storing a computer program. When the computer program is executed, it is used to implement the three-dimensional object detection method provided in any one of the above embodiments.
[0179] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0180] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0181] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A three-dimensional target detection method, characterized in that: include: Get the current image to be detected; Calculate a pseudo point cloud feature set of the current image to be detected; wherein the pseudo point cloud feature set includes each pseudo point cloud feature obtained by weighting the image feature elements at multiple depths using the depth weights of each image feature element on the image to be detected; Based on each of the image feature elements, predicting a semantic weight corresponding to each of the pseudo point cloud features; By comparing the weight corresponding to each of the pseudo point cloud features with a preset weight threshold, each of the pseudo point cloud features in the pseudo point cloud feature set is separated to obtain a foreground feature set belonging to a foreground area and a background feature set belonging to a background area; wherein the weight corresponding to the pseudo point cloud features includes the semantic weight and the depth weight; Converting the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively; In the feature channel dimension, the foreground bird's-eye view feature set and the background bird's-eye view feature set are cascaded to obtain a cascaded bird's-eye view feature set of the current image to be detected; Performing convolution processing on the cascaded bird's-eye view feature set of the current image to be detected by using a trained encoder to obtain a fused bird's-eye view feature set; The fused bird's-eye view feature set is used to perform target detection to obtain a current target detection result.
2. The method according to claim 1, characterized in that The convolution process is performed on the cascaded bird's-eye view feature set of the current image to be detected by the trained encoder to obtain a fused bird's-eye view feature set, including: Through the first-order module of the trained bird's-eye view encoder, the cascaded bird's-eye view feature set of the current image to be detected is convolved according to the dimension of the foreground bird's-eye view feature set to obtain the fused bird's-eye view feature set; wherein the first-order module of the bird's-eye view encoder is iteratively trained in advance based on the target detection error.
3. The method according to claim 2, characterized in that The method further comprises: Based on the error between the current target detection result and the current true result, the current loss gradient is detected, and the current loss gradient is transferred to the first-order module of the bird's-eye view encoder through the back propagation algorithm to optimize the parameters of the first-order module of the bird's-eye view encoder.
4. The method according to claim 1, characterized in that: The step of calculating a pseudo point cloud feature set of the current image to be detected includes: For each of the image feature elements of the current image to be detected, predicting a depth weight of the image feature element at a plurality of discretized depth values based on the image feature element; Respectively weighting the depth weight of the image feature element at each discretized depth value and the image feature element to obtain each pseudo point cloud feature corresponding to the image feature element; The pseudo point cloud features corresponding to the image feature elements are combined into a pseudo point cloud feature set of the current image to be detected.
5. The method according to claim 1, characterized in that The method separates each of the pseudo point cloud features in the pseudo point cloud feature set by comparing the weight corresponding to each of the pseudo point cloud features with a preset weight threshold, so as to obtain a foreground feature set belonging to a foreground area and a background feature set belonging to a background area, including: Filtering out from the pseudo point cloud feature set the pseudo point cloud features belonging to the foreground area whose semantic weight is not less than the foreground semantic threshold in the preset weight threshold and whose depth weight is not less than the foreground depth threshold in the preset weight threshold, to obtain the foreground feature set belonging to the foreground area; The pseudo point cloud features in the pseudo point cloud feature set retained after filtering are combined into the background feature set belonging to the background area.
6. The method according to claim 1, characterized in that The converting the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively, comprises: Calculating the coordinates of each pseudo point corresponding to each image feature element according to each of the depth values and the internal and external parameters of the camera that collects the current image to be detected; According to the set perception range and resolution, the three-dimensional space image composed of each of the pseudo points is divided into a plurality of square columns; Adding the features in the foreground feature set corresponding to each of the pseudo points whose coordinates are located in the same square column to obtain the foreground bird's-eye view features corresponding to each of the direction columns, and adding the features in the background feature set corresponding to each of the pseudo points whose coordinates are located in the same square column to obtain the background bird's-eye view features corresponding to each of the direction columns; The foreground bird's-eye view features corresponding to the direction columns are combined into the foreground bird's-eye view feature set, and the background bird's-eye view features corresponding to the direction columns are combined into the background bird's-eye view feature set.
7. A three-dimensional target detection device, characterized in that: include: An image acquisition unit, used to acquire the current image to be detected; A pseudo point cloud computing unit is used to calculate a pseudo point cloud feature set of the current image to be detected; wherein the pseudo point cloud feature set includes each pseudo point cloud feature obtained by weighting the image feature elements on the image to be detected using the depth weights of the image feature elements at multiple depths; A first prediction unit, configured to predict, based on each of the image feature elements, a semantic weight corresponding to each of the pseudo point cloud features; A separation unit, configured to separate each of the pseudo point cloud features in the pseudo point cloud feature set by comparing the weight corresponding to each of the pseudo point cloud features with a preset weight threshold, so as to obtain a foreground feature set belonging to a foreground area and a background feature set belonging to a background area; wherein the weight corresponding to the pseudo point cloud features includes the semantic weight and the depth weight; A conversion unit, used to convert the foreground feature set and the background feature set into a foreground bird's-eye view feature set and a background bird's-eye view feature set, respectively; A cascade operation unit, configured to perform a cascade operation on the foreground bird's-eye view feature set and the background bird's-eye view feature set in a feature channel dimension to obtain a cascaded bird's-eye view feature set of the current image to be detected; A fusion unit, configured to perform convolution processing on the cascaded bird's-eye view feature set of the current image to be detected through a trained encoder to obtain a fused bird's-eye view feature set; The detection unit is used to perform target detection using the fused bird's-eye view feature set to obtain a current target detection result.
8. The device according to claim 7, characterized in that The fusion unit comprises: The fusion subunit is used to perform a convolution operation on the cascaded bird's-eye view feature set of the current image to be detected according to the dimension of the foreground bird's-eye view feature set through the first-order module of the trained bird's-eye view encoder, so as to obtain the fused bird's-eye view feature set; wherein the first-order module of the bird's-eye view encoder is iteratively trained in advance based on the target detection error.
9. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the three-dimensional target detection method as described in any one of claims 1 to 6.
10. A computer storage medium, characterized in that: Used to store a computer program, which, when executed, is used to implement the three-dimensional target detection method as described in any one of claims 1 to 6.
Citation Information
Cited By
Rhizobium pustatae as well as exopolysaccharide, method and application of rhizobium pustatae
CN122465768A