A 3D target detection method and device, electronic equipment and storage medium

Abstract

This invention provides a 3D target detection method, apparatus, electronic device, and storage medium. The method includes: acquiring BEV features of an image at the current moment and pre-stored historical BEV features; fusing the historical BEV features with the BEV features of the image at the current moment to obtain BEV features with historical information at the current moment; and performing 3D target detection based on the BEV features with historical information at the current moment to obtain the 3D target detection result of the image at the current moment. In this invention, online fusion of BEV features is achieved, eliminating the need to store BEV features from multiple frames of images. This avoids the problem of model size increasing with the number of fused frames due to feature stitching, making it more suitable for scenarios with strict model requirements and easier to deploy. Since the historical BEV features are obtained by fusing the BEV features of images from various historical moments within a specific historical period prior to the current moment image frame by frame, the historical time-series information is utilized for 3D target detection, resulting in more accurate 3D detection results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a 3D target detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] 3D perception based on pure camera sensors has attracted a lot of attention from the industry and academia due to its low cost and easy deployment. 3D detection based on Bird's Eye View (BEV) is one of the mainstream methods for pure camera sensors, and a large number of excellent methods such as BEVDet, BEVDepth, BEVStereo, STS, etc. have appeared. However, 3D detection of occluded objects, small target objects, and accurate perception of object movement speed are still extremely challenging.

[0003] In the prior art, in order to cope with the above challenges, the quality of 3D detection based on BEV features is improved by simply splicing multiple frames of BEV features. This method of splicing multiple frames of BEV features has the following disadvantages: 1) multiple frames of BEV features need to be stored; 2) since the splicing is usually performed along the channel dimension, the size and computational complexity of the model will increase with the increase in the number of spliced frames; 3) only a limited number of frames can be used. These disadvantages are very unfriendly to deployment, and also cannot make the most of the timing information. SUMMARY

[0004] In view of the above problems, the embodiments of the present application provide a 3D target detection method, device, electronic equipment and storage medium to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of the embodiments of the present application, a 3D target detection method is disclosed, the method comprising:

[0006] obtaining a BEV feature of a current time image and a pre-stored historical BEV feature, the historical BEV feature being obtained by frame-by-frame fusion of BEV features of each historical time image in a specific historical period before the current time image;

[0007] fusing the historical BEV feature and the BEV feature of the current time image to obtain a BEV feature of the current time with historical information;

[0008] performing 3D target detection based on the BEV feature of the current time with historical information to obtain a 3D target detection result of the current time image.

[0009] Optionally, the method further comprises:

[0010] Save the BEV feature with historical information of the current moment as the historical BEV feature of the next moment image, and realize 3D target detection of the next moment image.

[0011] Optionally, the historical BEV feature is fused with the BEV feature of the current moment image to obtain the BEV feature with historical information of the current moment, including:

[0012] According to the extrinsic information of the image acquisition device and the vehicle movement parameter value, the coordinates of the historical BEV feature are aligned with the coordinates of the BEV feature of the current moment image to obtain the historical BEV feature under the coordinates of the current moment.

[0013] The historical BEV feature under the coordinates of the current moment is fused with the BEV feature of the current moment to obtain the BEV feature with historical information of the current moment.

[0014] Optionally, each moment image includes multiple perspective images of the same moment; the BEV feature of the current moment image is obtained, including:

[0015] The image features and the depth features of the multiple perspective images of the current moment are obtained respectively;

[0016] The image features of the multiple perspective images of the current moment are projected to a 3D space according to the depth features respectively to obtain the frustum features of the multiple perspective images of the current moment respectively;

[0017] The frustum features of the multiple perspective images of the current moment are projected to a BEV perspective to obtain the BEV feature of the current moment.

[0018] Optionally, the image features and the depth features of the multiple perspective images of the current moment are obtained, including:

[0019] The feature extraction is performed on the multiple perspective images of the current moment respectively to obtain the image features of the multiple perspective images of the current moment respectively;

[0020] According to the intrinsic information of the image acquisition device, the depth estimation is performed on the image features of each perspective image of the current moment to obtain the depth features of each perspective image of the current moment.

[0021] Optionally, the 3D target detection method is realized through a 3D target detection model, and a training sample of the 3D target detection model is a sample video composed of sample multi-perspective images of continuous multiple moments, and the sample multi-perspective images of multiple target moments in the sample video carry 3D target detection labels.

[0022] A second aspect of the embodiment of the application discloses a 3D target detection device, and the device includes:

[0023] characteristics of each historical time image in a specific historical period before the current time image are fused frame by frame to obtain the historical BEV characteristics;

[0024] a BEV time sequence information fusion module configured to fuse the historical BEV characteristics and the BEV characteristics of the current time image to obtain BEV characteristics of the current time with historical information;

[0025] a target detection module configured to perform 3D target detection based on the BEV characteristics of the current time with historical information to obtain a 3D target detection result of the current time image.

[0026] In a third aspect, the embodiment of the present application discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the 3D target detection method according to the first aspect of the present application.

[0027] In a fourth aspect, the embodiment of the present application discloses a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to implement the 3D target detection method according to the first aspect of the present application.

[0028] In a fifth aspect, the embodiment of the present application discloses a computer program product, which comprises computer readable code, and the computer readable code is executed to implement the 3D target detection method according to the first aspect of the present application.

[0029] The embodiment of the present application has the following advantages:

[0030] In the embodiment of the present application, based on the online fusion method of BEV characteristics, the BEV characteristics of multiple images do not need to be stored, but the BEV characteristics of the current time image and the historical BEV characteristics are used to implement 3D target detection, and the historical BEV characteristics are obtained by fusing the BEV characteristics of each historical time image in a specific historical period, and have rich historical time sequence information; the historical BEV characteristics are fused with the BEV characteristics of the current time image to obtain the BEV characteristics of the current time with historical information, and then 3D target detection is performed based on the BEV characteristics of the current time with historical information to obtain the 3D target detection result of the current time image.

[0031] Since the BEV features of multiple frames of images do not need to be stored, and the problem that the model increases with the increase of the number of fused frames due to feature splicing is avoided, the method of the embodiment is more friendly to scenarios with strict requirements on the model and is more convenient for deployment. At the same time, since the historical BEV features are obtained by frame-by-frame fusion of the BEV features of images at each historical time in a specific historical period, the current time BEV features with historical information obtained by fusion contain all the historical time sequence information, effectively improving the quality of the BEV features, better depicting occluded objects, small objects and the like, and thus the 3D detection result is more accurate. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0033] Figure 1 is a step flow chart of a 3D target detection method provided by the embodiment of the present application;

[0034] Figure 2 is a general framework schematic diagram of a 3D target detection method provided by the embodiment of the present application;

[0035] Figure 3 is a structural schematic diagram of a 3D target detection device provided by the embodiment of the present application;

[0036] Figure 4 is a schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0038] In the related art, the time sequence information fusion scheme of the BEV feature has the shortcomings of needing to store historical features, the model size and the amount of calculation increasing with the increase of the time sequence information, only being able to utilize a limited number of frames, not conforming to the real application scene, and the quality of the BEV feature being poor, and the model being inconvenient to deploy. Therefore, in order to overcome the limitations of the above problems, the applicant proposes the following technical concept: by frame-by-frame coordinate alignment and information fusion of historical information and current time information, online fusion of the BEV feature is realized, and 3D target detection is performed based on the BEV feature obtained by online fusion, thereby improving the quality of the BEV feature by utilizing historical time sequence information, and efficiently performing 3D target detection.

[0039] Based on the above technical concept, the embodiment of the present application provides a 3D target detection method, as shown in Figure 1 Figure 1 A step flow chart of a 3D target detection method provided by the embodiment of the present application, the method specifically includes steps S101 to S103:

[0040] Step S101: obtaining the BEV feature of the current time image and the pre-stored historical BEV feature, the historical BEV feature being obtained by frame-by-frame fusion of the BEV features of each historical time image in a specific historical period before the current time image.

[0041] In the embodiment, the BEV feature refers to the feature in the bird's eye view, that is, the BEV feature is a feature in a global view, which is a cross-view feature representation. The BEV feature well integrates the information of multiple images in different views, can effectively represent the information around the reference coordinate origin, and is conducive to 3D target detection under pure vision. The historical BEV feature is the BEV feature saved for 3D target detection of the previous time image. The historical BEV feature can be obtained by reading from the memory. The historical BEV feature is not the BEV feature of the previous time. The historical BEV feature is obtained by frame-by-frame fusion of the BEV features of each historical time image in a specific historical period before the current time image. Therefore, the historical BEV feature contains all historical time sequence information in a specific historical period before the current time. The specific historical period can refer to all time periods before the current time, or a period of time before the current time, for example, one hour before the current time. In the subsequent steps, by fusing the historical BEV feature with the BEV feature of the current time, the transmission and extreme utilization of the historical time sequence information can be realized.

[0042] ​The BEV feature of the current time image is obtained by processing the current time image. In this embodiment, the image at each time point within the current time point includes multiple perspective images at the same time point, and the objects in each perspective image are not completely the same. For example, for automatic driving, the multiple perspective images can be: a front perspective image, a left front perspective image, a right front perspective image, a rear perspective image, a left rear perspective image, and a right rear perspective image, etc. of the vehicle, wherein the device for image acquisition can be a vehicle-mounted camera. According to the different arrangement of the vehicle-mounted camera on different vehicles, the number of perspectives will be different, and accordingly, the number of multiple perspective images included in the image at each time point will be different. In addition, the image at the current time point can be a video frame image at the current time point extracted from a video, or an image at the current time point directly acquired by the image acquisition device.

[0043] In an optional embodiment, the BEV feature of the current time image is obtained, specifically including steps A1 to A3:

[0044] Step A1: obtaining the image feature and the depth feature of each of the multiple perspective images at the current time point.

[0045] In this embodiment, each perspective image has a corresponding image feature and a corresponding depth feature. For example, if there are 6 perspective images, there are 6 image features and 6 depth features, and each image feature corresponds to a depth feature. The image feature of each perspective image can represent the basic information of the perspective image, such as color information, texture information, shape information, etc. The depth feature can represent the depth distribution of each pixel in the image feature, and the actual distance of each object in each multi-perspective image to the image acquisition device can be calculated according to the depth feature.

[0046] Specifically, the obtaining of the image feature and the depth feature of each of the multiple perspective images at the current time point includes: performing feature extraction on each of the multiple perspective images at the current time point to obtain the image feature of each of the multiple perspective images at the current time point; and performing depth estimation on the image feature of each of the multiple perspective images at the current time point according to the intrinsic information of the image acquisition device to obtain the depth feature of each of the multiple perspective images at the current time point.

[0047] The intrinsic information of the image acquisition device refers to parameters related to the characteristics of the image acquisition device, and includes parameters such as focal length, pixel size, center of the image, and the number of pixels between the origin.

[0048] In practical applications, the image feature extraction network is used to complete the feature extraction of multiple perspective images at the current time. The image feature extraction network can be a neural network that has been pre-trained to have image feature extraction function, for example, using ResNet, ConvNeXt, etc. The image feature extraction network can be independent of the 3D target detection model in the embodiment; the image feature extraction network can also be integrated into the 3D target detection model in the embodiment as a subnetwork of the 3D target detection model.

[0049] The depth estimation network is also used to obtain the depth feature, wherein the depth estimation network can be a network composed of one or more convolution and fully connected operators. Specifically, the depth estimation network estimates the depth of the image feature of each perspective image at the current time according to the intrinsic information of the image acquisition device to obtain the depth feature of each perspective image at the current time. In addition, the depth estimation network can be independent of the 3D target detection model in the embodiment; the depth estimation network can also be integrated into the 3D target detection model in the embodiment as a subnetwork of the 3D target detection model.

[0050] Step A2: Project the image feature of each of the multiple perspective images at the current time to the 3D space according to the respective depth features to obtain the cone feature of each of the multiple perspective images at the current time.

[0051] In the embodiment, the image feature is a 2D feature. In order to realize the 3D target detection in the subsequent steps, it is necessary to project the image feature to the 3D space to obtain the feature in the 3D space, which is the cone space of the image acquisition device. Therefore, the feature in the 3D space is the cone feature. The cone refers to the area that can be seen by the image acquisition device, which is a cone from near to far, so it is called cone.

[0052] Since the depth feature is obtained by depth estimation of the image feature, the depth feature represents the depth distribution of each pixel in the image feature. Therefore, the image feature can be projected to the corresponding 3D space according to the depth feature. Specifically, the image feature and the depth feature of each perspective image at the current time are outer multiplied to obtain the cone feature corresponding to the perspective image.

[0053] Step A3: Project the cone feature of each of the multiple perspective images at the current time to the BEV perspective to obtain the BEV feature at the current time.

[0054] In this embodiment, the BEV feature of the current time obtained by projection is a same expression feature across views, and the BEV feature of the current time contains the frustum features of all views. For example, in an autonomous driving scenario, the frustum features of the front view image, the left front view image, the right front view image, the back view image, the left back view image, and the right back view image of the vehicle are projected into the same BEV view to obtain the BEV feature of the vehicle at the current time. The BEV feature is a HxWxC feature, where HxW is the spatial size in the BEV view, and C represents the feature dimension. Through the BEV feature of the vehicle at the current time, all the environmental information around the vehicle in the bird's eye view can be known.

[0055] Specifically, the frustum features of the multiple view images at the current time are projected into the BEV view, which can be multiplying the frustum features of the current view image by a transformation matrix to obtain the BEV feature.

[0056] Step S102: fuse the historical BEV feature and the BEV feature of the current time image to obtain a BEV feature of the current time with historical information.

[0057] In actual applications, a light-weight BEV feature fusion network is used to fuse the historical BEV feature and the BEV feature of the current time image. Specifically, the light-weight fusion network is composed of a Conv (i.e., vector convolution operation) and a 1x1 convolution.

[0058] In the BEV feature of the current time with historical information, there is historical time sequence information in a specific historical period before all times before the current time, and there is also real-time information at the current time. Therefore, in the subsequent step, based on the BEV feature of the current time with historical information, 3D target detection can be performed to accurately perceive occluded objects and small target objects.

[0059] In an optional embodiment, considering that the coordinate systems of the BEV features at different times are different, the BEV features in different coordinate systems cannot be directly fused. Therefore, in step S102, fusing the historical BEV feature and the BEV feature of the current time image to obtain a BEV feature of the current time with historical information includes steps B1 and B2:

[0060] Step B1: align the coordinates of the historical BEV feature with the coordinates of the BEV feature of the current time image according to the extrinsic information of the image acquisition device and the vehicle movement parameter value to obtain a historical BEV feature in the coordinates of the current time;

[0061] Step B2: fuse the historical BEV feature at the current time coordinate with the BEV feature at the current time to obtain the BEV feature at the current time with historical information.

[0062] In this embodiment, the coordinates of the origin of the coordinate system of each time image in the 3D space change over time, for example, the autonomous vehicle is constantly moving, and the BEV feature at each time is relative to the center point of the vehicle. Therefore, the origin of the coordinate system of the BEV feature at the current time is different from the origin of the coordinate system of the historical BEV feature. Further, the coordinates of the historical BEV feature need to be aligned with the coordinates of the BEV feature at the current time image to obtain the historical BEV feature at the current time coordinate.

[0063] In actual application, the extrinsic information of the image acquisition device refers to the parameters in the world coordinate system, such as the position and rotation direction of the camera. The vehicle movement parameter value refers to the driving speed, driving direction and turning angle of the vehicle. Therefore, according to the extrinsic information of the image acquisition device and the vehicle movement parameter value, the coordinate transformation relationship between the coordinate system of the historical BEV feature and the coordinate system of the BEV feature at the current time is determined, and then the coordinates of the historical BEV feature are aligned to the coordinates at the current time based on the coordinate transformation relationship. After the coordinate alignment is completed, the historical BEV feature and the BEV feature at the current time in the same coordinate system are fused to obtain the BEV feature at the current time with historical information.

[0064] In this embodiment, by aligning the coordinates of the historical BEV feature and the coordinates of the BEV feature at the current time, and fusing the historical BEV feature at the current time coordinate with the BEV feature at the current time, online fusion of the BEV feature is realized. Further, the BEV feature at the current time with historical information obtained by fusion is more consistent with the real application scenario, realizes the extreme use of historical time sequence information, effectively improves the quality of the BEV feature, better describes the occluded objects and small objects, and improves the 3D detection accuracy of the occluded objects and small objects in the subsequent steps. Compared with the past, each time needs to process multiple images. In this embodiment, the online fusion of the BEV feature avoids the problem that the model increases with the increase of the fusion frame number due to feature splicing, is more friendly to the scene with strict requirements for the model, is more convenient for deployment, and improves the processing efficiency of 3D target detection.

[0065] Step S103: based on the BEV feature at the current time with historical information, performing 3D target detection to obtain the 3D target detection result of the image at the current time.

[0066] In the embodiment, the BEV feature with historical information at the current time has rich historical time sequence information and real-time information, and the 3D target detection based on the BEV feature with historical information at the current time has good detection effect. Therefore, based on the online fusion manner of the BEV feature in the embodiment, the performance of the pure vision camera 3D target detection is improved, and compared with the 3D target detection method based on the depth camera, the scheme of the embodiment has high-precision 3D target detection effect while meeting the low cost.

[0067] In actual application, the BEV feature with historical information at the current time is processed by using a 3D detection head (i.e., 3D Detection Head) to obtain a 3D target detection result of the current time image. The 3D detection head can be composed of one or more convolution kernels. In the detection result, the position, size and category of the target object can be marked by using a marking box with different sizes and colors. For example, in the vehicle automatic driving scene, motor vehicles, pedestrians, cyclists or other obstacles around the vehicle are detected, and then the driving path of the vehicle is planned and the driving obstacle avoidance is realized based on the 3D detection result.

[0068] In an optional embodiment, in order to realize the transmission of the time sequence BEV feature, after obtaining the BEV feature with historical information at the current time in step S102, the method further includes: saving the BEV feature with historical information at the current time as a historical BEV feature of the next time image, to realize 3D target detection of the next time image.

[0069] In actual application, on the premise of adopting the above-mentioned strategy of "saving the BEV feature with historical information at the current time as a historical BEV feature of the next time image", when performing 3D target detection of each time image, only the BEV feature of the time image needs to be obtained, and the pre-saved historical BEV feature needs to be read. Similarly, after obtaining the BEV feature with historical information at the current time, it can be saved for use in 3D target detection of the next time image. Because 3D target detection of each time image only needs to process the current time image, the 3D target detection efficiency of the current time image is improved.

[0070] It should be noted that the historical BEV feature of the current moment is not the BEV feature of the previous moment, the historical BEV feature is obtained by frame-by-frame fusion of the BEV features of each historical moment image in a specific historical period before the current moment image, and has rich historical time sequence information; and the historical BEV feature is not a fixed BEV feature, but is updated with the BEV feature of the current moment image, thereby realizing the transmission of historical time sequence information, maximizing the use of historical time sequence information, effectively improving the quality of the BEV feature, better depicting occluded objects, small objects and the like, and improving the 3D detection accuracy of occluded objects and small objects in subsequent steps, to ensure the accuracy of the 3D target detection result. The limitation that only the BEV feature of a limited moment image can be used for 3D target detection in the prior art is overcome, and the historical time sequence information is maximally utilized.

[0071] In an optional embodiment, the 3D target detection method is realized by a 3D target detection model, and training samples of the 3D target detection model are a sample video composed of sample multi-view images of continuous multiple moments, and the sample multi-view images of multiple target moments in the sample video carry 3D target detection labels.

[0072] In this embodiment, the 3D target detection model includes an image feature extraction network, a depth estimation network, a view cone feature generation network, a BEV feature generation network, a BEV feature fusion network and a 3D detection head. Because each moment detection only needs to extract the BEV feature of the current moment, and the online fusion of the BEV feature is realized based on a light-weight BEV feature fusion network composed of a Conv and a 1x1 convolution, the 3D target detection model is a more light-weight model compared with the prior art such as BEVDet4D, SOLOFusion and BEVFormerV2, and can more efficiently perform 3D target detection. Therefore, the 3D target detection model in this embodiment is more convenient to deploy in actual application scenarios, and in particular for the automatic driving scenario with strict model requirements, the 3D target detection model can be well deployed on a vehicle.

[0073] The sample video carrying the 3D target detection label is input into the 3D target detection model for training, and each round of learning process includes two stages: learning on multi-view images of continuous multiple time instants without carrying the 3D target detection label, and 3D target detection on multi-view images of a target time instant carrying the 3D target detection label. Specifically, the 3D target detection model first learns on multi-view images of continuous multiple time instants without carrying the 3D target detection label, fuses and updates the historical BEV feature frame by frame; when the 3D target detection model learns on multi-view images of a target time instant carrying the 3D target detection label, 3D target detection is performed in combination with the current historical BEV feature and the BEV feature of the current time instant, to obtain the 3D target detection result of the target time instant, and the 3D target detection result is compared with the real result corresponding to the 3D target detection label to obtain a loss function of the prediction result, the parameters of the 3D target detection model are updated based on the loss function, and whether the training end condition is met is determined; if the training end condition is not met, the next round of learning is started, and if the training end condition is met, it means that the 3D target detection model has completed learning, at which time the training is ended, the model parameters obtained at this time are taken as the final parameters of the 3D target detection model, and the trained 3D target detection model is obtained. The training end condition can be that the loss function value reaches a preset threshold, or the loss function of continuous multiple rounds of learning does not decrease.

[0074] The 3D target detection model trained through the above training process has a 3D target detection function. In actual application, the multi-view images of the current time instant collected by the image collection devices of different perspectives are directly input into the trained 3D target detection model, the 3D target detection model processes the input multi-view images of the current time instant through feature extraction and depth estimation to obtain the image features and depth features of each of the multi-view images of the current time instant, then projects the image features of each of the multi-view images of the current time instant to the 3D space according to the respective depth features to obtain the respective frustum features of the multi-view images of the current time instant, and projects the respective frustum features of the multi-view images of the current time instant to the BEV perspective to obtain the BEV feature of the current time instant; then the coordinates of the historical BEV feature are aligned with the coordinates of the BEV feature of the current time instant, the historical BEV feature under the coordinates of the current time instant is fused with the BEV feature of the current time instant to obtain the BEV feature of the current time instant with historical information, and 3D target detection is performed based on the BEV feature of the current time instant with historical information to obtain the 3D target detection result.

[0075] It should be noted that the manner of implementing online fusion of BEV features in steps S101 to S102 in the above embodiment can also be applied to other application scenarios based on BEV features, for example, semantic segmentation can also be performed based on the BEV features with historical information obtained through online fusion, that is, perception tasks such as map segmentation and lane line segmentation. For example, in the process of automatic driving of a vehicle, the BEV features with historical information at the current time are obtained according to the methods in steps S102 and S102, and then semantic segmentation is performed based on the BEV features with historical information at the current time, to determine the driving environment of the vehicle, such as lane lines and driving areas.

[0076] In the embodiment of the application, based on a method of online fusion of BEV features, the BEV features of multiple frames of images do not need to be stored, but the BEV features of the current time image and the historical BEV features are used to realize 3D target detection, and the historical BEV features are obtained by frame-by-frame fusion of the BEV features of each historical time image in a specific historical period before the current time, and have rich historical time information. By fusing the historical BEV features with the BEV features of the current time image, the BEV features with historical information at the current time are obtained, and then 3D target detection is performed based on the BEV features with historical information at the current time, to obtain the 3D target detection result of the current time image.

[0077] Since the BEV features of multiple frames of images do not need to be stored, and the problem of increasing model size due to feature splicing as the number of fused frames increases is avoided, the method of the embodiment is more friendly to scenarios with strict requirements on the model, and is more convenient for deployment. Since the historical BEV features are obtained by frame-by-frame fusion of the BEV features of each historical time image in a specific historical period before the current time, the BEV features with historical information at the current time obtained through fusion contain all the historical time information, effectively improving the quality of the BEV features, and better describing occluded objects and small objects, so that the obtained 3D detection result is more accurate.

[0078] Figure 2 Taking 3D target detection of the current time image and the next time image as an example, the overall framework of the 3D target detection method in the embodiment is shown.

[0079] For 3D target detection at the current moment, first, the coordinates of the historical BEV feature (i.e. the BEV feature used for 3D target detection at the previous moment) are aligned with the coordinates of the BEV feature of the current moment image, and then the historical BEV feature under the coordinates of the current moment is obtained; then the historical BEV feature under the coordinates of the current moment is fused with the BEV feature of the current moment image to obtain the BEV feature of the current moment with historical information, and the BEV feature of the current moment with historical information is ensured; finally, based on the BEV feature of the current moment with historical information, 3D target detection is performed to obtain the 3D target detection result of the current moment image.

[0080] When 3D target detection is performed on the next moment image, the BEV feature of the current moment with historical information is used as the historical BEV feature of the next moment, the coordinates of the historical BEV feature of the next moment are aligned with the coordinates of the BEV feature of the next moment image, the historical BEV feature under the coordinates of the next moment is obtained, and the historical BEV feature under the coordinates of the next moment is fused with the BEV feature of the next moment image to obtain the BEV feature of the next moment with historical information, so as to realize 3D target detection of the next moment. According to the above method, the historical BEV feature is updated frame by frame according to the BEV features of different moments, the transmission of time sequence information, online fusion of BEV features and efficient 3D target detection are realized.

[0081] The embodiment of the application also provides a 3D target detection device, as shown in Figure 3 , Figure 3 A structure diagram of a 3D target detection device provided by the embodiment of the application, the device comprises:

[0082] The feature acquisition module 31 is configured to acquire the BEV feature of the current moment image and the historical BEV feature stored in advance, and the historical BEV feature is obtained by frame-by-frame fusion of the BEV features of each historical moment image in a specific historical period before the current moment image;

[0083] The BEV time sequence information fusion module 32 is configured to fuse the historical BEV feature with the BEV feature of the current moment image to obtain the BEV feature of the current moment with historical information.

[0084] The target detection module 33 is configured to perform 3D target detection based on the BEV feature of the current moment with historical information to obtain the 3D target detection result of the current moment image.

[0085] In an optional embodiment, the device further comprises:

[0086] The feature guarantee module is configured to save the BEV feature with historical information at the current time as historical BEV feature of the image at the next time, so as to realize 3D target detection of the image at the next time.

[0087] In an alternative embodiment, the BEV time sequence information fusion module comprises:

[0088] The coordinate alignment module is configured to align the coordinates of the historical BEV feature with the coordinates of the BEV feature of the image at the current time according to the extrinsic information of the image acquisition device and the vehicle movement parameter value, so as to obtain the historical BEV feature at the coordinates of the current time.

[0089] The feature fusion module is configured to fuse the historical BEV feature at the coordinates of the current time with the BEV feature at the current time, so as to obtain the BEV feature with historical information at the current time.

[0090] In an alternative embodiment, the image at each time comprises multiple perspective images at the same time; and the feature acquisition module comprises:

[0091] The feature extraction module is configured to acquire the image feature and the depth feature of each of the multiple perspective images at the current time.

[0092] The view cone feature generation module is configured to project the image feature of each of the multiple perspective images at the current time to a 3D space according to the depth feature of each of the multiple perspective images at the current time, so as to obtain the view cone feature of each of the multiple perspective images at the current time.

[0093] The view cone feature projection BEV module is configured to project the view cone feature of each of the multiple perspective images at the current time to a BEV perspective, so as to obtain the BEV feature at the current time.

[0094] In an alternative embodiment, the feature extraction module comprises:

[0095] The image feature module is configured to perform feature extraction on each of the multiple perspective images at the current time, so as to obtain the image feature of each of the multiple perspective images at the current time.

[0096] The depth estimation module is configured to perform depth estimation on the image feature of each of the perspective images at the current time according to the intrinsic information of the image acquisition device, so as to obtain the depth feature of each of the perspective images at the current time.

[0097] The embodiment of the application further provides an electronic device, which refers to Figure 4 , Figure 4 is a schematic diagram of the electronic device provided by the embodiment of the application. As shown in Figure 4As shown, the electronic device 100 comprises a memory 110 and a processor 120, the memory 110 and the processor 120 are connected through a bus in communication, the memory 110 stores a computer program, the computer program can run on the processor 120, and then the 3D target detection method disclosed in the embodiment of the application is realized.

[0098] The embodiment of the application further provides a computer readable storage medium, which stores a computer program / instruction, and the computer program / instruction is executed by a processor to realize the 3D target detection method disclosed in the embodiment of the application.

[0099] The embodiment of the application further provides a computer program product, which comprises computer readable code, and the computer readable code is executed to realize the 3D target detection method disclosed in the embodiment of the application.

[0100] Each of the embodiments in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same and similar parts between each embodiment are referred to each other.

[0101] The embodiment of the application is described with reference to the flowcharts and / or block diagrams of the method, device, electronic device and storage medium according to the embodiment of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal equipment to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal equipment produce a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for implementing the functions specified in the flowcharts and / or block diagrams.

[0102] These computer program instructions can also be stored in a computer readable storage medium, which can guide the computer or other programmable data processing terminal equipment to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product comprising instruction means, which realizes the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 The means for implementing the functions specified in the flowcharts and / or block diagrams.

[0103] These computer program instructions can also be loaded into the computer or other programmable data processing terminal equipment, so that a series of operation steps are performed on the computer or other programmable terminal equipment to produce a computer implemented process, and the instructions executed on the computer or other programmable terminal equipment provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1one or more processes and / or blocks Figure 1 steps of the functions specified in the one or more blocks.

[0104] While preferred embodiments of the application have been described, those skilled in the art will appreciate that other modifications and variations are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims, the application can be practiced otherwise than as specifically described. For example, while the application has been described in the context of a fully functioning data processing system, those skilled in the art will appreciate that the application includes at least one program and that the application can be implemented on other types of computers other than the ones described above.

[0105] Finally, it should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. It is also possible in the present disclosure that

[0106] The above provides a 3D target detection method, device, electronic equipment and storage medium provided by the application in detail, the principle and implementation mode of the application are described in this paper by applying specific examples, the above example is only used to help understand the method and core idea of the application; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation mode and application range will be changed; in view of the above, the content of the specification should not be understood as the limitation of the application.

Claims

1. A 3D object detection method, characterized in that, The method comprises: obtaining BEV features of a current time image and pre-stored historical BEV features, the historical BEV features being obtained by frame-by-frame fusion of BEV features of each historical time image within a specific historical period before the current time image; fusing the historical BEV features with the BEV features of the current time image to obtain BEV features of the current time with historical information; based on the BEV features of the current time with historical information, performing 3D target detection to obtain a 3D target detection result of the current time image; wherein the fusion of the historical BEV features with the BEV features of the current time image to obtain BEV features of the current time with historical information comprises: aligning coordinates of the historical BEV features with coordinates of the BEV features of the current time image according to extrinsic information of an image acquisition device and a vehicle movement parameter value to obtain historical BEV features under coordinates of the current time; fusing the historical BEV features under coordinates of the current time with the BEV features of the current time to obtain BEV features of the current time with historical information; the historical information comprises historical time sequence information and real-time information.

2. The method of claim 1, wherein, The method further comprises: saving the BEV features of the current time with historical information as historical BEV features of a next time image to realize 3D target detection of the next time image.

3. The method of claim 1, wherein, Each time image comprises multiple perspective images of the same time; obtaining BEV features of a current time image comprises: obtaining image features and depth features of the multiple perspective images of the current time respectively; projecting the image features of the multiple perspective images of the current time to a 3D space according to the depth features to obtain cone features of the multiple perspective images of the current time respectively; projecting the cone features of the multiple perspective images of the current time to a BEV perspective to obtain BEV features of the current time.

4. The method of claim 3, wherein, The obtaining of the image features and the depth features of the multiple perspective images of the current time comprises: performing feature extraction on the multiple perspective images of the current time respectively to obtain image features of the multiple perspective images of the current time respectively; performing depth estimation on the image features of each perspective image of the current time according to intrinsic information of an image acquisition device to obtain depth features of each perspective image of the current time.

5. The method according to any of claims 1 to 4, characterized in that, The 3D target detection method is realized by a 3D target detection model, and training samples of the 3D target detection model are sample videos composed of sample multi-perspective images of continuous multiple time points, and the sample multi-perspective images of multiple target time points in the sample videos carry 3D target detection labels.

6. A 3D object detection apparatus characterized by comprising: The device comprises: a feature acquisition module configured to obtain BEV features of a current time image and pre-stored historical BEV features, the historical BEV features being obtained by frame-by-frame fusion of BEV features of each historical time image within a specific historical period before the current time image; The BEV time sequence information fusion module is configured to fuse the historical BEV feature and a BEV feature of the current time image to obtain a BEV feature of the current time with historical information. The target detection module is configured to perform 3D target detection based on the BEV feature of the current time with historical information to obtain a 3D target detection result of the current time image. The BEV time sequence information fusion module is further configured to align coordinates of the historical BEV feature with coordinates of the BEV feature of the current time image according to external parameter information of the image acquisition device and a vehicle movement parameter value to obtain the historical BEV feature under coordinates of the current time. The historical BEV feature under the coordinates of the current time is fused with the BEV feature of the current time to obtain the BEV feature of the current time with historical information. The historical information includes historical time sequence information and real-time information.

7. An electronic device, comprising: The computer program / instruction is executed by the processor to implement the 3D target detection method according to any one of claims 1-5.

8. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, The computer program / instruction is executed by the processor to implement the 3D target detection method according to any one of claims 1-5.

9. A computer program product comprising computer readable code, characterized in that, The computer program / instruction is executed by the processor to implement the 3D target detection method according to any one of claims 1-5.

Images