3D target detection method, device, equipment and medium

Abstract

The embodiments of the present invention provide a 3D target detection method, apparatus, device and medium, the method comprising: obtaining image features at the current moment, initial instance features at the current moment and output instance features at the previous moment; interactively processing the image features at the current moment, the initial instance features at the current moment and the output instance features at the previous moment to obtain the output instance features at the current moment; performing 3D target detection based on the output instance features at the current moment to obtain the 3D target detection result at the current moment. In the embodiments of the present invention, there is no need to save a large image feature map, but rather to perform a time-series interaction based on instance features. The network only needs to save the instance features output at each moment, thereby reducing space consumption and facilitating chip deployment. Furthermore, by transmitting instance features moment by moment, a longer time sequence is achieved, effectively solving the target object occlusion problem and improving the accuracy of 3D target detection.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a 3D target detection method, device, equipment and medium. Background Art

[0002] Online video processing tasks (such as autonomous driving) require the continuous processing of images at different times to obtain a sense of the surrounding world. Single-frame object detection struggles to achieve satisfactory 3D detection results due to object occlusion and inaccurate depth estimation. Consequently, a growing number of methods are addressing the 3D object detection problem from a temporal perspective.

[0003] Existing technologies require storing large image feature maps, such as BEV features or image features, when processing multiple frames. These large feature maps pose storage challenges for image processing devices and increase the computational burden, which in turn reduces the real-time performance of the model. Furthermore, moving objects may be severely occluded during motion, so addressing this occlusion issue is also a pressing technical challenge. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a 3D object detection method, apparatus, device, and medium to overcome the above problems or at least partially solve the above problems.

[0005] According to a first aspect of an embodiment of the present invention, a 3D object detection method is disclosed, the method comprising:

[0006] Obtaining image features at a current moment, initial instance features at a current moment, and output instance features at a previous moment, wherein the output instance features at the previous moment are used to detect a 3D object at the previous moment;

[0007] Interactively processing the image feature at the current moment, the initial instance feature at the current moment, and the output instance feature at the previous moment to obtain the output instance feature at the current moment;

[0008] 3D object detection is performed based on the output instance features at the current moment to obtain the 3D object detection result at the current moment.

[0009] Optionally, interactively processing the image feature at the current moment, the initial instance feature at the current moment, and the output instance feature at the previous moment to obtain the output instance feature at the current moment includes:

[0010] Interactively processing the initial instance feature at the current moment and the output instance feature at the previous moment to obtain an interactive instance feature containing 3D spatial position information;

[0011] The interactive instance feature containing the 3D spatial position information and the image feature at the current moment are interactively processed to obtain the output instance feature at the current moment.

[0012] Optionally, interactively processing the initial instance feature at the current moment and the output instance feature at the previous moment to obtain the interactive instance feature containing 3D spatial position information includes:

[0013] Fusing the initial instance feature at the current moment with the output instance feature at the previous moment to obtain a fused instance feature;

[0014] Performing a first position encoding on the fused instance feature to obtain a fused instance feature containing 3D spatial position information, and performing a second position encoding on the initial instance feature at the current moment to obtain an initial instance feature containing 3D spatial position information at the current moment;

[0015] The fused instance feature containing the 3D spatial position information and the initial instance feature containing the 3D spatial position information at the current moment are interactively processed to obtain an interactive instance feature containing the 3D spatial position information.

[0016] Optionally, the method further includes:

[0017] Initialize the initial instance features at the current moment, as well as the initial 3D spatial position information at the current moment;

[0018] Performing a second position encoding on the initial instance feature at the current moment to obtain the initial instance feature containing 3D spatial position information at the current moment, including:

[0019] According to the initial 3D spatial position information at the current moment, a second position encoding is performed on the initial instance feature at the current moment to obtain the initial instance feature containing the 3D spatial position information at the current moment.

[0020] Optionally, the method further includes:

[0021] Obtaining 3D spatial position information at a previous moment corresponding to the output instance feature at the previous moment, wherein the 3D spatial position information at the previous moment represents the spatial position of the 3D target detected at the previous moment;

[0022] fusing the initial 3D spatial position information at the current moment with the 3D spatial position information at the previous moment to obtain fused 3D spatial position information;

[0023] Performing a first position encoding on the fused instance feature to obtain a fused instance feature containing 3D spatial position information includes:

[0024] According to the fused 3D spatial position information, a first position encoding is performed on the fused instance feature to obtain a fused instance feature containing the 3D spatial position information.

[0025] Optionally, interactively processing the interactive instance feature containing the 3D spatial position information and the image feature at the current moment to obtain the output instance feature at the current moment includes:

[0026] Performing a third position encoding on the image feature at the current moment to obtain an image feature containing 3D spatial position information;

[0027] The interactive instance feature containing the 3D spatial position information and the image feature containing the 3D spatial position information are interactively processed to obtain the output instance feature at the current moment.

[0028] Optionally, before fusing the initial 3D spatial position information at the current moment with the 3D spatial position information of the 3D object detected at the previous moment to obtain the fused 3D spatial position information, the method further includes:

[0029] Perform spatial transformation on the 3D spatial position information of the 3D target detected at the previous moment to obtain the 3D spatial position information of the previous moment in the current moment coordinate system;

[0030] Fusing the initial 3D spatial position information at the current moment with the 3D spatial position information at the previous moment to obtain fused 3D spatial position information, including:

[0031] The initial 3D spatial position information at the current moment and the 3D spatial position information at the previous moment in the current moment coordinate system are fused to obtain fused 3D spatial position information.

[0032] Optionally, the method further includes:

[0033] The output instance features of the current moment and the corresponding 3D spatial position information of the current moment are saved for combining with the image of the next moment to realize 3D target detection at the next moment, wherein the 3D spatial position information of the current moment represents the spatial position of the 3D target detected at the current moment.

[0034] Optionally, the image at each moment including the current moment is a multi-view image, and the multi-view image is obtained by stitching images of multiple perspectives collected at the same moment;

[0035] Get the image features at the current moment, including:

[0036] Feature extraction is performed on the image at the current moment to obtain image features at the current moment.

[0037] According to a second aspect of the embodiments of the present invention, a 3D object detection device is disclosed, comprising:

[0038] A feature acquisition module is used to acquire the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment, wherein the output instance features at the previous moment are used to detect the 3D object at the previous moment;

[0039] a feature interaction module, configured to interactively process the image feature at the current moment, the initial instance feature at the current moment, and the output instance feature at the previous moment to obtain the output instance feature at the current moment;

[0040] The target detection module is used to perform 3D target detection based on the output instance features at the current moment to obtain the 3D target detection result at the current moment.

[0041] According to a third aspect of an embodiment of the present invention, an electronic device is disclosed, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When executed by the processor, the 3D target detection method as described in the first aspect of the embodiment of the present invention is implemented.

[0042] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is disclosed, on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the 3D target detection method described in the first aspect of the embodiment of the present invention is implemented.

[0043] The embodiments of the present invention include the following advantages:

[0044] In the embodiment of the present invention, there is no need to save a large image feature map, but 3D target detection is performed by using the transmission of instance features. First, the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment are obtained. Then, the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment are interactively processed to obtain the output instance features at the current moment. Finally, 3D target detection is performed based on the output instance features at the current moment to obtain the 3D target detection result at the current moment. Since the time-series interaction is based on instance features, the network only needs to save the instance features output at each moment, thereby reducing space consumption and making it easier to deploy chips. Moreover, by transmitting instance features moment by moment, the effect of longer time-series information interaction is achieved, thereby effectively solving the target occlusion problem and improving the accuracy of 3D target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 This is a flowchart of a 3D object detection method provided by an embodiment of the present invention;

[0047] Figure 2 This is a schematic diagram of the overall framework of a 3D object detection method algorithm provided by an embodiment of the present invention;

[0048] Figure 3 1 is a schematic diagram of an interaction process among an image feature at a current moment, an initial instance feature at a current moment, and an output instance feature at a previous moment, provided by an embodiment of the present invention;

[0049] Figure 4 It is a structural diagram of a 3D target detection device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0051] The embodiment of the present invention provides a 3D target detection method, such as Figure 1 As shown, Figure 1 A flowchart of a 3D object detection method provided by an embodiment of the present invention includes steps S101 to S103:

[0052] Step S101: Acquire the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment, wherein the output instance features at the previous moment are used to detect the 3D target at the previous moment.

[0053] In step S101, image features are features that can represent image information in an image. The image features at the current moment are image features corresponding to the image at the current moment (i.e., the image for which 3D object detection is to be performed). Specifically, obtaining the image features at the current moment includes performing feature extraction on the image at the current moment to obtain the image features at the current moment.

[0054] In practical applications, the image feature extraction at the current moment is completed through the image feature extraction network. The image feature extraction network can be a pre-trained neural network with image feature extraction function (for example, Backbone network) independent of the 3D target detection network used in this embodiment; the image feature extraction network can also be the 3D target detection network integrated in this embodiment as a subnetwork of the 3D target detection network.

[0055] In step S101, instance features are features that characterize object information in an image. The initial instance features at the current moment are the initialized features corresponding to the image at the current moment. These initialized instance features are a set of instance features obtained by randomly initializing the image at the current moment. The output instance features at the previous moment are the instance features corresponding to the image at the previous moment. The output instance features at the previous moment are read from memory.

[0056] It should be noted that the image at the current moment may be a video frame image extracted from an online video. The image at each moment, including the current moment, is a multi-perspective image, which is obtained by splicing images from multiple perspectives captured at the same moment. For example, for autonomous driving, the multiple perspectives may be: the vehicle's front perspective, left front perspective, right front perspective, rear perspective, left rear perspective, and right rear perspective, etc. The image at each moment includes the above-mentioned images captured simultaneously from multiple perspectives. Among them, the component that performs image acquisition may be a vehicle-mounted camera. Depending on the different arrangements of the vehicle-mounted cameras on different vehicles, the number of perspectives will vary, and accordingly, the number of sub-images included in the image at each moment will vary.

[0057] Step S102: interactively processing the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment to obtain the output instance features at the current moment.

[0058] In this embodiment, a feature interaction network is used to interactively process the current image features, the current initial instance features, and the previous output instance features. The feature interaction network refers to a network with feature interaction capabilities, and a Transformer network can generally be used for feature interaction. The feature interaction network can be independent of the 3D object detection network in this embodiment, or it can be integrated into the 3D object detection network in this embodiment as a subnetwork of the 3D object detection network.

[0059] During interactive processing, the initialized instance features at the current moment can learn the 3D spatial position information of the instance features corresponding to the output instance features at the previous moment and the image information in the image features at the current moment, enabling the transfer of temporal information between different moments and obtaining accurate instance features containing 3D spatial position information. This allows 3D object detection to be performed based on the output instance features in subsequent steps.

[0060] In an optional embodiment, interactively processing the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment to obtain the output instance features at the current moment includes steps A1 and A2:

[0061] Step A1: interactively process the initial instance features at the current moment and the output instance features at the previous moment to obtain interactive instance features containing 3D spatial position information.

[0062] Step A2: interactively process the interactive instance feature containing the 3D spatial position information and the image feature at the current moment to obtain the output instance feature at the current moment.

[0063] In this embodiment, instance features from two consecutive moments are first interacted: the initial instance features at the current moment are interacted with the output instance features from the previous moment. Because the output instance features from the previous moment contain the 3D spatial position information from that moment, the initial instance features at the current moment can learn the 3D spatial position information from that moment during the interaction, thereby capturing the temporal motion information of the target object. After the interaction process in step A1, the interactive instance features containing the 3D spatial position information are obtained.

[0064] Then, the interactive instance features containing 3D spatial position information obtained in step A1 are interacted with the image features at the current moment, so that the image features at the current moment can be used to further correct the interactive instance features containing 3D spatial position information, so that the 3D position information contained in the output instance features at the current moment is more accurate, thereby making the results of 3D target detection more accurate.

[0065] In an optional embodiment, interactively processing the initial instance feature at the current moment and the output instance feature at the previous moment to obtain the interactive instance feature containing 3D spatial position information specifically includes the following steps:

[0066] Step A11: Fusing the initial instance feature at the current moment and the output instance feature at the previous moment to obtain a fused instance feature.

[0067] Step A12: Perform a first position encoding on the fused instance feature to obtain a fused instance feature containing 3D spatial position information, and perform a second position encoding on the initial instance feature at the current moment to obtain an initial instance feature containing 3D spatial position information at the current moment.

[0068] Step A13: interactively process the fused instance feature containing the 3D spatial position information and the initial instance feature containing the 3D spatial position information at the current moment to obtain an interactive instance feature containing the 3D spatial position information.

[0069] In this embodiment, in order to achieve 3D target detection, the 3D spatial position information corresponding to the instance feature also needs to be considered. Therefore, the instance feature and the 3D spatial position information corresponding to the instance feature are fused to obtain the instance feature containing the 3D spatial position information. Since the dimension of the instance feature and the dimension of the 3D spatial position information corresponding to the instance feature are different, they cannot be directly fused. Therefore, the instance feature and the 3D spatial position information corresponding to the instance feature are fused by position encoding, that is, the dimension of the 3D spatial position information is mapped to the same dimension as the instance feature, and the mapped 3D spatial position information is fused with the instance feature to obtain the instance feature containing the 3D spatial position information. For example, the dimension of the instance feature representing 100 objects is 100×256, and the dimension of the 3D spatial position information corresponding to the instance feature is 100×3. The dimension of the 3D spatial position information 100×3 is mapped to 100×256, and then the 3D spatial position information with a dimension of 100×256 is fused with the instance feature with a dimension of 100×256 to obtain the instance feature containing the 3D spatial position information.

[0070] Specifically, the first position encoding involves mapping the dimensions of the 3D spatial position information corresponding to the fused instance features, fusing the mapped 3D spatial position information with the fused instance features, and thereby obtaining fused instance features containing the 3D spatial position information. In practical applications, the 3D spatial position information corresponding to the fused instance features can be input into a layer of an MLP network. The MLP network outputs 3D spatial position information with the same dimensions as the fused instance features. The output of the MLP network is then fused with the fused instance features to achieve the first position encoding.

[0071] Second position encoding involves mapping the dimensions of the 3D spatial position information corresponding to the initial instance features at the current moment, fusing the mapped 3D spatial position information with the initial instance features, and thereby obtaining the initial instance features at the current moment that contain the 3D spatial position information. In practical applications, the 3D spatial position information corresponding to the initial instance features at the current moment can be input into a layer of an MLP network. The MLP network outputs 3D spatial position information with the same dimensions as the initial instance features at the current moment. The output of the MLP network is then fused with the initial instance features at the current moment to achieve second position encoding.

[0072] In this embodiment, after performing the first position encoding on the fused instance features and the second position encoding on the initial instance features at the current moment, when performing feature interaction processing, the instance feature similarity and the position distance of the instance features in the 3D space can be considered simultaneously to obtain the interactive instance features containing 3D spatial position information.

[0073] In step A12, a second position encoding is performed on the initial instance feature at the current moment to obtain the initial instance feature containing 3D spatial position information at the current moment, including:

[0074] Initialize the initial instance feature at the current moment and the initial 3D spatial position information at the current moment; perform a second position encoding on the initial instance feature at the current moment according to the initial 3D spatial position information at the current moment to obtain the initial instance feature at the current moment containing the 3D spatial position information.

[0075] In this embodiment, initializing the initial instance feature at the current moment refers to initializing the image at the current moment to obtain the initial instance feature at the current moment. The initial 3D spatial position information at the current moment is the 3D spatial position information corresponding to the initial instance feature at the current moment. Specifically, based on the initial 3D spatial position information at the current moment, the initial instance feature at the current moment is subjected to a second position encoding, including: mapping the dimension of the initial 3D spatial position information at the current moment to the same dimension as the initial instance feature at the current moment, and then fusing the mapped initial 3D spatial position information at the current moment with the initial instance feature at the current moment.

[0076] In step A12, a first position encoding is performed on the fused instance feature to obtain a fused instance feature containing 3D spatial position information, including:

[0077] Obtaining the 3D spatial position information of the previous moment corresponding to the output instance feature of the previous moment, the 3D spatial position information of the previous moment represents the spatial position of the 3D target detected at the previous moment; fusing the initial 3D spatial position information of the current moment with the 3D spatial position information of the previous moment to obtain fused 3D spatial position information; performing a first position encoding on the fused instance feature according to the fused 3D spatial position information to obtain a fused instance feature containing the 3D spatial position information.

[0078] In this embodiment, the 3D spatial position information at the previous moment refers to the 3D object detection result at the previous moment. After completing the 3D object detection at the previous moment, the 3D object detection result at the previous moment is saved in the memory, and then the 3D spatial position information at the previous moment is read from the memory. Specifically, based on the fused 3D spatial position information, the fused instance feature is first position-encoded, including: mapping the dimension of the fused 3D spatial position information to the same dimension as the fused instance feature, and then fusing the mapped fused 3D spatial position information with the fused instance feature.

[0079] In an optional embodiment, interactively processing the interactive instance feature containing the 3D spatial position information and the image feature at the current moment to obtain the output instance feature at the current moment includes the following steps A21 and A22:

[0080] Step A21: performing third position encoding on the image features at the current moment to obtain image features containing 3D spatial position information.

[0081] Step A22: interactively process the interactive instance feature containing the 3D spatial position information and the image feature containing the 3D spatial position information to obtain the output instance feature at the current moment.

[0082] In this embodiment, the image features at the current moment are two-dimensional features. To achieve 3D object detection in 3D space, it is necessary to bind each pixel in the image features at the current moment to the corresponding 3D spatial position. Specifically, the image features at the current moment are subjected to third position encoding so that each pixel in the image features at the current moment has 3D spatial position information.

[0083] The third position encoding involves encoding the imaging ray of each pixel in the current image feature (i.e., the line connecting each pixel and the camera's optical center). Specifically, the imaging ray of each pixel is uniformly sampled to obtain a number of sampling points. The 3D spatial position information (i.e., 3D coordinates) of each sampling point is mapped to the same dimension as the current image feature, and then fused with the current image feature to achieve the third position encoding. In practical applications, an MLP network can be used to map the 3D spatial position information of each sampling point to the same dimension as the current image feature.

[0084] In this embodiment, after performing the third position encoding on the image features at the current moment, the interactive instance features containing 3D spatial position information can be further corrected based on the image features containing 3D spatial position information, so that the 3D spatial position information of the output instance features is more accurate.

[0085] Step S103: performing 3D object detection based on the output instance features at the current moment to obtain the 3D object detection result at the current moment.

[0086] In this embodiment, the output instance feature at the current moment contains the 3D spatial position information corresponding to the instance feature, and 3D object detection is then performed using the output instance feature at the current moment to detect the specific spatial position of each instance feature (3D object).

[0087] In an optional embodiment, when the coordinates of the origin of the image at each moment in 3D space change over time, for example, when an autonomous vehicle is constantly moving, and the detection results at each moment are referenced relative to the center point of the vehicle, the coordinate system origins corresponding to the images at different moments are different. Furthermore, before step S102 of fusing the initial 3D spatial position information at the current moment with the 3D spatial position information of the 3D object detected at the previous moment to obtain the fused 3D spatial position information, the following steps may also be included:

[0088] Perform spatial transformation on the 3D spatial position information of the 3D target detected at the previous moment to obtain the 3D spatial position information of the previous moment in the current moment coordinate system;

[0089] The initial 3D spatial position information at the current moment and the 3D spatial position information at the previous moment are fused to obtain fused 3D spatial position information, including: fusing the initial 3D spatial position information at the current moment and the 3D spatial position information at the previous moment in the coordinate system of the current moment to obtain fused 3D spatial position information.

[0090] In this embodiment, since the coordinate origin of the image at each moment changes over time in the 3D space, when processing the image at the current moment, it is necessary to perform coordinate transformation on the 3D spatial position information corresponding to the instance features in different origin coordinate systems, that is, to transform the 3D spatial position information of the target object detected at the previous moment to the coordinate system corresponding to the coordinate origin at the current moment before interactive processing can be performed.

[0091] In an optional embodiment, after completing the 3D object detection in step S103, the following steps are further included:

[0092] The output instance features of the current moment and the corresponding 3D spatial position information of the current moment are saved for combining with the image of the next moment to realize 3D target detection at the next moment, wherein the 3D spatial position information of the current moment represents the spatial position of the 3D target detected at the current moment.

[0093] In this embodiment, the 3D spatial position information at the current moment is saved to indicate the specific 3D spatial position of each instance feature, so that when interactive processing is performed at the next moment, the 3D spatial position information can be converted into coordinate encoding, and information interaction can be performed according to the distance between the instance features.

[0094] In an embodiment of the present invention, there is no need to save a large image feature map, but 3D target detection is performed by using the transmission of instance features. First, the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment are obtained. Then, the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment are interactively processed to obtain the output instance features at the current moment. Finally, 3D target detection is performed based on the output instance features at the current moment to obtain the 3D target detection result at the current moment. Since the interaction is based on instance features for time series, the network only needs to save the instance features output at each moment, thereby reducing space consumption and making it easier to deploy chips. Moreover, by transmitting instance features moment by moment, the effect of longer time series information interaction is achieved, thereby effectively solving the target occlusion problem and improving the accuracy of 3D target detection.

[0095] Figure 2 Taking 3D object detection of an image at a current moment and an image at a next moment as an example, the overall framework of the 3D object detection algorithm in this embodiment is illustrated.

[0096] First, the multi-view image at the current moment is input into the feature extraction network for feature extraction to obtain the image features at the current moment. Then, the initial instance features at the current moment and the output instance features at the previous moment (which have undergone coordinate transformation) obtained from the memory are input into a feature interaction network (such as a Transformer) for interactive processing. In the interactive processing network, the initial instance features at the current moment interact with the image features at the current moment and the output instance features at the previous moment to obtain the output instance features at the current moment. 3D object detection is then performed based on the output instance features at the current moment to detect the position information of the target object in 3D space. Finally, the output instance features at the current moment and the corresponding 3D spatial position information at the current moment are saved for use in 3D object detection at the next moment. When performing 3D object detection at the next moment, the output instance features at the current moment are retrieved from the memory, and after undergoing coordinate transformation, they are input into the feature interaction network together with the image features and the initial instance features at the next moment to interact and obtain the output instance features at the next moment to achieve 3D object detection at the next moment.

[0097] Figure 3 It illustrates the process of interaction between the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment in the feature interaction network.

[0098] The feature interaction process includes two cross attention operations. The first cross attention operation interactively processes the initial instance features at the current moment and the output instance features at the previous moment to obtain interactive instance features containing 3D spatial position information, thereby realizing the interaction of temporal information. The second cross attention operation interactively processes the interactive instance features containing 3D spatial position information and the image features at the current moment to obtain the output instance features at the current moment.

[0099] Specifically, for the first cross attention operation: first, the initial instance feature at the current moment and the output instance feature at the previous moment are fused to obtain a fused instance feature; then, the fused instance feature is first positionally encoded and the initial instance feature at the current moment is second positionally encoded to obtain a fused instance feature containing 3D spatial position information and an initial instance feature at the current moment containing 3D spatial position information, respectively; finally, the fused instance feature containing 3D spatial position information and the initial instance feature at the current moment containing 3D spatial position information are input into the first cross attention for interactive processing to obtain an interactive instance feature containing 3D spatial position information. During the interactive processing, the initial instance feature containing 3D spatial position information is used as the query, and the fused instance feature containing 3D spatial position information is then used as the key and value for interaction.

[0100] For the second cross attention operation: First, perform the third position encoding on the image features at the current moment to obtain image features containing 3D spatial position information; then input the interaction instance features containing 3D spatial position information and the image features containing 3D spatial position information into the second cross attention for interactive processing, obtaining the output instance features of the current moment containing 3D spatial position information. During the interactive processing, the interaction instance features containing 3D spatial position information are used as queries, and the image features containing 3D spatial position information are used as keys and values ​​for interaction.

[0101] The embodiment of the present invention also provides a 3D target detection device, such as Figure 4 As shown, Figure 4 A schematic diagram of the structure of a 3D object detection device provided in an embodiment of the present invention, the device comprising:

[0102] A feature acquisition module 41 is configured to acquire image features at a current moment, initial instance features at a current moment, and output instance features at a previous moment, wherein the output instance features at a previous moment are used to detect a 3D object at a previous moment;

[0103] A feature interaction module 42 is configured to interactively process the image feature at the current moment, the initial instance feature at the current moment, and the output instance feature at the previous moment to obtain the output instance feature at the current moment;

[0104] The object detection module 43 is configured to perform 3D object detection based on the output instance features at the current moment, and obtain the 3D object detection result at the current moment.

[0105] In an optional embodiment, the feature interaction module includes:

[0106] A first feature interaction submodule is configured to interactively process the initial instance feature at the current moment and the output instance feature at the previous moment to obtain an interactive instance feature containing 3D spatial position information;

[0107] The second feature interaction submodule is used to interactively process the interaction instance feature containing the 3D spatial position information and the image feature at the current moment to obtain the output instance feature at the current moment.

[0108] In an optional embodiment, the first feature interaction submodule includes:

[0109] A first feature fusion unit is configured to fuse the initial instance feature at the current moment with the output instance feature at the previous moment to obtain a fused instance feature;

[0110] a first position encoding unit, configured to perform a first position encoding on the fused instance feature to obtain a fused instance feature containing 3D spatial position information, and to perform a second position encoding on the initial instance feature at the current moment to obtain an initial instance feature containing 3D spatial position information at the current moment;

[0111] The first feature interaction unit is used to interactively process the fused instance feature containing the 3D spatial position information and the initial instance feature containing the 3D spatial position information at the current moment to obtain an interactive instance feature containing the 3D spatial position information.

[0112] In an optional embodiment, the feature interaction module further includes:

[0113] Initialization feature module, used to initialize the initial instance features at the current moment, as well as the initial 3D spatial position information at the current moment;

[0114] The first position encoding subunit is configured to perform second position encoding on the initial instance feature at the current moment according to the initial 3D spatial position information at the current moment, so as to obtain the initial instance feature at the current moment containing the 3D spatial position information.

[0115] In an optional embodiment, the feature interaction module further includes:

[0116] An instance feature acquisition module, configured to obtain 3D spatial position information at a previous moment corresponding to an output instance feature at a previous moment, wherein the 3D spatial position information at a previous moment represents a spatial position of a 3D object detected at a previous moment;

[0117] a second position encoding subunit, configured to fuse the initial 3D spatial position information at the current moment with the 3D spatial position information at the previous moment to obtain fused 3D spatial position information;

[0118] The third position encoding subunit is configured to perform a first position encoding on the fused instance feature according to the fused 3D spatial position information to obtain a fused instance feature containing the 3D spatial position information.

[0119] In an optional embodiment, the second feature interaction submodule includes:

[0120] a second position encoding unit, configured to perform a third position encoding on the image feature at the current moment to obtain an image feature containing 3D spatial position information;

[0121] The second feature interaction unit is configured to interactively process the interaction instance feature containing the 3D spatial position information and the image feature containing the 3D spatial position information to obtain the output instance feature at the current moment.

[0122] In an optional embodiment, the feature interaction module includes:

[0123] The coordinate transformation module is used to perform spatial transformation on the 3D spatial position information of the 3D target detected at the previous moment to obtain the 3D spatial position information of the previous moment in the current moment coordinate system;

[0124] The third feature fusion submodule is used to fuse the initial 3D spatial position information at the current moment with the 3D spatial position information at the previous moment in the current moment coordinate system to obtain fused 3D spatial position information.

[0125] In an optional embodiment, the 3D object detection device further includes:

[0126] A feature storage module is used to store the output instance features at the current moment and the corresponding 3D spatial position information at the current moment, so as to combine with the image at the next moment to realize 3D target detection at the next moment, wherein the current 3D spatial position information represents the spatial position of the 3D target detected at the current moment.

[0127] In an optional embodiment, the 3D object detection device further includes:

[0128] An image acquisition module is used to acquire and process images, wherein the images at each moment including the current moment are multi-view images, and the multi-view images are obtained by splicing images of multiple perspectives acquired at the same moment;

[0129] The feature extraction module is used to extract features from the image at the current moment to obtain the image features at the current moment.

[0130] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the 3D target detection method described in the embodiment of the present invention.

[0131] An embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the 3D object detection method described in the embodiment of the present invention is implemented.

[0132] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0133] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, devices, and media according to the embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0134] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0136] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0137] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0138] The above is a detailed introduction to the 3D target detection method, device, equipment and medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A 3D object detection method, characterized in that: The method comprises: Obtaining image features at a current moment, initial instance features at a current moment, and output instance features at a previous moment, wherein the output instance features at the previous moment are used to detect a 3D object at the previous moment; Interactively processing the image feature at the current moment, the initial instance feature at the current moment, and the output instance feature at the previous moment to obtain the output instance feature at the current moment; The interactive process is implemented through a feature interactive network; the interactive process includes: The initial instance feature at the current moment learns the 3D spatial position information in the output instance feature at the previous moment and the image information in the image feature at the current moment to obtain the output instance feature at the current moment; 3D object detection is performed based on the output instance features at the current moment to obtain the 3D object detection result at the current moment.

2. The method according to claim 1, characterized in that The interactive processing of the image feature at the current moment, the initial instance feature at the current moment, and the output instance feature at the previous moment to obtain the output instance feature at the current moment includes: Interactively processing the initial instance feature at the current moment and the output instance feature at the previous moment to obtain an interactive instance feature containing 3D spatial position information; The interactive instance feature containing the 3D spatial position information and the image feature at the current moment are interactively processed to obtain the output instance feature at the current moment.

3. The method according to claim 2, characterized in that The interactive processing of the initial instance feature at the current moment and the output instance feature at the previous moment to obtain the interactive instance feature containing 3D spatial position information includes: Fusing the initial instance feature at the current moment with the output instance feature at the previous moment to obtain a fused instance feature; Performing a first position encoding on the fused instance feature to obtain a fused instance feature containing 3D spatial position information, and performing a second position encoding on the initial instance feature at the current moment to obtain an initial instance feature containing 3D spatial position information at the current moment; The fused instance feature containing the 3D spatial position information and the initial instance feature containing the 3D spatial position information at the current moment are interactively processed to obtain an interactive instance feature containing the 3D spatial position information.

4. The method according to claim 3, characterized in that The method further comprises: Initialize the initial instance features at the current moment, as well as the initial 3D spatial position information at the current moment; Performing a second position encoding on the initial instance feature at the current moment to obtain the initial instance feature containing 3D spatial position information at the current moment, including: According to the initial 3D spatial position information at the current moment, a second position encoding is performed on the initial instance feature at the current moment to obtain the initial instance feature containing the 3D spatial position information at the current moment.

5. The method according to claim 4, characterized in that The method further comprises: Obtaining 3D spatial position information at a previous moment corresponding to the output instance feature at the previous moment, wherein the 3D spatial position information at the previous moment represents the spatial position of the 3D target detected at the previous moment; fusing the initial 3D spatial position information at the current moment with the 3D spatial position information at the previous moment to obtain fused 3D spatial position information; Performing a first position encoding on the fused instance feature to obtain a fused instance feature containing 3D spatial position information includes: According to the fused 3D spatial position information, a first position encoding is performed on the fused instance feature to obtain a fused instance feature containing the 3D spatial position information.

6. The method according to claim 2, characterized in that Interactively processing the interactive instance feature containing the 3D spatial position information and the image feature at the current moment to obtain the output instance feature at the current moment includes: Performing a third position encoding on the image feature at the current moment to obtain an image feature containing 3D spatial position information; The interactive instance feature containing the 3D spatial position information and the image feature containing the 3D spatial position information are interactively processed to obtain the output instance feature at the current moment.

7. The method according to claim 5, characterized in that Before fusing the initial 3D spatial position information at the current moment with the 3D spatial position information of the 3D object detected at the previous moment to obtain the fused 3D spatial position information, the method further includes: Perform spatial transformation on the 3D spatial position information of the 3D target detected at the previous moment to obtain the 3D spatial position information of the previous moment in the current moment coordinate system; Fusing the initial 3D spatial position information at the current moment with the 3D spatial position information at the previous moment to obtain fused 3D spatial position information, including: The initial 3D spatial position information at the current moment and the 3D spatial position information at the previous moment in the current moment coordinate system are fused to obtain fused 3D spatial position information.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The output instance features of the current moment and the corresponding 3D spatial position information of the current moment are saved for combining with the image of the next moment to realize 3D target detection at the next moment, wherein the 3D spatial position information of the current moment represents the spatial position of the 3D target detected at the current moment.

9. The method according to any one of claims 1 to 7, characterized in that: The image at each moment including the current moment is a multi-view image, and the multi-view image is obtained by stitching images of multiple perspectives collected at the same moment; Get the image features at the current moment, including: Feature extraction is performed on the image at the current moment to obtain image features at the current moment.

10. A 3D object detection device, characterized in that: The device comprises: A feature acquisition module is used to acquire the image features at the current moment, the initial instance features at the current moment, and the output instance features at the previous moment, wherein the output instance features at the previous moment are used to detect the 3D object at the previous moment; a feature interaction module, configured to interactively process the image feature at the current moment, the initial instance feature at the current moment, and the output instance feature at the previous moment to obtain the output instance feature at the current moment; The interactive process is implemented through a feature interactive network; the interactive process includes: The initial instance feature at the current moment learns the 3D spatial position information in the output instance feature at the previous moment and the image information in the image feature at the current moment to obtain the output instance feature at the current moment; The target detection module is used to perform 3D target detection based on the output instance features at the current moment to obtain the 3D target detection result at the current moment.

11. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method implements the 3D target detection method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the 3D object detection method according to any one of claims 1 to 9 is implemented.

Images