Target detection method and electronic device

Abstract

The application provides a target detection method and an electronic device. The method is applied to a target detection model. The target detection model comprises a backbone network, a joint decoder, and a detection head comprising a two-dimensional detection head and a three-dimensional detection head. The joint decoder obtains target two-dimensional query feature information and target three-dimensional query feature information of each target to be detected according to multiple image feature information obtained by the backbone network according to multiple images corresponding to different perspectives and three-dimensional query feature information of multiple targets to be detected corresponding to the multiple images. The two-dimensional detection head obtains two-dimensional detection results corresponding to the multiple targets to be detected according to the target two-dimensional query feature information. The three-dimensional detection head obtains three-dimensional detection results corresponding to the multiple targets to be detected according to the target three-dimensional query feature information. In this way, more accurate target detection can be realized through the combination of two-dimensional detection and three-dimensional detection, more comprehensive and accurate detection results can be obtained, and the accuracy of the detection results is improved.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a target detection method and electronic device. Background Technology

[0002] Object detection is a core technology in computer vision, aiming to identify the category of target objects in images or videos and accurately locate their positions. The target object is the object to be detected, and its position is usually represented by a bounding box. With the continuous development of deep learning, object detection technology has made significant progress and is widely used in various fields such as autonomous driving, security monitoring, medical image analysis, and industrial quality inspection.

[0003] Taking the field of assisted driving as an example, the more accurate the target detection results obtained from target detection methods, the more precise the driving control based on those results will be. Therefore, how to perform target detection more accurately to obtain more accurate results becomes particularly important. Summary of the Invention

[0004] This application provides a target detection method and electronic device that can perform target detection more accurately, thereby obtaining more accurate detection results and improving the accuracy of the detection results.

[0005] To address the aforementioned technical problems, in a first aspect, embodiments of this application provide a target detection method applied to a target detection model. The target detection model includes a backbone network, a joint decoder, and a detection head. The detection head includes a two-dimensional detection head and a three-dimensional detection head. The method includes: the backbone network obtaining multiple image feature information based on multiple images corresponding to different viewpoints, and inputting the multiple image feature information into the joint decoder; the joint decoder obtaining target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional feature information of multiple targets to be detected corresponding to the multiple images and the multiple image feature information, inputting the target two-dimensional query feature information into the two-dimensional detection head, and inputting the target three-dimensional query feature information into the three-dimensional detection head; the two-dimensional detection head obtaining two-dimensional detection results corresponding to multiple targets to be detected based on the target two-dimensional query feature information; and the three-dimensional detection head obtaining three-dimensional detection results corresponding to multiple targets to be detected based on the target three-dimensional query feature information.

[0006] By employing the above technical solution, the joint decoder based on the target detection model obtains the target's two-dimensional and three-dimensional query feature information based on multiple image feature information and the three-dimensional query feature information of multiple targets to be detected. The two-dimensional detection head based on the joint decoder obtains the two-dimensional detection results of multiple targets to be detected based on the target's two-dimensional query feature information, and the three-dimensional detection head based on the joint decoder obtains the three-dimensional detection results of multiple targets to be detected based on the target's three-dimensional query feature information. Thus, the target detection model performs both two-dimensional and three-dimensional target detection based on the three-dimensional query feature information. This combination of two-dimensional and three-dimensional target detection achieves more accurate target detection, and the obtained detection results include both two-dimensional and three-dimensional detection results, resulting in more comprehensive and accurate detection results and improving the accuracy of the detection results.

[0007] In one possible implementation of the first aspect above, the three-dimensional feature information includes three-dimensional query feature information and three-dimensional attribute feature information. The joint decoder obtains target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional feature information of multiple targets to be detected corresponding to multiple images and multiple image feature information. This includes: the joint decoder obtains multiple sets of two-dimensional query feature information corresponding to each target to be detected based on multiple image feature information, corresponding three-dimensional query feature information and three-dimensional attribute feature information, as target two-dimensional query feature information; and obtains second three-dimensional query feature information based on multiple sets of two-dimensional query feature information, as target three-dimensional query feature information.

[0008] In one possible implementation of the first aspect above, the joint decoder obtains multiple sets of two-dimensional query feature information corresponding to each target to be detected based on multiple image feature information, corresponding three-dimensional query feature information, and three-dimensional attribute feature information, as target two-dimensional query feature information; and obtains second three-dimensional query feature information based on the multiple sets of two-dimensional query feature information, as target three-dimensional query feature information. This includes: the joint decoder initializes the three-dimensional query feature information to obtain fourth three-dimensional query feature information; performs clustering processing on the three-dimensional attribute feature information to obtain second three-dimensional attribute feature information; determines the camera parameters of the camera used to capture the multiple images; and obtains the transformation moments corresponding to the multiple targets to be detected based on the camera parameters and the second three-dimensional attribute information. The process involves: obtaining multiple two-dimensional query feature information corresponding to multiple targets based on the fourth three-dimensional query feature information and the transformation matrix; grouping these multiple two-dimensional query feature information to obtain multiple sets of two-dimensional query feature information for multiple targets; obtaining updated multiple sets of two-dimensional query feature information corresponding to multiple targets based on the multiple sets of two-dimensional query feature information and multiple image feature information, which are then used as target two-dimensional query feature information; obtaining third three-dimensional query feature information corresponding to each target based on the updated multiple sets of two-dimensional query feature information and the transformation matrix; and obtaining second three-dimensional query feature information corresponding to each target based on multiple image feature information and the third three-dimensional query feature information, which are then used as target three-dimensional query feature information.

[0009] In one possible implementation of the first aspect described above, the joint decoder includes multiple decoding layers. The backbone network inputs multiple image feature information into the joint decoder, including: the backbone network inputs multiple image feature information into each decoding layer respectively; the three-dimensional feature information includes three-dimensional query feature information and three-dimensional attribute feature information. Then, the joint decoder obtains target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional query feature information of multiple targets to be detected corresponding to the multiple input images and the multiple image feature information, including: each decoding layer obtains multiple sets of two-dimensional query feature information corresponding to each target based on the multiple image feature information, the corresponding first three-dimensional query feature information, and the first three-dimensional attribute feature information, and obtains second three-dimensional query feature information based on the multiple sets of two-dimensional query feature information. The second three-dimensional query feature information is used as the first three-dimensional query feature information corresponding to the next level of decoding layer, so that the next level of decoding layer can perform corresponding processing, until the last level of decoding layer completes processing, and obtains the target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target to be detected. The target three-dimensional query feature information includes the second three-dimensional query feature information corresponding to each target to be detected obtained by each level of decoding layer, and the target two-dimensional query feature information includes multiple sets of two-dimensional query feature information corresponding to each target to be detected obtained by each level of decoding layer, or the target three-dimensional query feature information includes the second three-dimensional query feature information corresponding to each target to be detected obtained by the last level of decoding layer, and the target two-dimensional query feature information includes multiple sets of two-dimensional query feature information corresponding to each target to be detected obtained by the last level of decoding layer.

[0010] Among them, the first three-dimensional query feature information corresponding to the first level of decoding layer is three-dimensional query feature information, and the first three-dimensional attribute feature information corresponding to each level of decoding layer is three-dimensional attribute feature information.

[0011] By adopting the above technical solution, the multi-level decoding layer updates the query feature information step by step, which can obtain more accurate and comprehensive query feature information, and thus obtain more accurate target detection results.

[0012] In one possible implementation of the first aspect described above, the joint decoder includes multiple decoding layers. The backbone network inputs multiple image feature information into the joint decoder included in the target detection model, including: the backbone network inputs multiple image feature information into the decoding layers of the joint decoder; the joint decoder obtains target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional feature information of multiple targets to be detected corresponding to multiple images and the multiple image feature information, including: each decoding layer obtains multiple sets of two-dimensional query feature information corresponding to each target to be detected based on the input multiple image feature information and three-dimensional feature information, as target two-dimensional query feature information, and obtains second three-dimensional query feature information based on the multiple sets of two-dimensional query feature information, as target three-dimensional query feature information.

[0013] In one possible implementation of the first aspect described above, each decoding layer includes a two-dimensional decoding layer and a three-dimensional decoding layer. The backbone network inputs multiple image feature information to each decoding layer, including: the backbone network inputs multiple image feature information to the two-dimensional and three-dimensional decoding layers included in each decoding layer; the method further includes that each decoding layer obtains multiple sets of two-dimensional query feature information and second-dimensional query feature information corresponding to each target to be detected in the following manner: the two-dimensional decoding layer obtains multiple sets of two-dimensional query feature information corresponding to each target to be detected based on multiple image feature information, first-dimensional query feature information, and first-dimensional attribute feature information, and obtains third-dimensional query feature information corresponding to each target to be detected based on the multiple sets of two-dimensional query feature information, and inputs the third-dimensional query feature information to the corresponding three-dimensional decoding layer; the three-dimensional decoding layer obtains second-dimensional query feature information corresponding to each target to be detected based on multiple image feature information and third-dimensional query feature information.

[0014] By adopting the above technical solution, the three-dimensional query feature information is updated step by step based on the two-dimensional decoding layer and the three-dimensional decoding layer, resulting in multiple sets of two-dimensional query feature information and new three-dimensional query feature information. Target detection is then performed by combining the multiple sets of two-dimensional query feature information and the new three-dimensional query feature information, enabling more comprehensive and accurate target detection.

[0015] In one possible implementation of the first aspect described above, the two-dimensional decoding layer includes a distribution module, a deformable attention module, and an aggregation module. The method further includes the two-dimensional decoding layer obtaining multiple sets of two-dimensional query feature information and third-dimensional query feature information in the following manner: the distribution module initializes the first-dimensional query feature information to obtain fourth-dimensional query feature information, and performs clustering processing on the first-dimensional attribute feature information to obtain second-dimensional attribute feature information; the distribution module determines the camera parameters of the camera used to capture multiple images, obtains transformation matrices corresponding to multiple targets to be detected based on the camera parameters and the second-dimensional attribute information, and inputs the transformation matrices to the aggregation module; the distribution module then... The fourth three-dimensional query feature information and the transformation matrix are used to obtain multiple two-dimensional query feature information corresponding to multiple targets to be detected. The multiple two-dimensional query feature information is then grouped to obtain multiple sets of two-dimensional query feature information for multiple targets to be detected. The multiple sets of two-dimensional query feature information are then input into the deformable attention module. The deformable attention module obtains updated multiple sets of two-dimensional query feature information corresponding to multiple targets to be detected based on the multiple sets of two-dimensional query feature information and multiple image feature information. The updated multiple sets of two-dimensional query feature information are then input into the aggregation module. The aggregation module obtains the third three-dimensional query feature information corresponding to each target to be detected based on the updated multiple sets of two-dimensional query feature information and the transformation matrix.

[0016] Using the above technical solution, the query feature information is updated based on the first three-dimensional query feature information and image feature information to obtain multiple sets of updated two-dimensional query feature information. Based on these multiple sets of two-dimensional query feature information, two-dimensional target detection can be performed more accurately.

[0017] In one possible implementation of the first aspect above, the three-dimensional decoding layer includes a deformable aggregation module. The method further includes that the three-dimensional decoding layer obtains the second three-dimensional query feature information in the following manner: the deformable aggregation module obtains the second three-dimensional query feature information corresponding to each target to be detected based on multiple image feature information and the third three-dimensional query feature information.

[0018] Using the above technical solution, the deformable aggregation module updates its data again based on multiple input image feature information and the third 3D query feature information, obtaining the updated second 3D query feature information corresponding to each target to be detected. In this way, new 3D query feature information is gradually obtained, and then more accurate 3D target detection is performed based on the new 3D query feature information.

[0019] In one possible implementation of the first aspect above, the method further includes the distribution module obtaining multiple sets of two-dimensional query feature information in the following manner:

[0020]

[0021] in, For multiple sets of two-dimensional query feature information, for The transformation matrix, where M is the number of two-dimensional query feature information and N is the number of first three-dimensional query feature information. This is the first three-dimensional query feature information.

[0022] In one possible implementation of the first aspect above, the method further includes the aggregation module obtaining the third-dimensional query feature information in the following manner:

[0023]

[0024] in, For third-dimensional query feature information, For the transformation matrix, This is the updated set of two-dimensional query feature information.

[0025] In one possible implementation of the first aspect described above, the two-dimensional detection result includes the category of each target to be detected and the two-dimensional detection box information, and the three-dimensional detection result includes the category of each target to be detected and the three-dimensional detection box information.

[0026] In one possible implementation of the first aspect mentioned above, the target detection model is a detection model obtained by updating the model based on the corresponding loss. The loss includes two-dimensional loss and three-dimensional loss, and the loss is determined based on historical two-dimensional detection results and historical three-dimensional detection results.

[0027] By adopting the above technical solution, the target detection model is updated based on two-dimensional loss and three-dimensional loss, so that the updated target detection model can perform target detection more accurately.

[0028] Secondly, this application also discloses a target detection model update method, which determines historical two-dimensional detection results and historical three-dimensional detection results, which are obtained according to the target detection method provided by any of the implementation methods in the first aspect. Based on the historical two-dimensional detection results and historical three-dimensional detection results, the loss corresponding to the target detection model is determined, including two-dimensional loss and three-dimensional loss. The target detection model is updated based on the loss.

[0029] Thirdly, this application also discloses an electronic device, including a target detection model, which implements the target detection method provided by any of the implementations of the first aspect above.

[0030] Fourthly, this application also discloses a computer-readable storage medium storing a computer program that can be executed by an electronic device to implement the target detection method provided in any of the implementations of the first aspect.

[0031] Fifthly, this application also discloses a computer program product, including a computer program, which, when executed by an electronic device, implements the target detection method provided by any of the implementations of the first aspect. Attached Figure Description

[0032] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0033] Figure 1 This is a schematic diagram illustrating the principle of a target detection method in the prior art;

[0034] Figure 2 A schematic diagram illustrating the principle of the target detection method provided in this embodiment of the invention;

[0035] Figure 3 A schematic diagram of the structure of a target detection model provided in an embodiment of the present invention;

[0036] Figure 4 This is a schematic flowchart of a target detection method provided in an embodiment of the present invention;

[0037] Figure 5 A schematic diagram illustrating the principle of grouping in the distribution module provided in this embodiment of the invention;

[0038] Figure 6 This is a schematic diagram of two-dimensional query feature information for grouping provided in an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram illustrating the principle of two-dimensional target detection using the two-dimensional detection head provided in an embodiment of the present invention.

[0040] Figure 8 This is a schematic diagram illustrating the principle of three-dimensional target detection using the three-dimensional detection head provided in this embodiment of the invention.

[0041] Figure 9 This is a schematic diagram illustrating the principle of loss processing based on a loss function, provided in an embodiment of the present invention.

[0042] Figure 10 This is a schematic diagram illustrating a process by which the target detection model provided in an embodiment of the present invention obtains two-dimensional and three-dimensional target query feature information.

[0043] Figure 11 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0044] Taking the field of autonomous driving as an example, the more accurate the target detection results obtained from target detection methods, the more precise the driving control based on those results will be. Therefore, it is particularly important to find ways to perform target detection more accurately to obtain more precise results.

[0045] like Figure 1 As shown, traditional target detection methods include, for example... Figure 1 The multiview shown in section (a) Figure 3 3D object detection involves inputting images corresponding to different viewpoints into a 3D detector to generate 3D bounding boxes (3D boxes). However, this method only detects the 3D boxes and their associated attributes, resulting in limited detection content and incomplete or inaccurate object detection. Furthermore, some existing object detection methods demonstrate that using the detection results of a 2D detector to initialize 3D query feature information (i.e., 3D query) can achieve better 3D performance, such as... Figure 1 The multi-view based on a two-dimensional (2D) detector shown in section (b) Figure 3 3D object detection involves inputting images corresponding to different viewpoints into a 2D detector and a 3D detector, respectively, so that the 2D detector can obtain 2D detection boxes, and then inputting the 2D detection boxes into the 3D detector to obtain 3D detection boxes. In this way, the 2D detector is only used for initialization once, and the output 2D detection results are relatively coarse, and the utilization efficiency of the 2D detector is limited. The query for initializing the 3D detector depends on the results of the 2D detector, which places certain requirements on the performance of the 2D detector. The features of the 2D detection results are not explicitly used in the 3D detector.

[0046] Based on this, this application proposes a multi-view object detection method that combines 2D and 3D methods, such as... Figure 2 As shown, images corresponding to different viewpoints are input into the joint detector so that the joint detector outputs 2D detection boxes and 3D detection boxes, and the 2D detection boxes and 3D detection boxes are related. In this way, 2D and 3D features can be effectively combined to achieve simultaneous detection of 2D and 3D targets.

[0047] Specifically, the target detection model obtains multiple image feature information from multiple images corresponding to different viewpoints. Then, based on the multiple image information and the 3D queries of multiple targets to be detected corresponding to these images (as an example of 3D query feature information), it obtains 2D and 3D target query feature information, thereby yielding 2D and 3D detection results. In this way, by combining 2D and 3D detection, it achieves simultaneous detection of 2D and 3D targets, resulting in more comprehensive and accurate target detection results.

[0048] The target detection method provided in this application is applied to a target detection model, such as... Figure 3 As shown, the target detection model includes a backbone network, a joint decoder, and a detection head, which includes a two-dimensional detection head and a three-dimensional detection head.

[0049] The object detection model can be a neural network model, such as a feedforward neural network (FNN) or a recurrent neural network (RNN).

[0050] like Figure 4 As shown, the target detection model achieves target detection through the following steps.

[0051] S100, the backbone network obtains multiple image feature information based on multiple input images corresponding to different viewpoints.

[0052] In S200, the backbone network inputs multiple image feature information into the joint decoder.

[0053] S300, the joint decoder obtains the target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional query feature information of multiple targets to be detected corresponding to multiple input images and multiple image feature information.

[0054] S400 inputs the target two-dimensional query feature information into the two-dimensional detection head.

[0055] S500 inputs the target's 3D query feature information into the 3D detection head.

[0056] S600: The two-dimensional detection head obtains two-dimensional detection results corresponding to multiple targets based on the input target two-dimensional query feature information.

[0057] The S700 3D detection head obtains 3D detection results for multiple targets based on the input target 3D query feature information.

[0058] In this application, a joint decoder based on an object detection model obtains the target's two-dimensional and three-dimensional query feature information based on multiple image feature information and the three-dimensional query feature information of multiple targets to be detected. A two-dimensional detection head based on the joint decoder obtains the two-dimensional detection results of multiple targets to be detected based on the target's two-dimensional query feature information, and a three-dimensional detection head based on the joint decoder obtains the three-dimensional detection results of multiple targets to be detected based on the target's three-dimensional query feature information. Thus, the object detection model performs both two-dimensional and three-dimensional target detection based on the three-dimensional query feature information. This combination of two-dimensional and three-dimensional target detection achieves more accurate target detection, and the obtained detection results include both two-dimensional and three-dimensional detection results, resulting in more comprehensive and accurate detection results.

[0059] Taking target detection in the process of vehicle assisted driving as an example, firstly, step S100 is executed, whereby the electronic device acquires multiple images of the vehicle's surroundings from different perspectives captured by the vehicle's surround-view camera and other devices. These multiple images are then input into the backbone network of the target detection model. The backbone network extracts image features based on the input multiple images corresponding to different perspectives, thereby obtaining multiple image feature information.

[0060] Further, in step S200, the backbone network inputs multiple image feature information into the joint decoder.

[0061] One approach is to use a ResNet50 pre-trained on ImageNet as the backbone network for image feature extraction.

[0062] Further, step S300 is executed to obtain the 3D query of each target to be detected corresponding to the image from each viewpoint (as an example of three-dimensional query feature information), and input the image feature information and 3D query into the joint decoder so that the joint decoder obtains the target 2D query (as an example of target two-dimensional query feature information) and target 3D query (as an example of target three-dimensional query feature information) of multiple targets based on the multiple image feature information and 3D query.

[0063] Here, 3D query represents the three-dimensional feature information of each target to be detected.

[0064] For example, the three-dimensional feature information includes three-dimensional query feature information and three-dimensional attribute feature information. The joint decoder obtains target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional feature information of multiple targets to be detected corresponding to multiple images and multiple image feature information. This includes: the joint decoder obtains multiple sets of two-dimensional query feature information corresponding to each target based on multiple image feature information, corresponding three-dimensional query feature information and three-dimensional attribute feature information, as target two-dimensional query feature information; and obtains second three-dimensional query feature information based on multiple sets of two-dimensional query feature information, as target three-dimensional query feature information.

[0065] Furthermore, the joint decoder obtains multiple sets of two-dimensional query feature information corresponding to each target to be detected based on multiple image feature information, corresponding three-dimensional query feature information, and three-dimensional attribute feature information, as target two-dimensional query feature information. It then obtains second three-dimensional query feature information based on these multiple sets of two-dimensional query feature information, also as target three-dimensional query feature information. This includes: the joint decoder initializes the three-dimensional query feature information to obtain fourth three-dimensional query feature information; it clusters the three-dimensional attribute feature information to obtain second three-dimensional attribute feature information; it determines the camera parameters of the camera used to capture multiple images; and it obtains the transformation matrix corresponding to multiple targets to be detected based on the camera parameters and the second three-dimensional attribute information; based on the fourth... The three-dimensional query feature information and transformation matrix are used to obtain multiple two-dimensional query feature information corresponding to multiple targets to be detected. The multiple two-dimensional query feature information is then grouped to obtain multiple sets of two-dimensional query feature information for multiple targets to be detected. Based on the multiple sets of two-dimensional query feature information and multiple image feature information, the updated multiple sets of two-dimensional query feature information corresponding to multiple targets to be detected are obtained, which are used as target two-dimensional query feature information. Based on the updated multiple sets of two-dimensional query feature information and transformation matrix, the third three-dimensional query feature information corresponding to each target to be detected is obtained. Based on multiple image feature information and the third three-dimensional query feature information, the second three-dimensional query feature information corresponding to each target to be detected is obtained, which are used as target three-dimensional query feature information.

[0066] In one implementation of this application, see further details. Figure 3 The joint decoder includes multiple decoding layers.

[0067] The backbone network inputs multiple image feature information to each level of the decoding layer.

[0068] Furthermore, the joint decoder obtains target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional query feature information of multiple targets corresponding to multiple input images and multiple image feature information. This includes: each decoding layer obtains multiple sets of two-dimensional query feature information corresponding to each target based on the multiple input image feature information and three-dimensional feature information, which are used as target two-dimensional query feature information, and obtains second three-dimensional query feature information based on the multiple sets of two-dimensional query feature information, which are used as target three-dimensional query feature information.

[0069] For example, the joint decoder contains three joint decoding layers. Each decoding layer obtains multiple sets of 2D queries corresponding to each target to be detected based on multiple image feature information and 3D queries, and obtains 3D query' (as an example of the second three-dimensional query feature information) based on the multiple sets of 2D queries. The multiple sets of 2D queries obtained by each decoding layer are used as the target 2D query, and the 3D query' obtained by each decoding layer is used as the target 3D query.

[0070] Furthermore, the output 3D query from the previous decoding layer can be used as the input 3D query for the next decoding layer to perform layer updates.

[0071] Therefore, in the implementation of this application, the joint decoder obtains target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional feature information of multiple targets to be detected corresponding to multiple input images and multiple image feature information. This includes: each level of the decoding layer obtains multiple sets of two-dimensional query feature information corresponding to each target based on the multiple input image feature information, the first three-dimensional query feature information, and the first three-dimensional attribute feature information; obtains second three-dimensional query feature information based on the multiple sets of two-dimensional query feature information; and inputs the second three-dimensional query feature information as the first three-dimensional query feature information of the next level of the decoding layer to the next level of the decoding layer so that the next level of the decoding layer can perform corresponding processing until the last level of the decoding layer completes the processing to obtain target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target.

[0072] The target three-dimensional query feature information includes the second three-dimensional query feature information corresponding to each target to be detected obtained from each level of the decoding layer, and the target two-dimensional query feature information includes multiple sets of two-dimensional query feature information corresponding to each target to be detected obtained from each level of the decoding layer, or the target three-dimensional query feature information includes the second three-dimensional query feature information corresponding to each target to be detected obtained from the last level of the decoding layer, and the target two-dimensional query feature information includes multiple sets of two-dimensional query feature information corresponding to each target to be detected obtained from the last level of the decoding layer.

[0073] For example, taking a three-level decoding layer, the first-level decoding layer obtains multiple sets of 2D queries based on multiple image feature information, 3D queries (i.e., the first three-dimensional query feature information corresponding to the first-level decoding layer), and 3D anchors (i.e., the first three-dimensional attribute feature information). Based on these multiple sets of 2D queries, it obtains a 3D query' (as an example of the second three-dimensional query feature information obtained by the first-level decoding layer). The 3D query' is then input into the second-level decoding layer. The second-level decoding layer obtains multiple sets of 2D queries based on multiple image feature information, 3D queries' (i.e., the first three-dimensional query feature information corresponding to the second-level decoding layer), and 3D anchors (i.e., the first three-dimensional attribute feature information). Based on these multiple sets of 2D queries, it obtains a 3D query' (as an example of the second three-dimensional query feature information obtained by the second-level decoding layer). The 3D query' is then input into the third-level decoding layer. The second-level decoding layer obtains multiple sets of 2D queries based on multiple image feature information, 3D queries' (i.e., the first three-dimensional query feature information corresponding to the third-level decoding layer), and 3D anchors (i.e., the first three-dimensional attribute feature information). Based on these multiple sets of 2D queries, it obtains a 3D query' (as an example of the second three-dimensional query feature information obtained by the second-level decoding layer). The 3D query' is then input into the third-level decoding layer. The second-level decoding layer obtains multiple sets of 2D queries based on multiple image feature information, 3D queries' (i.e., the first three-dimensional query feature information corresponding to the third-level decoding layer), and 3D anchors (i.e., the first three-dimensional attribute feature information). 'query' (as an example of the second three-dimensional query feature information corresponding to the third-level decoding layer).

[0074] Each decoding layer can use multiple sets of 2D queries as the target 2D query and each decoding layer can use a 3D query' as the target 3D query. Alternatively, the last decoding layer can use multiple sets of 2D queries as the target 2D query and the last decoding layer can use a 3D query' as the target 3D query.

[0075] It should be noted that for the first level of decoding layer, the first three-dimensional query feature information (i.e., 3D query') corresponding to the first level of decoding layer is three-dimensional query feature information (3D query), and the first three-dimensional attribute feature information corresponding to each level of decoding layer is three-dimensional attribute feature information.

[0076] Furthermore, in this application, each decoding layer includes a two-dimensional decoding layer (i.e., a 2D decoding layer) and a three-dimensional decoding layer (i.e., a 3D decoding layer), and the backbone network inputs multiple image feature information to the two-dimensional decoding layer and the three-dimensional decoding layer included in each decoding layer.

[0077] Each decoding layer obtains the second three-dimensional query feature information and multiple sets of two-dimensional query feature information corresponding to each target to be detected in the following way.

[0078] First, the two-dimensional decoding layer obtains multiple sets of two-dimensional query feature information corresponding to each target to be detected based on multiple input image feature information, first three-dimensional query feature information (or three-dimensional query feature information), and first three-dimensional attribute feature information. Then, it obtains third three-dimensional query feature information corresponding to each target to be detected based on the multiple sets of two-dimensional query feature information and inputs the third three-dimensional query feature information to the corresponding three-dimensional decoding layer.

[0079] In the implementation of this application, the input of the target detection model also includes a 3D anchor (as an example of the first three-dimensional attribute information), which contains the reference position, size and related attribute information of the target to be detected.

[0080] For example, see [link to example]. Figure 3 The two-dimensional decoding layer includes a distribution module, a deformable attention module, and an aggregation module.

[0081] The two-dimensional decoding layer obtains the third-dimensional query feature information in the following way.

[0082] First, the distribution module performs random initialization processing on the input 3D query (for the distribution module of the first level decoding layer, it is the three-dimensional query feature information; for the distribution modules of other levels decoding layers, it is the first three-dimensional query feature information) to obtain an initialized 3D query with dimensions [n, 256] (as an example of the fourth three-dimensional query feature information). Then, it performs clustering processing on the input 3D anchor (that is, the first three-dimensional attribute feature information, also known as the three-dimensional attribute feature information) to obtain a clustered 3D anchor with dimensions [n, 11] (as an example of the second three-dimensional attribute information).

[0083] Specifically, the clustering process can be K-means clustering, with 11-dimensional attributes [x,y,z,l,w,h,sin,cos,vx,vy,vz], where x,y,z are the three-dimensional coordinates, l,w,h are the length, width, and height of the three-dimensional detection box, sin,cos are the sine and cosine values, and vx,vy,vz are the velocities of the target relative to the x, y, and z directions.

[0084] Furthermore, the distribution module determines the camera parameters of the camera used to capture multiple images, obtains the transformation matrix corresponding to multiple targets to be detected based on the camera parameters and the second and third-dimensional attribute information, and inputs the transformation matrix into the aggregation module.

[0085] For example, such as Figure 5As shown, the distribution module projects the 3D anchor (anchor3d) onto the multi-view images based on the camera parameters, obtaining a transformation matrix. This transformation matrix is ​​a 0-1 matrix with dimensions [n, m], used to implement the mapping relationship for distributing the 3D query (query3d) to different camera views. , M represents the number of two-dimensional query feature information, and N represents the number of first three-dimensional query feature information.

[0086] Furthermore, the distribution module obtains multiple two-dimensional query feature information corresponding to multiple targets to be detected based on the fourth three-dimensional query feature information and the transformation matrix, groups the multiple two-dimensional query feature information to obtain multiple sets of two-dimensional query feature information for multiple targets to be detected, and inputs the multiple sets of two-dimensional query feature information into the deformable attention module.

[0087] For example, based on the transformation matrix, the distribution module is able to distribute 3D queries and 3D anchors to multiple views.

[0088] like Figure 5 and Figure 6 As shown, if the 3D anchor corresponding to a 3D query for a target to be detected spans two viewpoints, it will be copied for 2D target detection on different cameras (reference_points2d). The 3D anchor is projected onto the multi-view image, and the center point of the bounding rectangle is selected as the reference point for subsequent deformable attention (DA) module processing.

[0089] In other words, based on the number of 3D anchors corresponding to the 3D query of each target to be detected on each image, the N value of the corresponding transformation matrix can be determined, which in turn determines the number of 2D queries in the group.

[0090] Furthermore, the distribution module obtains multiple 2D queries based on the 3D query and the transformation matrix.

[0091] The distribution module obtains multiple sets of two-dimensional query feature information in the following way:

[0092]

[0093] in, For multiple sets of two-dimensional query feature information, for The transformation matrix, where M is the number of two-dimensional query feature information and N is the number of first three-dimensional query feature information. This is the first three-dimensional query feature information.

[0094] Trans is a matrix containing only two values, {0, 1}. Trans[n, m] represents the nth 3D query to be distributed to m positions of 2D queries. If the nth 3D query is to be distributed into two 2D queries (with indices m1 and m2), then Trans[n, m1] and Trans[n, m2] are equal to 1. During matrix multiplication, this 3D query is naturally copied and distributed into one or more queries.

[0095] Taking four targets to be detected as an example, the center point and six circumscribed corner points of the 3D anchor corresponding to the 3D query of target 1 are projected onto the viewpoints of each camera through camera parameters K. When a point falls inside the image and its depth is greater than 0, it is considered that the 3D anchor needs to be distributed to this image viewpoint. It can then be distributed into two 2D queries (2D query1 and 2D query2) based on the transformation matrix. Similarly, the 3D query of target 2 is distributed into two 2D queries (2D query3 and 2D query4) based on the transformation matrix. The 3D query of target 3 is transformed into a 2D query5 based on the transformation matrix, and the 3D query of target 4 is transformed into a 2D query6 based on the transformation matrix. In this way, the 3D anchors of all targets to be detected are grouped in turn to obtain the relationship of n 3D anchors distributed to the multi-view view. The Trans matrix is ​​then filled to obtain the Trans matrix, which is used to obtain multiple 2D queries.

[0096] For multiple 2D queries, based on the projection of the 3D anchor on the view (image), 2D query1 and 2D query3 located in the same view can be grouped into one group, and 2D query2, 2D query4, 2D query5 and 2D query6 located in another view can be grouped into another group, thus obtaining multiple groups of 2D queries.

[0097] Furthermore, the deformable attention module obtains updated sets of two-dimensional query feature information corresponding to multiple targets to be detected based on multiple sets of two-dimensional query feature information and multiple image feature information, and inputs the updated sets of two-dimensional query feature information into the aggregation module.

[0098] For example, the deformable attention module samples the image features from each viewpoint individually to update each group of 2D queries by combining the grouped multi-view image features, resulting in updated multiple groups of 2D queries.

[0099] The aggregation module obtains the third and third-dimensional query feature information corresponding to each target to be detected based on the updated multiple sets of two-dimensional query feature information and transformation matrix.

[0100] For example, the aggregation module reassembles the updated 2D query into a 3D query for use in the subsequent 3D decoding layer.

[0101] The aggregation module obtains the third-dimensional query feature information in the following way:

[0102]

[0103] in, For third-dimensional query feature information, For the transformation matrix, This is the updated set of two-dimensional query feature information.

[0104] Furthermore, the 3D decoding layer obtains the second 3D query feature information corresponding to each target to be detected based on the input multiple image feature information and the third 3D query feature information.

[0105] See also Figure 3 The 3D decoding layer includes a deformable aggregation module.

[0106] The 3D decoding layer obtains the second 3D query feature information in the following way: the deformable aggregation module obtains the second 3D query feature information corresponding to each target to be detected based on the input multiple image feature information and the third 3D query feature information.

[0107] For example, the deformable aggregation module samples image feature information from multiple views, samples image feature information of multiple targets to be detected from multiple views, and dynamically aggregates these image features through learnable weights to interact with the 3D query, so that useful 3D query information of the targets to be detected can be retrieved from multiple views based on the 3D query, and the updated 3D query' (i.e., the second three-dimensional query feature information) is obtained.

[0108] Furthermore, the updated 2D queries output from each level of the 2D decoding layer are used as target 2D queries and input into the 2D detection head to obtain 2D detection results. Similarly, the updated 3D queries output from each level of the 3D decoding layer are used as target 3D queries and input into the 3D detection head to obtain 3D detection results. In this way, by inputting the updated 2D queries from each decoding layer into the 2D detection head for 2D target detection, more accurate 2D detection results can be obtained. Similarly, by inputting the updated 3D queries from each decoding layer into the 3D detection head for 3D target detection, more accurate 3D detection results can be obtained, significantly improving the accuracy of target detection results.

[0109] Alternatively, the updated multiple sets of 2D queries' output by the last-level 2D decoding layer can be used as the target 2D query and input into the 2D detection head to obtain the 2D detection result. Similarly, the updated 3D query' output by the last-level 3D decoding layer can be used as the target 3D query and input into the 3D detection head to obtain the 3D detection result. Because the multiple decoding layers update sequentially, the 2D query' output by the last-level decoding layer is the latest 2D query after multiple updates, and the 3D query' output by the last-level decoding layer is the latest 3D query after multiple updates. Therefore, by performing target detection only based on the multiple sets of 2D queries' and 3D queries' output by the last-level decoding layer, accurate detection results can still be obtained while reducing the number of 2D and 3D detection heads required.

[0110] It should be noted that the number of input 3D queries is the same as the number of final output 3D queries. Therefore, the 3D anchors corresponding to the 3D queries can be reused in multi-level decoding layers.

[0111] Further, in step S400, the two-dimensional decoding layer inputs the target 2D query to the two-dimensional detection head (i.e., the 2D detection head).

[0112] For example, the 2D detection head is placed behind the 2D decoding layer to receive the target 2D query in the packet.

[0113] Further, in step S500, the 3D decoding layer inputs the target 3D query to the 3D detection head (i.e., the 3D detection head).

[0114] For example, the 3D detection head is placed behind the 3D decoding layer to receive the target 3D query.

[0115] Further, in step S600, the 2D detection head obtains the two-dimensional detection results corresponding to multiple targets based on the input target two-dimensional query feature information.

[0116] The two-dimensional detection results include the category of each target to be detected and the two-dimensional detection frame information.

[0117] For example, such as Figure 7 As shown, the 2D detection head receives the grouped 2D query (i.e., target 2D query feature information). The 2D queries corresponding to multiple views from different perspectives share the weight of the same 2D detection head, and obtain the category and 2D detection box information of each target to be detected (i.e., the 2D detection results on multiple views).

[0118] Further, in step S700, the 3D detection head obtains 3D detection results corresponding to multiple targets based on the input target 3D query feature information.

[0119] The 3D detection results include the category of each target to be detected and the 3D detection bounding box information.

[0120] For example, such as Figure 8 As shown, the 3D detection head receives a 3D query (i.e., target 3D query feature information) and outputs the category of the target to be detected, the 3D detection box, and the relevant attribute information of the target to be detected (i.e., the 3D detection result on the bird's eye view (BEV)).

[0121] Furthermore, in the implementation of this application, the target detection model is a detection model obtained by updating the model based on the corresponding loss. The loss includes two-dimensional loss and three-dimensional loss, and the loss is determined based on historical two-dimensional detection results and historical three-dimensional detection results.

[0122] For example, similar to DETR (a method for 2D object detection) and DETR3D (a method for 3D object detection), ground truth matching is first performed on the grouped 2D and 3D detection boxes output by the object detection model based on a loss function. A loss is then applied to the matching results, and backpropagation is performed to update the model gradient. In 2D object detection, each group of 2D detection boxes is equivalent to an independent image, generating a loss. The 2D detection losses from all groups are then summed together to update the gradient.

[0123] Among them, such as Figure 9 As shown, the loss function can perform 2D loss and 3D loss. 2D loss includes 2D class classification loss, 2D bounding box IOU (Intersection over Union) loss, and 2D bounding box regression loss. 3D loss includes 3D class classification loss, 3D bounding box regression loss, and 3D attribute regression loss.

[0124] In this implementation, S400 and S500 can be executed simultaneously, and S600 and S700 can be executed simultaneously.

[0125] In another implementation of this application, such as Figure 10 As shown, the specific principle of the object detection model for object detection is as follows:

[0126] Multi-view images (i.e., images corresponding to different perspectives) are input into the backbone network. The backbone network extracts image features to obtain multi-view image feature information. Then, the 3D queries corresponding to multiple images are input into the joint decoder. Each layer of the joint decoder includes a 2D decoding layer and a 3D decoding layer. The allocation module of the 2D decoding layer obtains grouped 2D queries and a transformation matrix based on the input 3D query (first 3D query feature information). The transformation matrix is ​​sent to the aggregation module, and the grouped 2D queries are sent to their respective deformable attention modules. At least one deformable attention module is included, and the number of deformable attention modules corresponds to the number of 2D query groups. The deformable attention module obtains updated 2D queries based on the input grouped 2D queries and multiple image feature information. The updated 2D queries are sent to the aggregation module. The aggregation module obtains updated 3D queries based on the transformation matrix and the updated 2D queries. The query is sent to the deformable aggregation module of the 3D decoding layer. The deformable aggregation module performs fusion processing based on the multiple input image feature information and the updated 3D query to obtain the updated 3D query (target 3D query feature information).

[0127] Thus, the initialized 3D query is progressively updated by the joint decoder to obtain multiple sets of 2D query' and 3D query'. These updated sets of 2D query' and 3D query' are then processed by their respective detection heads to output 2D and 3D detection results, respectively. Within each decoding layer, the 3D query passes through both the 2D and 3D decoding layers, interacting sequentially with the image features from multiple views. In the 2D decoding layer, the 3D query is distributed to grouped single-view 2D object detections on various views. These 2D queries' of the objects to be detected are then fused in the aggregation module to obtain the 3D query, which is then fed into the subsequent 3D decoding layer for 3D object detection, resulting in the final 3D query'.

[0128] The target detection method provided in this application is actually a joint 2D and 3D multi-view target detection method. Based on a joint decoder, it progressively updates the 3D query using image feature information from multiple views, resulting in multiple updated 2D queries and updated 3D queries. The 2D detection head obtains 2D detection results based on the multiple 2D queries, and the 3D detection head obtains 3D detection results based on the 3D queries. Thus, this joint 2D and 3D target detection method can perform target detection better and obtain more accurate and comprehensive results.

[0129] The target detection method provided in this application can be applied to electronic devices, where a target detection model is deployed to execute the target detection method. Specifically, the electronic device can be a vehicle, a remote calculator, or similar device.

[0130] The target detection method provided in this application can also be used for target detection in virtual scenes.

[0131] This application also discloses a target detection model update method, which determines historical two-dimensional detection results and historical three-dimensional detection results, which are obtained according to the aforementioned target detection method. Based on the historical two-dimensional detection results and historical three-dimensional detection results, the loss corresponding to the target detection model is determined, including two-dimensional loss and three-dimensional loss. The target detection model is updated based on the loss.

[0132] Please see Figure 11 , Figure 11 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device may include: transceiver 121, processor 122, and memory 123.

[0133] The processor 122 executes computer execution instructions stored in the memory, causing the processor 122 to perform the technical solution of the target detection method in the above embodiments. The processor 122 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital data processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0134] The memory 123 is connected to the processor 122 via the system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0135] For example, and not as a limitation, memory 123 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 123 may include removable or non-removable (or fixed) media. Where appropriate, memory 123 may be internal or external to the integrated gateway device. In a particular embodiment, memory 123 is non-volatile solid-state memory. In a particular embodiment, memory 123 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable read-only ROM (PROM), an erasable programmable read-only ROM (EPROM), an electrically erasable programmable read-only ROM (EEPROM), an electrically alterable read-only ROM (EAROM), or flash memory, or a combination of two or more of these. Transceiver 121 can be used to obtain the task to be run and its configuration information.

[0136] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0137] This application also provides a chip for executing instructions, which is used to execute the target detection method described in the above embodiments.

[0138] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on the processor of an electronic device, the processor of the electronic device performs the target detection method described in the above embodiments.

[0139] In some possible implementations, various aspects of the methods provided in this application may also be implemented as a program product, which includes program code. When the program product is run on the processor of an electronic device, the program code is used to cause the processor of the electronic device to perform the steps of the methods according to the various exemplary implementations of this application described above. For example, the electronic device may perform the target detection method described in the embodiments of this application.

[0140] The program product may take the form of any combination of one or more readable media. A readable medium may be a readable data medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CDROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium, and when the at least one processor executes the computer program, it can implement the technical solution of the target detection method in the above embodiments.

[0142] It should be noted that, in addition to the specific embodiments described above, those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. Although the description of this application is presented in conjunction with preferred embodiments, this does not mean that the features of this invention are limited to this implementation. On the contrary, the purpose of describing the invention in conjunction with the implementation is to cover other options or modifications that may be derived from this application. To provide a thorough understanding of this application, many specific details are included in the above description, and this application may also be implemented without using these details. Furthermore, to avoid confusion or obscuring the focus of this application, some specific details will be omitted in the description. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be combined with each other.

[0143] It should be noted that in this specification, similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0144] It should be noted that the terms "first," "second," "third," and "fourth" are used only for distinguishing descriptions and should not be interpreted as indicating or implying relative importance.

[0145] It should be noted that some structural or methodological features may be shown in the accompanying drawings in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, including structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0146] Although this application has been illustrated and described with reference to certain preferred embodiments, those skilled in the art should understand that the above description is a further detailed explanation of the application in conjunction with specific implementations, and should not be construed as limiting the specific implementation of the application to these descriptions. Those skilled in the art can make various changes in form and detail, including some simple deductions or substitutions, without departing from the spirit and scope of this application.

Claims

1. A target detection method, characterized in that, The method is applied to an object detection model, which includes a backbone network, a joint decoder, and a detection head. The joint decoder includes multiple decoding layers, and the detection head includes a two-dimensional detection head and a three-dimensional detection head. The backbone network obtains multiple image feature information based on multiple images corresponding to different viewpoints, and inputs the multiple image feature information into the decoding layers at each level respectively; Each decoding layer obtains target 2D query feature information and target 3D query feature information corresponding to each target based on the 3D feature information of multiple targets to be detected corresponding to the multiple images and the feature information of the multiple images. The target 2D query feature information is input to the 2D detection head, and the target 3D query feature information is input to the 3D detection head. The target 2D query feature information is the target 2D query of the target to be detected, and the target 3D query feature information is the target 3D query of the target to be detected. The two-dimensional detection head obtains the two-dimensional detection results corresponding to the multiple targets to be detected based on the target two-dimensional query feature information; The 3D detection head obtains the 3D detection results corresponding to the multiple targets to be detected based on the target 3D query feature information.

2. The target detection method according to claim 1, characterized in that, The three-dimensional feature information includes three-dimensional query feature information and three-dimensional attribute feature information. Then, each decoding layer at each level obtains target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target based on the three-dimensional feature information of the multiple targets to be detected corresponding to the multiple images and the multiple image feature information, including: Each decoding layer at each level obtains multiple sets of two-dimensional query feature information corresponding to each target to be detected based on the multiple image feature information, the corresponding first three-dimensional query feature information, and the first three-dimensional attribute feature information. It then obtains second three-dimensional query feature information based on these multiple sets of two-dimensional query feature information and inputs this second three-dimensional query feature information as the first three-dimensional query feature information corresponding to the next level decoding layer. This allows the next level decoding layer to perform corresponding processing until the last level decoding layer completes its processing, resulting in target two-dimensional query feature information and target three-dimensional query feature information corresponding to each target to be detected. The target three-dimensional query feature information includes the second three-dimensional query feature information corresponding to each target to be detected obtained by each level decoding layer, and the target two-dimensional query feature information includes the multiple sets of two-dimensional query feature information corresponding to each target to be detected obtained by each level decoding layer. Alternatively, the target three-dimensional query feature information includes the second three-dimensional query feature information corresponding to each target to be detected obtained by the last level decoding layer, and the target two-dimensional query feature information includes the multiple sets of two-dimensional query feature information corresponding to each target to be detected obtained by the last level decoding layer.

3. The target detection method according to claim 2, characterized in that, Each of the decoding layers includes a two-dimensional decoding layer and a three-dimensional decoding layer. The backbone network inputs the multiple image feature information to the decoding layers at each level, including: The backbone network inputs the multiple image feature information to the two-dimensional decoding layer and the three-dimensional decoding layer included in each level of the decoding layer; The method further includes obtaining the multiple sets of two-dimensional query feature information and the second three-dimensional query feature information corresponding to each of the targets to be detected by the decoding layers at each level in the following manner: The two-dimensional decoding layer obtains the multiple sets of two-dimensional query feature information corresponding to each of the targets to be detected based on the multiple image feature information, the first three-dimensional query feature information, and the first three-dimensional attribute feature information, and obtains the third three-dimensional query feature information corresponding to each of the targets to be detected based on the multiple sets of two-dimensional query feature information, and inputs the third three-dimensional query feature information to the corresponding three-dimensional decoding layer; The three-dimensional decoding layer obtains the second three-dimensional query feature information corresponding to each of the targets to be detected based on the multiple image feature information and the third three-dimensional query feature information.

4. The target detection method according to claim 3, characterized in that, The two-dimensional decoding layer includes a distribution module, a deformable attention module, and an aggregation module. The method further includes the two-dimensional decoding layer obtaining the third three-dimensional query feature information in the following manner: The distribution module initializes the first three-dimensional query feature information to obtain the fourth three-dimensional query feature information, and performs clustering processing on the first three-dimensional attribute feature information to obtain the second three-dimensional attribute feature information. The distribution module determines the camera parameters of the camera used to capture the multiple images, obtains the transformation matrix corresponding to the multiple targets to be detected based on the camera parameters and the second three-dimensional attribute feature information, and inputs the transformation matrix to the aggregation module. The distribution module obtains multiple two-dimensional query feature information corresponding to the multiple targets to be detected based on the fourth three-dimensional query feature information and the transformation matrix, groups the multiple two-dimensional query feature information to obtain the multiple groups of two-dimensional query feature information corresponding to the multiple targets to be detected, and inputs the multiple groups of two-dimensional query feature information into the deformable attention module. The deformable attention module obtains the updated set of two-dimensional query feature information corresponding to the multiple targets to be detected based on the multiple sets of two-dimensional query feature information and the multiple image feature information, and inputs the updated set of two-dimensional query feature information into the aggregation module; The aggregation module obtains the third three-dimensional query feature information corresponding to each of the targets to be detected based on the updated multiple sets of two-dimensional query feature information and the transformation matrix.

5. The target detection method according to claim 4, characterized in that, The three-dimensional decoding layer includes a deformable aggregation module, and the method further includes that the three-dimensional decoding layer obtains the second three-dimensional query feature information in the following manner: The deformable aggregation module obtains the second three-dimensional query feature information corresponding to each of the targets to be detected based on the multiple image feature information and the third three-dimensional query feature information.

6. The target detection method according to claim 5, characterized in that, The method further includes the distribution module obtaining the multiple sets of two-dimensional query feature information in the following manner: in, For the multiple sets of two-dimensional query feature information, for The transformation matrix, where M is the number of two-dimensional query feature information and N is the number of first three-dimensional query feature information. This refers to the first three-dimensional query feature information.

7. The target detection method according to claim 6, characterized in that, The method further includes the aggregation module obtaining the third three-dimensional query feature information in the following manner: in, The third three-dimensional query feature information, The transformation matrix is... This refers to the updated set of two-dimensional query feature information.

8. The target detection method according to claim 7, characterized in that, The two-dimensional detection result includes the category of each target to be detected and the two-dimensional detection box information, and the three-dimensional detection result includes the category of each target to be detected and the three-dimensional detection box information.

9. The target detection method according to any one of claims 1-8, characterized in that, The target detection model is a detection model obtained by updating the model based on the corresponding loss. The loss includes two-dimensional loss and three-dimensional loss, and the loss is determined based on historical two-dimensional detection results and historical three-dimensional detection results.

10. An electronic device, characterized in that, The electronic device includes a target detection model, which is used to implement the target detection method as described in any one of claims 1-9.

Images