A three-dimensional target detection method, device and equipment

Abstract

The present disclosure belongs to the field of computer vision, and particularly relates to a three-dimensional target detection method, device and equipment. The method comprises the following steps: acquiring multiple images of a target scene from different perspectives, extracting three-dimensional position perception features of the target scene from the images, and sampling query points according to a preset rule; acquiring radar point features of the target scene, expanding the query points based on position information of significant radar point features, performing target query in a set composed of the three-dimensional position perception features based on position features of the query points, and performing a target detection task based on obtained query features, wherein the target detection task includes but is not limited to target classification and positioning tasks; and correcting the query features based on radar point features in a neighborhood of the query result, and performing target speed prediction based on the corrected query features.

Description

Technical Field

[0001] This disclosure belongs to the field of computer vision, and specifically relates to a three-dimensional target detection method, apparatus and device. Background Technology

[0002] Three-dimensional target detection refers to target detection tasks such as locating and identifying objects in three-dimensional space using sensor data. These target detection tasks often also include determining the velocity of the object.

[0003] 3D object detection often enhances performance by leveraging complementary features from data from various sensor types. For example, RGB cameras, which acquire 2D images, are often paired with distance sensors (such as LiDAR) because they can provide rich semantic information, to improve the accuracy of 3D object detection.

[0004] The integration of cameras and radar is also known as radar-visual fusion. Current radar-visual fusion frameworks place the data of each modality in an equal position and process the data input with a symmetrical encoder (therefore, the modality is unknown during radar-visual fusion). This processing method makes it difficult to maximize the physical characteristics of each modality, resulting in poor performance of 3D target detection methods based on radar-visual fusion. Summary of the Invention

[0005] This disclosure proposes a 3D target detection scheme based on a modality-aware radar-view fusion framework, which avoids the problem of poor performance of existing 3D target detection schemes due to their modality-agnostic radar-view fusion framework.

[0006] A first aspect of this disclosure provides a three-dimensional target detection method, comprising:

[0007] Acquire multiple images of the target scene from different perspectives, extract three-dimensional position perception features of the target scene from the images, and sample query points according to preset rules;

[0008] The radar point features of the target scene are obtained, the query point is expanded based on the location information of the significant radar point features, the target is queried in the set composed of the three-dimensional location perception features based on the location features of the query point, and the target detection task is executed based on the obtained query features. The target detection task includes, but is not limited to, target classification and localization tasks.

[0009] The query features are corrected based on the radar point features in the neighborhood of the query results, and the target velocity is predicted based on the corrected query features.

[0010] In some embodiments, extracting the three-dimensional position-aware features of the target scene from the image includes:

[0011] For each of the images, extract the image features of the image;

[0012] Based on the perspectives of multiple images, a transformation matrix is ​​determined between the images. For each pixel of each image, the three-dimensional position corresponding to the pixel is determined and encoded based on the transformation matrix to obtain the three-dimensional position-aware feature corresponding to the image.

[0013] The three-dimensional position-aware features corresponding to the target scene are composed of the three-dimensional position-aware features corresponding to all the images of the target scene.

[0014] In some embodiments, expanding the query point based on the location information of significant radar point features includes:

[0015] The query point obtained by sampling the three-dimensional position-aware features according to a preset rule is used as the first query point;

[0016] Based on preset criteria, significant radar points are selected from the radar point features, and a second query point is determined based on the location information of the radar features of the significant radar points.

[0017] The query point is composed of the second query point and the first query point.

[0018] In some embodiments, after the location information based on significant radar point features is used to expand the query point, the method further includes:

[0019] Extract the location features of the first query point to form the first query;

[0020] The radar features of the second query point are used to obtain a second query based on location encoding, and the first query and the second query are combined to form a hybrid query.

[0021] The radar features in the hybrid query are enhanced based on the cross-attention mechanism.

[0022] In some embodiments, prior to enhancing the radar features in the hybrid query based on the cross-attention mechanism, the method further includes:

[0023] The local feature representation of each radar feature is enhanced through an inner product self-attention mechanism.

[0024] In some embodiments, performing a target query based on the location features of the query point within a set of three-dimensional location-aware features includes:

[0025] The three-dimensional position-aware features are planarized and used as the key matrix and value matrix, respectively;

[0026] The query matrix is ​​composed of the aforementioned mixed queries;

[0027] The key matrix, the value matrix, and the query matrix are input into the decoder of the transformer architecture for target decoding, and the query features are output.

[0028] In some embodiments, correcting the query features based on radar point features in the neighborhood of the query result includes:

[0029] For each target query result output from the decoder, aggregated features of radar point features within multiple distance scales in the neighborhood are obtained;

[0030] The aggregated features at each distance scale are weighted and aggregated to obtain radar aggregated features;

[0031] The radar aggregation feature is concatenated with the query feature to form the modified query feature. In some embodiments, the aggregation feature for obtaining radar features within multiple distance scales in the neighborhood includes:

[0032] Using the location of the target query result as the query center, and using a preset distance scale as the search radius, a sphere search is performed on the radar point features, and the searched radar point features are aggregated into aggregated features corresponding to the distance scale;

[0033] Adjust the search radius of the ball search to a second preset scale, obtain the aggregated features corresponding to the second preset scale, and repeat this step a preset number of times.

[0034] A second aspect of this disclosure provides a three-dimensional target detection device, comprising:

[0035] The acquisition module is used to acquire multiple images of the target scene from different perspectives, extract the three-dimensional position perception features of the target scene from the images, and sample query points according to preset rules;

[0036] The first fusion module is used to acquire radar point features of the target scene, expand the query point based on the location information of the significant radar point features, perform target query in a set composed of the three-dimensional location perception features based on the location features of the query point, and perform target detection task based on the obtained query features, wherein the target detection task includes, but is not limited to, target classification and localization tasks.

[0037] The second fusion module is used to correct the query features based on radar point features within multiple distance scales in the neighborhood of the query result and to predict the target velocity based on the corrected query features.

[0038] A third aspect of this disclosure provides a three-dimensional target detection device, including a memory and a processor:

[0039] The memory is used to store computer programs;

[0040] The processor is configured to implement the method according to the first aspect of this disclosure when executing the computer program.

[0041] In summary, the three-dimensional target detection methods, apparatuses, and devices provided in the embodiments of this disclosure integrate spatial and velocity information from radar data into a camera-based main three-dimensional target detector, thereby constructing a modality-aware radar-visual fusion model that uses the weak modality (radar) to assist the strong modality (camera). Therefore, it is possible to deeply explore the contribution of modal characteristics provided by radar-visual data to specific tasks. At the same time, based on a hierarchical fusion method with query as the interaction, deep interaction of features is achieved by mapping the characteristic information of radar to the target query in multiple layers, maximizing the preservation of the physical characteristics of the modality, thereby significantly improving the performance of three-dimensional target detection. Attached Figure Description

[0042] The features and advantages of this disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the scope of this disclosure in any way.

[0043] Figure 1 This is a schematic diagram of a computer system to which this disclosure applies;

[0044] Figure 2 This is a flowchart illustrating a three-dimensional target detection method according to some embodiments of the present disclosure;

[0045] Figure 3 This is the radar-visual fusion framework used in some embodiments of this disclosure;

[0046] Figure 4 This is a comparison between the traditional radar-visual fusion paradigm and the radar-visual fusion paradigm used in some embodiments of this disclosure;

[0047] Figure 5 The mAP and mAVE results for camera-only methods CMT-C and RaCFusion under different lighting and weather conditions;

[0048] Figure 6 The results are mAP and mAVE values ​​for camera methods CMT-C and RaCFusion at different target velocities.

[0049] Figure 7 The results are mAP and mAVE values ​​for camera methods CMT-C and RaCFusion at different detection distances.

[0050] Figure 8 This is a schematic diagram of a three-dimensional target detection device according to some embodiments of the present disclosure;

[0051] Figure 9This is a schematic diagram of a three-dimensional target detection device according to an embodiment of the present disclosure. Detailed Implementation

[0052] In the following detailed description, numerous specific details of this disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these details. It should be understood that the terms “system,” “apparatus,” “unit,” and / or “module” used in this disclosure are a method of distinguishing different parts, elements, sections, or components at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if they can achieve the same purpose.

[0053] It should be understood that when a device, unit, or module is referred to as being "on," "connected to," or "coupled to" another device, unit, or module, it may be directly connected to or coupled to, or communicate with, other devices, units, or modules, or there may be intermediate devices, units, or modules present, unless the context explicitly indicates otherwise. For example, the term "and / or" as used in this disclosure includes any one and all combinations of one or more of the associated listed items.

[0054] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As shown in this specification and claims, unless the context clearly indicates otherwise, words such as "a," "an," "an," and / or "the" do not specifically refer to the singular and may include the plural. Generally speaking, the terms "comprising" and "including" only indicate the inclusion of explicitly identified features, integrals, steps, operations, elements, and / or components, and such expressions do not constitute an exclusive list, in which other features, integrals, steps, operations, elements, and / or components may also be included.

[0055] Referring to the following description and accompanying drawings, these and other features and characteristics, operating methods, functions of related structural elements, combinations of parts, and economics of manufacture of this disclosure can be better understood, wherein the description and drawings form part of the specification. However, it is clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this disclosure. It is understood that the drawings are not drawn to scale.

[0056] Various structural diagrams are used in this disclosure to illustrate various variations of embodiments according to this disclosure. It should be understood that the preceding or following structures are not intended to limit this disclosure. The scope of protection of this disclosure is defined by the claims.

[0057] Figure 1 This is a schematic diagram of a computer system applicable to this disclosure. Figure 1The system shown includes a 3D target detection server connected to an image sensor and a distance sensor. The 3D target detection server acquires multiple images of the target scene from different perspectives from the image sensor, acquires spatial position information and velocity information of the target scene from the distance sensor, and performs target detection on the target scene based on the images, the distance information and the velocity information.

[0058] The image sensor can be a camera array to acquire multiple images of the target scene from different perspectives, or it can be a single camera device that simultaneously acquires multiple images of the target scene from different perspectives. The images can provide semantic information about the target scene. The distance sensor can be a lidar or millimeter-wave radar. In addition to providing spatial location information of the target scene, radar signals can also utilize the Doppler effect to determine the relative radial velocity of objects. The 3D target detection server can be a single machine, a cluster, or a distributed server.

[0059] Figure 2 This is a flowchart illustrating a three-dimensional target detection method according to some embodiments of the present disclosure. In some embodiments, the three-dimensional target detection method is executed by a three-dimensional target detection server, and the three-dimensional target detection method includes the following steps:

[0060] S210, acquire multiple images of the target scene from different perspectives, extract the three-dimensional position perception features of the target scene from the images, and sample query points according to preset rules.

[0061] Specifically, in a typical autonomous vehicle setup, N cameras are used to capture multiple images from different perspectives. Figure 3 This is the radar-visual fusion framework used in some embodiments of this disclosure. For example... Figure 3 As shown, ResNet or VoVNet and FPN (Image Backbone Network) are first used to extract image features. For each image, each pixel can be formulated as a series of points in the camera's frustum coordinates. Next, the transformation matrix between images is determined, and the 3D position corresponding to each pixel is calculated for each image using the transformation matrix, obtaining the 3D position-aware features corresponding to the image. The 3D position-aware features corresponding to all viewpoints constitute the 3D position-aware features corresponding to the target scene. Thus, a camera-based master 3D detector is constructed.

[0062] At the same time, a preset number of sampling points are determined in the three-dimensional position perception features according to preset rules, and the position information of the sampling points is extracted as the initial query.

[0063] S220, acquire radar point features of the target scene, expand the query point based on the location information of significant radar point features, perform target query in the set composed of the three-dimensional location perception features based on the location features of the query point, and perform target detection task based on the obtained query features, wherein the target detection task includes, but is not limited to, target classification and localization tasks.

[0064] Specifically, the radar point features of the target scene are obtained. A set abstraction (SA) layer is used to extract the radar point features and generate a representative set of radar point features.

[0065] Then, representative radar point features are processed by a position encoder to obtain a radar query. Q Radar =Ψ(P′) R ), where Ψ is the MLP layer, It is a radar characteristic, N R This refers to the number of radar queries, and the obtained radar query Q. Radar This makes it easier for the decoder to detect target objects. At the same time, because some objects may be undetectable due to limitations of the radar sensor, the initial query Q0 is retained. By combining the two types of object queries, a hybrid query Q is obtained. hybrid formal:

[0066]

[0067] Q hybrid ={Q0; Q Radar}

[0068] Among them, Q Radar This is a radar query, Q0 is the initial query, Q hybrid It is a mixed query, N R N is the number of radar queries, N0 is the number of initial queries, and C is the vector dimension of the features.

[0069] Q Radar The feature space of Q0 in Q hybrid Generating features in different domains may hinder the effective location features of the query. To address this issue, this disclosure utilizes radar-assisted feature fusion, fusing the mixed query with radar features through attention. Specifically, firstly, the local feature representation of each radar feature is optionally enhanced through an inner product self-attention mechanism. Next, a multifunctional cross-attention mechanism is employed to flexibly adjust the mixed query by enhancing the radar features. Finally, the two types of queries form a query matrix, while the enhanced radar features form a key-value matrix. Formally:

[0070]

[0071]

[0072] in, These are the extracted radar point features; SelfAttn is a self-attention mechanism. It is the enhanced radar point feature, CrossAttn is the cross-attention mechanism, Q′ hybrid This is the updated hybrid radar query.

[0073] like Figure 3 As shown, some embodiments of this disclosure flatten the three-dimensional position-aware features corresponding to the target scene into tokens, which serve as K and V of the decoder in the tansform structure, and fuse the hybrid query Q′ with radar features. hybrid Q, acting as the decoder, performs target queries and executes target detection tasks based on the obtained query features. These target detection tasks generally include target classification and localization tasks.

[0074] S230, the query features are corrected based on the radar point features in the neighborhood of the query results, and the target velocity is predicted based on the corrected query features.

[0075] First, speed-driven soft association is performed. Specifically, for the i-th target query result output from the decoder, its position is considered the query center, and soft association is performed with radar point features from the radar backbone network. Then, a ball query is applied, using the PointNet module to aggregate and associate object-centric features from the associated radar point features. Similarly, neighborhood-aware features can be obtained by appropriately expanding the search radius of the ball query. This provides the feature for the i-th query. In this way, we associate each target query with radar point features using multiple distance scales (corresponding to different search radii for different ball queries) to extract features from the vicinity of the object and extended local regions.

[0076] Then, multi-scale feature aggregation is performed. Specifically, after obtaining aggregated features from each scale, they are concatenated together, and then the concatenated features are input into an aggregation network. The aggregation operation adaptively adjusts to the desired scale. and Weighting is applied. The calculation formula is as follows:

[0077] Where ω1 and ω2 are the weights output from the aggregation network. It is an object-centric feature. Neighborhood-aware features, η i These are query features resulting from multi-scale aggregation.

[0078] The output η (the updated query feature mentioned above) of this module is compared with the original query feature Q. f Connect them together for velocity prediction tasks (which are decoupled from regression tasks).

[0079]

[0080] Where [·] represents concatenation on the channel, FFN vel It is the velocity regression feedforward network decoupled from the regression task, Q f These are the original query features. This refers to the prediction speed corresponding to the target query.

[0081] Traditional radar-visual fusion methods utilize a symmetrical architecture for image and radar inputs (modal agnostic fusion), treating the fusion mechanism as a "black box," such as... Figure 4 As shown in (a). This modality-agnostic approach fails to fully utilize the unique advantages of each modality in contributing to accurate 3D object detection, thus hindering further improvements. In view of this, this disclosure proposes a modality-agnostic fusion paradigm—a radar-visual fusion method. The roles of RGB image and radar data in 3D object detection can be elucidated as follows: the image acts as an intelligent leader, understanding semantic information; millimeter-wave radar data acts as a professional advisor, further providing spatial prioritization and velocity cues. For example... Figure 4 As shown in (b), the radar-visual fusion method disclosed herein fuses the most valuable spatial information and velocity attributes from radar data in a hierarchical manner. Specifically, in the first layer, the target query in the image branch is enhanced with spatial information obtained from the radar data. In the second layer, the query features for velocity prediction are further enhanced with radar data, thereby reducing errors in velocity estimation. Therefore, the radar-visual fusion method disclosed herein can deeply explore the contribution of modal characteristics provided by radar-visual data to specific tasks, while realizing deep interaction of features, maximizing the preservation of the physical properties of the modalities, thereby significantly improving the performance of 3D target detection.

[0082] The RaCFusion method presented in this disclosure was tested on the NuScenes dataset. NuScenes is a large-scale and challenging dataset for 3D detection. It provides camera images, LiDAR points, and millimeter-wave radar points. The main evaluation metrics used in nuScenes are mAP and nuScenes Detection Score (NDS). Additionally, there is mAVE, representing absolute velocity error in m / s; a decrease in mAVE indicates a more accurate velocity estimate. NDS is a combined metric of mAP and other attribute measures. Testing was conducted on a server equipped with a 48-core Intel(R) Xeon(R) Gold 6226 CPU, 376GB of memory, and eight 32GB NVIDIA Tesla V100 GPUs.

[0083] Test results:

[0084] 1. Robustness to light and weather conditions:

[0085] We categorized the scenarios in the validation set into Sunny / Rainy / Day / Night by searching for the keywords "rain" and "night" in the description of each scenario. For example... Figure 5 As shown, RaCFusion significantly improves performance under all challenging conditions by integrating camera images and radar points through a modality-aware fusion strategy. Notably, RaCFusion improves mAP by 5.1% in rainy scenarios and mAVE by 42.9% in nighttime scenarios, outperforming baseline methods that use only cameras.

[0086] 2. Robustness to different object velocities:

[0087] Objects with a speed greater than 0.5 m / s are considered to be moving objects, and 5 m / s and 10 m / s are used as the dividing lines between low and medium motion states. Figure 6 The results show the mAP and mAVE of the camera-only detector and RaCFusion for objects moving at different speeds. It is evident that RaCFusion can significantly improve detection performance and speed estimation when target speed changes. Experimental results demonstrate that RaCFusion achieves a more significant performance improvement by fully utilizing radar information, validating its ability to effectively handle complex driving scenarios.

[0088] 3. Robustness to detection distance:

[0089] Figure 7Detection accuracy and velocity estimation are presented across different distance ranges. Within the 20-meter range, RaCFusion shows a more significant improvement. This is primarily due to the smaller positional error of the radar point for objects within this range. Beyond 20 meters, our RaCFusion consistently outperforms the baseline using only the camera, particularly in achieving a stable reduction in velocity error (-25.9% to -17.7%). The results demonstrate that our RaCFusion fusion framework is less sensitive to distance and allows for detection at greater radar ranges.

[0090] The experimental results above show that RaCFusion is a robust, stable, and accurate 3D detection method based on radar camera fusion.

[0091] Figure 8 This is a schematic diagram of a three-dimensional target detection device according to some embodiments of the present disclosure. Figure 8 As shown, the 3D target detection device 800 includes an acquisition module 810, a first fusion module 820, and a second fusion module 830. In some embodiments of this disclosure, the 3D target detection function can be executed by a 3D target detection server. Wherein:

[0092] The acquisition module 810 is used to acquire multiple images of the target scene from different perspectives, extract the three-dimensional position perception features of the target scene from the images, and sample query points according to preset rules.

[0093] The first fusion module 820 is used to acquire radar point features of the target scene, expand the query point based on the location information of the significant radar point features, perform target query in the set composed of the three-dimensional location perception features based on the location features of the query point, and perform target detection task based on the obtained query features, wherein the target detection task includes, but is not limited to, target classification and localization tasks.

[0094] The second fusion module 830 is used to correct the query features based on radar point features within multiple distance scales in the neighborhood of the query result and to predict the target velocity based on the corrected query features.

[0095] One embodiment of this disclosure provides a three-dimensional target detection device. For example... Figure 9 As shown, the three-dimensional target detection device 900 includes a memory 920 and a processor 910. The memory 920 is used to store computer programs; the processor 910 is used to implement, when the computer program is executed, [the following is unclear and requires further context: "to implement"]. Figure 2 The method described in S210-S230.

[0096] In summary, the three-dimensional target detection methods, apparatuses, and devices provided in the embodiments of this disclosure integrate spatial and velocity information from radar data into a camera-based main three-dimensional target detector, thereby constructing a modality-aware radar-visual fusion model that uses the weak modality (radar) to assist the strong modality (camera). Therefore, it is possible to deeply explore the contribution of modal characteristics provided by radar-visual data to specific tasks. At the same time, based on a hierarchical fusion method with query as the interaction, deep interaction of features is achieved by mapping the characteristic information of radar to the target query in multiple layers, maximizing the preservation of the physical characteristics of the modality, thereby significantly improving the performance of three-dimensional target detection.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding descriptions in the foregoing device embodiments, and will not be repeated here.

[0098] Although the subject matter described herein is provided in the general context of execution on a computer system in conjunction with an operating system and applications, those skilled in the art will recognize that other implementations can also be executed in conjunction with other types of program modules. Generally, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. Those skilled in the art will understand that the subject matter described herein can be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframes, etc., and can also be used in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may reside on both local and remote memory storage devices.

[0099] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0100] It should be understood that the specific embodiments described above are merely illustrative or explanatory of the principles of this disclosure and do not constitute a limitation thereof. Therefore, any modifications, equivalent substitutions, improvements, etc., made without departing from the spirit and scope of this disclosure should be included within the protection scope of this disclosure. Furthermore, the appended claims are intended to cover all variations and modifications falling within the scope and boundaries of the appended claims, or equivalent forms of such scope and boundaries.

Claims

1. A three-dimensional target detection method, characterized in that, include: Acquire multiple images of the target scene from different perspectives, extract three-dimensional position perception features of the target scene from the images, and sample query points according to preset rules; The radar point features of the target scene are obtained, the query point is expanded based on the location information of the significant radar point features, the target is queried in the set composed of the three-dimensional location perception features based on the location features of the query point, and the target detection task is executed based on the obtained query features. The target detection task includes, but is not limited to, target classification and localization tasks. The query features are corrected based on the radar point features in the neighborhood of the query results, and the target velocity is predicted based on the corrected query features. The expansion of the query point based on the location information of significant radar point features includes: The query point obtained by sampling the three-dimensional position-aware features according to a preset rule is used as the first query point; Based on preset criteria, significant radar points are selected from the radar point features, and a second query point is determined based on the location information of the radar features of the significant radar points. The query point is composed of the second query point and the first query point; After expanding the query point with location information based on significant radar point features, it also includes: Extract the location features of the first query point to form the first query; The radar features of the second query point are used to obtain a second query based on location encoding, and the first query and the second query are combined to form a hybrid query. The radar features in the hybrid query are enhanced based on the cross-attention mechanism; Before enhancing the radar features in the hybrid query based on the cross-attention mechanism, the following is also included: The local feature representation of each radar feature is enhanced through an inner product self-attention mechanism; The target query based on the location features of the query point in the set of three-dimensional location-aware features includes: The three-dimensional position-aware features are planarized and used as the key matrix and value matrix, respectively; The query matrix is ​​composed of the aforementioned mixed queries; The key matrix, the value matrix, and the query matrix are input into the decoder of the transformer architecture for target decoding, and the query features are output.

2. The method according to claim 1, characterized in that, The extraction of three-dimensional position-aware features of the target scene from the image includes: For each of the images, extract the image features of the image; Based on the perspectives of multiple images, a transformation matrix is ​​determined between the images. For each pixel of each image, the three-dimensional position corresponding to the pixel is determined and encoded based on the transformation matrix to obtain the three-dimensional position-aware feature corresponding to the image. The three-dimensional position-aware features corresponding to the target scene are composed of the three-dimensional position-aware features corresponding to all the images of the target scene.

3. The method according to claim 1, characterized in that, The step of correcting the query features based on the radar point features in the neighborhood of the query result includes: For each target query result output by the decoder, aggregated features of radar point features within multiple distance scales in the neighborhood are obtained; The aggregated features at each distance scale are weighted and aggregated to obtain radar aggregated features; The radar aggregation feature is connected with the query feature to form the modified query feature.

4. The method according to claim 3, characterized in that, The aggregated features for acquiring radar features within multiple range scales in the neighborhood include: Using the location of the target query result as the query center, and using a preset distance scale as the search radius, a sphere search is performed on the radar point features, and the searched radar point features are aggregated into aggregated features corresponding to the distance scale; Adjust the search radius of the ball search to a second preset scale, obtain the aggregated features corresponding to the second preset scale, and repeat this step a preset number of times.

5. A three-dimensional target detection device, characterized in that, include: The acquisition module is used to acquire multiple images of the target scene from different perspectives, extract the three-dimensional position perception features of the target scene from the images, and sample query points according to preset rules; The first fusion module is used to acquire radar point features of the target scene, expand the query point based on the location information of the significant radar point features, perform target query in a set composed of the three-dimensional location perception features based on the location features of the query point, and perform target detection task based on the obtained query features, wherein the target detection task includes, but is not limited to, target classification and localization tasks. The second fusion module is used to correct the query features based on radar point features within multiple distance scales in the neighborhood of the query result and to predict the target velocity based on the corrected query features. The expansion of the query point based on the location information of significant radar point features includes: The query point obtained by sampling the three-dimensional position-aware features according to a preset rule is used as the first query point; Based on preset criteria, significant radar points are selected from the radar point features, and a second query point is determined based on the location information of the radar features of the significant radar points. The query point is composed of the second query point and the first query point; After expanding the query point with location information based on significant radar point features, it also includes: Extract the location features of the first query point to form the first query; The radar features of the second query point are used to obtain a second query based on location encoding, and the first query and the second query are combined to form a hybrid query. The radar features in the hybrid query are enhanced based on the cross-attention mechanism; Before enhancing the radar features in the hybrid query based on the cross-attention mechanism, the following is also included: The local feature representation of each radar feature is enhanced through an inner product self-attention mechanism; The target query based on the location features of the query point in the set of three-dimensional location-aware features includes: The three-dimensional position-aware features are planarized and used as the key matrix and value matrix, respectively; The query matrix is ​​composed of the aforementioned mixed queries; The key matrix, the value matrix, and the query matrix are input into the decoder of the transformer architecture for target decoding, and the query features are output.

6. A three-dimensional target detection device, characterized in that, Including memory and processor: The memory is used to store computer programs; The processor is configured to, when executing the computer program, implement the method according to any one of claims 1-4.

Images