Target detection method and device, electronic equipment and machine readable storage medium

Abstract

The application provides a target detection method and device, electronic equipment and machine readable storage medium. The method comprises: determining a candidate target frame according to a target query; performing radar point sampling according to the candidate target frame to obtain to-be-processed radar points, and performing feature extraction on the to-be-processed radar points to obtain to-be-fused radar features; performing image feature sampling according to the candidate target frame to obtain to-be-fused image features; for the to-be-fused radar features and the to-be-fused image features that are associated with each other, using a cross attention mechanism to perform feature deep fusion on the to-be-fused radar features and the to-be-fused image features to obtain deep fusion features; and performing target detection on the deep fusion features. The method can optimize the target detection performance.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a target detection method, apparatus, electronic device, and machine-readable storage medium. Background Technology

[0002] In autonomous driving, millimeter-wave radar and cameras are widely used. Cameras possess rich texture information, enabling accurate acquisition of target size and category information; however, their ranging and velocity measurement performance is generally average, and they are susceptible to effects such as lighting, rain, and fog. Millimeter-wave radar, on the other hand, offers excellent ranging and velocity measurement performance and strong robustness, but it has limitations in angle measurement and static obstacle recognition. Therefore, sensor fusion can be used to leverage the strengths of both sensors and compensate for their weaknesses, thereby improving target detection performance.

[0003] In recent years, several RV (Radar and Vision, millimeter-wave radar and video) fusion 3D detection schemes have emerged, extending the multi-view image 3D detection model into a multi-modal fusion 3D target detection framework. Their main implementation schemes include:

[0004] Reference points are generated using a Query (target query in Transformer). First, the nearest radar point cloud is sampled around the reference point, and features are extracted using a fully connected network. Second, the reference points are projected onto each camera, and image feature points are extracted using bilinear interpolation. Finally, the image features and the matched radar features are concatted and fused together before being fed into the detection head to predict the 3D target.

[0005] Practice has shown that using the concat method to fuse image features and radar features results in overlapping feature fusion performance, which affects the performance of 3D target detection. Summary of the Invention

[0006] In view of this, this application provides a target detection method, apparatus, electronic device, and machine-readable storage medium to optimize target detection performance.

[0007] Specifically, this application is implemented through the following technical solution:

[0008] According to a first aspect of the embodiments of this application, a target detection method is provided, comprising:

[0009] Candidate target boxes are determined based on the target query;

[0010] Radar point sampling is performed based on the candidate target boxes to obtain radar points to be processed, and feature extraction is performed on the radar points to be processed to obtain radar features to be fused; and,

[0011] Image features are sampled based on the candidate target boxes to obtain the features of the image to be fused;

[0012] For interrelated radar features and image features to be fused, a cross-attention mechanism is used to perform deep feature fusion to obtain deep fused features; where the interrelated radar features and image features to be fused are those associated with the same candidate target box.

[0013] Target detection is performed on the deep fusion features.

[0014] According to a second aspect of the embodiments of this application, a target detection apparatus is provided, comprising:

[0015] The determination unit is used to determine candidate target boxes based on the target query.

[0016] The radar feature acquisition unit is used to sample radar points based on the candidate target box to obtain radar points to be processed, and to extract features from the radar points to be processed to obtain radar features to be fused.

[0017] An image feature acquisition unit is used to sample image features based on the candidate target boxes to obtain the features of the image to be fused.

[0018] The feature fusion unit is used to perform deep feature fusion on the interrelated radar features and image features to be fused using a cross-attention mechanism to obtain deep fused features; wherein, the interrelated radar features and image features to be fused are radar features and image features to be fused that are associated with the same candidate target box.

[0019] The target detection unit is used to perform target detection on the deep fusion features.

[0020] According to a third aspect of the present application, an electronic device is provided, including a processor and a memory, the memory storing machine-executable instructions executable by the processor, the processor being configured to execute the machine-executable instructions to implement the method provided in the first aspect.

[0021] According to a fourth aspect of the embodiments of this application, a machine-readable storage medium is provided, wherein machine-executable instructions are stored therein, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0022] The technical solution provided in this application can bring at least the following beneficial effects:

[0023] Once candidate target boxes are determined based on the target query, on the one hand, radar point sampling is performed based on the candidate target boxes to obtain radar points to be processed, and features are extracted from these radar points to obtain radar features to be fused; on the other hand, image feature sampling is performed based on the candidate target boxes to obtain image features to be fused. For interrelated radar features and image features to be fused, a cross-attention mechanism is used to perform deep feature fusion to obtain deep fused features, and target detection is performed on the deep fused features. By using the cross-attention mechanism to perform deep adjustment and fusion of radar features and image features, the data characteristics of sensor data from different modalities are fully considered, the feature fusion performance is optimized, higher quality deep fused features are obtained, and thus, the target detection performance is optimized. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating an exemplary embodiment of the target detection method of this application;

[0025] Figure 2 This is a schematic diagram of the framework of a fusion detection system shown in an exemplary embodiment of this application;

[0026] Figure 3 This is a schematic diagram illustrating the workflow of a multimodal data association module according to an exemplary embodiment of this application;

[0027] Figure 4 This is a schematic diagram illustrating a feature deep fusion process according to an exemplary embodiment of this application;

[0028] Figure 5 This is a schematic diagram of the structure of a target detection device shown in an exemplary embodiment of this application;

[0029] Figure 6 This is a schematic diagram of the hardware structure of an electronic device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0030] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0031] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0032] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, and to make the above-mentioned objectives, features and advantages of the embodiments of this application more apparent and understandable, the technical solutions in the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0033] It should be noted that the sequence number of each step in the embodiments of this application does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0034] Please see Figure 1 This is a flowchart illustrating a target detection method provided in an embodiment of this application, as shown below. Figure 1 As shown, the target detection method may include the following steps:

[0035] Step S100: Determine candidate target boxes based on the target query.

[0036] For example, candidate bounding boxes can be generated using neural networks, such as multiple fully connected layers, based on the target query.

[0037] For example, the above candidate target boxes are 3D target boxes.

[0038] Step S110: Sampling radar points based on candidate target boxes to obtain radar points to be processed, and extracting features from the radar points to be processed to obtain radar features to be fused.

[0039] Step S120: Sampling image features based on candidate target boxes to obtain the features of the image to be fused.

[0040] In this embodiment of the application, for any candidate target box, on the one hand, radar point sampling can be performed based on the candidate target box to obtain the radar point to be processed.

[0041] For example, a certain number of radar points can be sampled within the candidate target box as radar points to be processed.

[0042] For the radar points to be processed obtained from sampling, feature extraction can be performed to obtain the radar features to be fused.

[0043] For example, the raw features of a radar may include point cloud coordinates (three-dimensional coordinates), lateral and longitudinal velocities, RCS (Radar Cross Section, also known as radar cross section), and SNR (Signal to Noise Ratio).

[0044] For radar points to be processed, feature extraction can be performed using neural networks (such as multi-layer fully connected layers) based on the original features of the radar points to be processed, so as to obtain the radar features to be fused.

[0045] For example, neural networks can be used to increase the dimensionality of the original features of the radar points to be processed, thereby obtaining the radar features to be fused.

[0046] Taking the original features of radar, including point cloud coordinates (3D), horizontal depth, vertical depth, RCS and SNR, as an example, the original features have a dimension of 7. After being processed by a neural network to increase the dimension, 64 / 128 / 256-dimensional radar features to be fused can be obtained.

[0047] For example, within a candidate target bounding box, multiple radar points to be processed can be sampled, so that a target query can obtain multiple radar features (one radar point to be processed corresponds to one radar feature).

[0048] On the other hand, image features can be sampled based on the candidate target box to obtain the features of the image to be fused.

[0049] For example, in order to achieve image feature sampling, images from different perspectives can be acquired using pre-deployed multi-view cameras, and image features can be extracted from the acquired images to obtain multiple image feature maps.

[0050] For example, assuming that there are M (M>1) cameras (also called webcams) with different perspectives deployed on a vehicle, the M frames of images captured by the M cameras at the same time (allowing for tolerable errors between the acquisition times of different cameras) can be used to extract image features and obtain an M*N image feature map.

[0051] For any candidate bounding box, the center point of the candidate bounding box can be used as a reference point. Based on the pre-calibrated intrinsic and extrinsic parameters of each camera, the reference point can be projected onto each image feature map through coordinate system transformation. Then, the feature point corresponding to the reference point in each image feature map can be obtained by using linear interpolation.

[0052] For example, in the case of M*N image feature maps, the reference point can correspond to M*N feature points.

[0053] Feature aggregation can be performed on the feature points corresponding to the reference points in each image feature map to obtain aggregated features. For example, cross-attention mechanism can be used to aggregate the feature points corresponding to the reference points in each image feature map.

[0054] For example, in order to optimize the performance of radar feature sampling and image feature sampling, the target query can be iteratively updated using the above-mentioned image feature sampling method, and radar feature sampling and image feature sampling can be performed based on the iteratively updated target query.

[0055] For example, if the candidate target box corresponding to the target query is determined and the aggregation feature is determined using the method described in the above embodiments, the target query can be updated based on the aggregation feature to obtain the updated target query.

[0056] For the updated target query, candidate target boxes can be determined again in the manner described in the above embodiments, and aggregated features can be determined. In this way, the target query can be iteratively updated until the number of iterations reaches the preset number.

[0057] After completing the iterative update of the target query, candidate target boxes can be determined based on the updated target query, and radar point sampling and feature extraction can be performed in accordance with the method described in the above embodiments to obtain radar features to be fused, and image feature sampling can be performed to obtain image features to be fused.

[0058] It should be noted that the initial target query can be generated through random initialization.

[0059] Step S130: For the interrelated radar features and image features to be fused, the cross-attention mechanism is used to perform deep feature fusion to obtain deep fused features; wherein, the interrelated radar features and image features to be fused are radar features and image features to be fused that are associated with the same candidate target box.

[0060] In this embodiment, considering that radar point clouds and images are two different types of sensor data with their own characteristics, different weights are needed when fusing them. In addition, considering that the points in the radar point cloud include points reflected by the target as well as noise points, different weights are needed when fusing different radar points. Therefore, the method of fusing radar features and image features using the contact method has obvious defects, and the feature fusion performance is cross-functional.

[0061] In view of the above problems, in the embodiments of this application, for radar features and image features to be fused that are associated with the same candidate target box (hereinafter referred to as mutually associated radar features and image features to be fused), the cross attention mechanism can be used to perform feature deep fusion on the radar features and image features to be fused to obtain deep fused features.

[0062] Step S140: Perform target detection on the deep fusion features.

[0063] In this embodiment of the application, when the deep fusion features are obtained in accordance with the manner described in the above embodiments, target detection can be performed based on the deep fusion features to achieve the prediction of the position and category of the target box.

[0064] For example, the category of the target box is determined by the requirements of the application. In autonomous driving scenarios, there are categories such as motor vehicles, non-motor vehicles, and people.

[0065] The dimensions of the target bounding box include the coordinates of the center point, length, width, height, and angles.

[0066] It can be seen that, in Figure 1 In the illustrated method flow, after candidate target boxes are determined based on the target query, on the one hand, radar point sampling is performed based on the candidate target boxes to obtain radar points to be processed, and features are extracted from the radar points to be processed to obtain radar features to be fused; on the other hand, image feature sampling is performed based on the candidate target boxes to obtain image features to be fused; for the interrelated radar features to be fused and image features to be fused, a cross-attention mechanism is used to perform deep feature fusion to obtain deep fused features, and target detection is performed on the deep fused features. By using the cross-attention mechanism to perform deep adjustment and fusion of radar features and image features, the data characteristics of sensor data of different modalities are fully considered, the feature fusion performance is optimized, higher quality deep fused features are obtained, and thus, the target detection performance is optimized.

[0067] In some embodiments, the above-mentioned sampling of radar points based on candidate target boxes to obtain radar points to be processed may include:

[0068] For any candidate bounding box, expand the candidate bounding box to obtain the processed candidate bounding box.

[0069] Within the range of the processed candidate target box, select K1 radar points that are closest to the center point of the processed candidate target box, and determine the selected radar points as radar points to be processed; K1 is a positive integer.

[0070] For example, in order to improve the reliability of radar point sampling, radar point sampling can be constrained based on candidate target boxes.

[0071] For any candidate target box, when radar point sampling is performed based on the target box, radar point sampling can be performed within the range of the candidate target box, and radar points outside the range of the candidate target box will not be sampled.

[0072] Considering that the accuracy of candidate target boxes generated based on the target query is not very high, in order to improve the reliability of the constraints of candidate target boxes, the candidate target boxes can be expanded.

[0073] Accordingly, for any candidate target box, the candidate target box can be expanded to obtain a processed candidate target box, and K1 radar points within the range of the processed candidate target box that are closest to the center point of the processed candidate target box can be selected as radar points to be processed.

[0074] Where K1 is an integer, and K1 > 1.

[0075] In one example, for any candidate bounding box, the expansion coefficient of the candidate bounding box is determined based on the distance from the center point of the candidate bounding box to the origin, and is positively correlated with that distance.

[0076] For example, considering that the distance of the reference point is usually greater, the expansion coefficient of the candidate target box can be determined based on the distance from the center point of the candidate target box to the origin when expanding the candidate target box.

[0077] Taking autonomous driving scenarios as an example, the origin can be the center position of the vehicle (i.e., the vehicle currently using the solution provided in this application embodiment for target detection).

[0078] For example, the expansion coefficient of the candidate target box is positively correlated with the distance from the center point of the candidate target box to the origin; that is, the greater the distance, the larger the expansion coefficient.

[0079] In one example, for any processed candidate target box, if the number of radar points within the range of the processed candidate target box is less than K1, it is padded with 0s to make up K1 radar points to be processed.

[0080] For example, considering that in real-world scenarios, there may be fewer than K1 radar points within the candidate target bounding box, K1 radar points to be processed can be padded with zeros to improve data processing efficiency while ensuring the reliability of the sampled radar points.

[0081] In some embodiments, the above-mentioned deep feature fusion of the radar features to be fused and the image features to be fused using the cross-attention mechanism to obtain deep fused features includes:

[0082] Based on the features of the image to be fused, a target query is generated using a fully connected network.

[0083] Based on the radar characteristics to be fused, a key and a value are generated using a fully connected network.

[0084] Generate an attention matrix based on the query and the key;

[0085] Based on the attention matrix and the value, the deep fusion features are determined.

[0086] For example, the fully connected networks used to generate the Query, Key, and Value are different fully connected networks.

[0087] In one example, an attention matrix is ​​generated based on the query and the key, implemented as follows:

[0088] A

[0089] Where A is the attention matrix;

[0090] Based on the attention matrix and the Value, the deep fusion features are determined in the following way:

[0091]

[0092] Where C represents the deep fusion feature and d represents the normalization factor.

[0093] To enable those skilled in the art to better understand the technical solutions provided in the embodiments of this application, the technical solutions provided in the embodiments of this application are described below with reference to specific examples.

[0094] In this embodiment, several vehicle-mounted cameras and millimeter-wave radars can be configured.

[0095] In one example, cameras are installed at four locations on the front, rear, left, and right of the vehicle, while angular millimeter-wave radars are installed at the four corners of the vehicle.

[0096] Please see Figure 2 This is a schematic diagram of the framework of a fusion detection system provided in an embodiment of this application, as shown below. Figure 2 As shown, the fusion detection system may include: an image feature extraction module, a multimodal data association module, a radar feature extraction module, a feature deep fusion module, and a 3D target prediction module.

[0097] The main processes of this fusion detection system for target detection include:

[0098] Multi-view camera images are processed by an image feature extraction module to obtain 2D image features. Candidate target bounding boxes are generated based on a target query. Image features and radar point clouds are sampled based on these candidate bounding boxes to complete multimodal data association. The radar point clouds sampled for each target query are then processed by a radar feature extraction module to obtain radar features. The associated radar features and image features are then fed into a feature deep fusion module for depth adjustment and fusion, and finally into a 3D target prediction module to obtain the final 3D detection result.

[0099] The following sections will provide a detailed explanation of the functional implementation of each module.

[0100] 1. Image Feature Extraction Module

[0101] Feature extraction is performed on images from M camera viewpoints. The feature extractor typically includes a backbone network and a neck network (a type of network that extracts more complex features, usually located after the backbone network and before the prediction head).

[0102] In one example, bcakbone can use ResNet101, and neck can use an N-layer FPN (Feature Pyramid Network). After image feature extraction, M*N image feature maps are generated.

[0103] 2. Multimodal data association module

[0104] The multimodal data association module can include an image sampler and a radar sampler, which sample and associate images and millimeter-wave radar data respectively. Figure 3 As shown, the entire process can include three steps:

[0105] 2.1 Target query generates candidate target boxes (3D target boxes) through a network (such as multi-layer fully connected networks).

[0106] For example, considering that the accuracy of candidate target boxes generated based on target queries is not very high, the reliability of the constraints on sampled radar points based on candidate target boxes can be improved by expanding the candidate target boxes.

[0107] Considering the characteristics of visual imaging, the greater the distance, the less accurate the localization becomes. Therefore, the expansion coefficient (also known as the scaling factor) for expanding the candidate target box is related to the distance.

[0108] For example, the formula for calculating the scaling factor σ can be:

[0109]

[0110] Where d is the distance from the center point of the candidate target box to the origin, and dmax is the preset maximum detection distance.

[0111] For example, the length and width of the candidate target box can be expanded according to the scaling factor σ. Within the expanded candidate target box, K1 radar points that are closest to the center point of the expanded candidate target box can be selected as radar points to be processed, thereby realizing radar point sampling.

[0112] For example, if there are fewer than K1 radar points within the expanded candidate target frame, they can be padded with zeros.

[0113] 2.2 Image feature sampling.

[0114] For example, the center point of the candidate target box can be used as a reference point. Based on the pre-calibrated intrinsic and extrinsic parameters of each camera, the reference point can be projected onto each image feature map through coordinate system transformation. Then, the feature points corresponding to the reference point in each image feature map can be obtained by using linear interpolation (one candidate target box corresponds to M*N feature points).

[0115] 2.3. For the obtained M*N feature points, use the cross-attention mechanism to aggregate the features and obtain aggregated features.

[0116] It should be noted that, in order to optimize the performance of radar feature sampling and image feature sampling, the above-mentioned image feature sampling method can be used to iteratively update the target query, and radar feature sampling and image feature sampling can be performed based on the iteratively updated target query.

[0117] 3. Radar Feature Extraction Module

[0118] For example, the raw features of a radar may include point cloud coordinates (three-dimensional coordinates), lateral and longitudinal velocities, RCS, SNR, etc.

[0119] Based on the original characteristics of the radar points to be processed, feature extraction can be performed using neural networks (such as multi-layer fully connected layers) to obtain the radar features to be fused.

[0120] For example, for a target query, K1 radar features to be fused can be obtained.

[0121] 4. Feature Deep Fusion Module

[0122] After processing by the above modules, image features and radar features are associated in a one-to-many (K1) manner. These features are then input into the feature deep fusion module, which is implemented as follows: Figure 4 As shown.

[0123] Considering that radar point clouds and images are two different types of sensor data, each with its own characteristics; furthermore, for the radar features to be fused corresponding to different radar points to be processed, their importance is not entirely the same due to the different locations of the radar points to be processed. The closer a point is to the center point, the higher the probability that it belongs to that target, while the farther away from the center point, the more likely it is to be a point of other targets or noise.

[0124] Therefore, the method of fusing radar features and image features using the contact method has obvious drawbacks, and the feature fusion performance is inconsistent.

[0125] In this embodiment, a cross-attention mechanism can be introduced for deep feature fusion.

[0126] For example, image features are used to generate a target query Query through a fully connected network, and neighborhood radar features are used to generate a key and a value through a fully connected network.

[0127] For example, suppose the features of the images to be fused are The radar features to be fused are The mapping can be performed using a multilayer perceptron (MLP) network, as shown in the following formula:

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135] Where C1 is the dimension of the image features to be fused, C2 is the dimension of the radar features to be fused, d = C3 is the scaling factor (also known as the normalization factor), A is the attention matrix, which is used to measure the importance of each radar feature, and C is the deep fusion feature.

[0136] 5. 3D Target Prediction Module

[0137] For example, target detection can be performed based on deep fusion features to predict the location and category of the target bounding box.

[0138] For example, the category of the target box is determined by the requirements of the application. In autonomous driving scenarios, there are categories such as motor vehicles, non-motor vehicles, and people.

[0139] The dimensions of the target bounding box include the coordinates of the center point, length, width, height, and angles.

[0140] The method provided in this application has been described above. The apparatus provided in this application is described below:

[0141] Please see Figure 5 This is a schematic diagram of the structure of a target detection device provided in an embodiment of this application, as shown below. Figure 5 As shown, the target detection device may include:

[0142] Determining unit 510 is used to determine candidate target boxes based on the target query;

[0143] The radar feature acquisition unit 520 is used to sample radar points based on the candidate target box to obtain radar points to be processed, and to extract features from the radar points to be processed to obtain radar features to be fused.

[0144] The image feature acquisition unit 530 is used to sample image features based on the candidate target boxes to obtain the image features to be fused.

[0145] The feature fusion unit 540 is used to perform deep feature fusion on the radar features and image features to be fused that are related to each other using a cross-attention mechanism to obtain deep fused features; wherein, the radar features and image features to be fused that are related to the same candidate target box are radar features and image features to be fused.

[0146] The target detection unit 550 is used to perform target detection on the deep fusion features.

[0147] In some embodiments, the radar feature acquisition unit 520 samples radar points based on the candidate target boxes to obtain radar points to be processed, including:

[0148] For any candidate bounding box, expand the candidate bounding box to obtain the processed candidate bounding box.

[0149] Within the range of the processed candidate target box, select K1 radar points that are closest to the center point of the processed candidate target box, and determine the selected radar points as radar points to be processed; K1 is an integer and K1>1.

[0150] In some embodiments, for any candidate target box, the expansion coefficient of the candidate target box is determined based on the distance from the center point of the candidate target box to the origin, and is positively correlated with the distance.

[0151] In some embodiments, for any processed candidate target box, if the number of radar points within the range of the processed candidate target box is less than K1, K1 radar points to be processed are added by padding with zeros.

[0152] In some embodiments, the feature fusion unit 540 utilizes a cross-attention mechanism to perform deep feature fusion on the radar features to be fused and the image features to be fused, obtaining deeply fused features, including:

[0153] Based on the features of the image to be fused, a target query is generated using a fully connected network.

[0154] Based on the radar characteristics to be fused, a key and a value are generated using a fully connected network.

[0155] Generate an attention matrix based on the query and the key;

[0156] Based on the attention matrix and the value, the deep fusion features are determined.

[0157] In some embodiments, the feature fusion unit 540 generates an attention matrix based on the Query and the Key, which is achieved in the following way:

[0158] A = Query * Key T

[0159] Where A is the attention matrix;

[0160] The feature fusion unit 540 determines the deep fusion features based on the attention matrix and the Value, and achieves this in the following way:

[0161]

[0162] Where C represents the deep fusion feature and d represents the normalization factor.

[0163] This application provides an electronic device including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the target detection method described above.

[0164] Please see Figure 6This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor 601 and a memory 602 storing machine-executable instructions. The processor 601 and the memory 602 can communicate via a system bus 603. Furthermore, by reading and executing the machine-executable instructions corresponding to the target detection logic in the memory 602, the processor 601 can execute the target detection method described above.

[0165] The memory 602 mentioned in this document can be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, etc. For example, machine-readable storage media can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0166] In some embodiments, a machine-readable storage medium, such as Figure 6 The memory 602 in the machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the target detection method described above. For example, the storage medium may be ROM, RAM, CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0167] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0168] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target detection method, characterized in that, include: Candidate target boxes are determined based on the target query. For any candidate target box, its center point is used as a reference point. The corresponding feature points in multiple image feature maps are determined, and these feature points are aggregated to obtain aggregated features. The target query is then updated based on these aggregated features. After iterative updates of the target query are completed, candidate target boxes are determined based on the updated target query. The multiple image feature maps are obtained by extracting image features from images captured by pre-deployed multi-view cameras from different perspectives. Radar point sampling is performed based on the candidate target boxes to obtain radar points to be processed, and feature extraction is performed on the radar points to be processed to obtain radar features to be fused; the radar points are collected by millimeter-wave radar equipment, and... Image features are sampled based on the candidate target boxes to obtain the features of the image to be fused; For interrelated radar features and image features to be fused, a cross-attention mechanism is used to perform deep feature fusion to obtain deep fused features; where the interrelated radar features and image features to be fused are those associated with the same candidate target box. Target detection is performed on the deep fusion features; The method of using a cross-attention mechanism to perform deep feature fusion on the radar features and image features to be fused, to obtain deep fused features, includes: Based on the features of the image to be fused, a target query is generated using a fully connected network. Based on the radar characteristics to be fused, a key and a value are generated using a fully connected network. Generate an attention matrix based on the query and the key; Based on the attention matrix and the value, the deep fusion features are determined.

2. The method according to claim 1, characterized in that, The step of sampling radar points based on the candidate target boxes to obtain radar points to be processed includes: For any candidate bounding box, expand the candidate bounding box to obtain the processed candidate bounding box. Within the range of the processed candidate target box, select K1 radar points that are closest to the center point of the processed candidate target box, and determine the selected radar points as radar points to be processed; K1 is an integer and K1>1.

3. The method according to claim 2, characterized in that, For any candidate bounding box, the expansion coefficient of the candidate bounding box is determined based on the distance from the center point of the candidate bounding box to the origin, and is positively correlated with this distance.

4. The method according to claim 2, characterized in that, For any processed candidate target box, if the number of radar points within the range of the processed candidate target box is less than K1, it is padded with 0 to make K1 radar points to be processed.

5. The method according to claim 4, characterized in that, The generation of the attention matrix based on the Query and the Key is achieved in the following way: Where A is the attention matrix; The deep fusion features are determined based on the attention matrix and the Value, which is achieved in the following way: Where C represents the deep fusion feature and d represents the normalization factor.

6. A target detection device, characterized in that, include: A determining unit is used to determine candidate target boxes based on a target query. For any candidate target box, using the center point of the candidate target box as a reference point, feature points corresponding to the reference point in multiple image feature maps are determined. These feature points are then aggregated to obtain aggregated features. The target query is updated based on these aggregated features. After iterative updates of the target query are completed, candidate target boxes are determined based on the updated target query. The multiple image feature maps are obtained by extracting image features from images acquired from different perspectives by pre-deployed multi-view cameras. The radar feature acquisition unit is used to sample radar points based on the candidate target box to obtain radar points to be processed, and to extract features from the radar points to be processed to obtain radar features to be fused; the radar points are acquired by millimeter-wave radar equipment. An image feature acquisition unit is used to sample image features based on the candidate target boxes to obtain the features of the image to be fused. The feature fusion unit is used to perform deep feature fusion on the interrelated radar features and image features to be fused using a cross-attention mechanism to obtain deep fused features; wherein, the interrelated radar features and image features to be fused are radar features and image features to be fused that are associated with the same candidate target box. The target detection unit is used to perform target detection on the deep fusion features; The feature fusion unit utilizes a cross-attention mechanism to perform deep feature fusion on the radar features and image features to be fused, resulting in deeply fused features, including: Based on the features of the image to be fused, a target query is generated using a fully connected network. Based on the radar characteristics to be fused, a key and a value are generated using a fully connected network. Generate an attention matrix based on the query and the key; Based on the attention matrix and the value, the deep fusion features are determined.

7. The apparatus according to claim 6, characterized in that, The radar feature acquisition unit samples radar points based on the candidate target boxes to obtain radar points to be processed, including: For any candidate bounding box, expand the candidate bounding box to obtain the processed candidate bounding box. Within the range of the processed candidate target box, select K1 radar points that are closest to the center point of the processed candidate target box, and determine the selected radar points as radar points to be processed; K1 is an integer, and K1>1; For any candidate bounding box, the expansion coefficient of the candidate bounding box is determined based on the distance from the center point of the candidate bounding box to the origin, and is positively correlated with this distance; Wherein, for any processed candidate target box, if the number of radar points within the range of the processed candidate target box is less than K1, it is padded with 0 to make K1 radar points to be processed. And / or, The feature fusion unit generates an attention matrix based on the Query and the Key, which is achieved in the following way: Where A is the attention matrix; The feature fusion unit determines the deep fusion features based on the attention matrix and the Value, and achieves this in the following way: Where C represents the deep fusion feature and d represents the normalization factor.

8. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the method as described in any one of claims 1-5.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-5.

Images