A target detection method that combines wireless and visual technologies

Abstract

This invention provides a target detection method that integrates wireless and visual methods, mainly comprising: detecting a target in a first image data using an image-based target detector; tracking a target in a second image data using a radar-based target tracker; generating bounding box proposals based on the bounding boxes output by the image-based target detector and the 3D bounding boxes generated by the radar-based target tracker; cropping the global multimodal feature map based on the bounding box proposals to obtain local features of the image; constructing a fusion network to predict each bounding box in the bounding box proposals and the corresponding confidence score for each bounding box, and determining whether each bounding box should be retained based on the predicted confidence score. This invention not only improves the accuracy of target detection in unconventional environments but also saves computational costs.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more particularly to a target detection method that integrates wireless and visual technologies. Background Technology

[0002] Object detection is a key research area in computer vision and a crucial foundation for understanding high-level semantic information in images. In recent years, the development of deep learning has driven advancements in image object detection technology, resulting in significant improvements in both accuracy and speed. Many convolutional neural network object recognizers have been proposed. While these image-based detectors have achieved good performance, they are easily affected by adverse environmental conditions such as fog, smoke, and poor lighting. Summary of the Invention

[0003] To address the shortcomings of single sensors in providing a complete and accurate description of targets, and the fact that detection accuracy is easily affected by environmental factors in unconventional environments, this invention proposes a target detection method that integrates wireless and visual technologies. This method incorporates millimeter-wave radar as an auxiliary technology, thereby improving the accuracy of target detection in unconventional environments.

[0004] The technical means employed in this invention are as follows:

[0005] A target detection method integrating wireless and visual technologies includes the following steps:

[0006] First image data containing human targets is acquired based on a camera; correspondingly, second image data containing human targets is acquired based on radar.

[0007] The first image data is used to detect targets using an image-based target detector; the second image data is used to track targets using a radar-based target tracker, wherein the radar-based target tracker is used to generate a 3D bounding box after clustering point cloud data using the DBSCAN algorithm.

[0008] Based on the bounding boxes output by the image-based target detector and the 3D bounding boxes produced by the radar-based target tracker, bounding box suggestion information is generated.

[0009] Global feature extraction is performed on the first image data and the second image data to obtain a global multimodal feature map;

[0010] The global multimodal feature map is cropped based on the bounding box suggestion information to obtain the local features of the image;

[0011] A fusion network is constructed to predict each bounding box in the bounding box proposal information and the corresponding confidence score for each bounding box. Based on the predicted confidence score, it is determined whether each bounding box should be retained.

[0012] Furthermore, object detection is performed on the first image data using an image-based object detector, including outputting bounding boxes, class scores, and confidence scores through a CNN network. Specifically:

[0013] Given an image I, use feature extractor F body First, extract the internal feature map f = F. body (I);

[0014] Another network F head The feature map is processed to generate a set of output boxes B:

[0015] B = F head (F body (I))

[0016] In the overall architecture, the bounding box detection results and the internal feature map f are used by the subsequent fusion framework.

[0017] B is filtered by a confidence threshold to output the bounding box.

[0018] Furthermore, the radar-based target tracker is used to generate 3D bounding boxes after clustering point cloud data using the DBSCAN algorithm, including:

[0019] Obtain a 4-dimensional vector for each point, consisting of its x, y, z axis coordinates and radial velocity;

[0020] The DBSCAN algorithm is used to identify foreground objects from an image, where each point of the foreground object is marked by a cluster index or outlier flag.

[0021] For each cluster, search for its outermost point and approximate the size of the 3D bounding box based on the outermost point.

[0022] Furthermore, the DBSCAN algorithm is used to identify foreground targets from the image, each point of which is marked by a cluster index or outlier flag. The method also includes estimating the center position of each cluster and the velocity along the z-axis by averaging the corresponding values ​​of all points in the cluster.

[0023] Furthermore, the radar-based target tracker, used to generate 3D bounding boxes after clustering point cloud data using the DBSCAN algorithm, also includes:

[0024] Slice the 3D bounding box along the z-axis to obtain a cross section;

[0025] These cross sections are projected onto a 2D image, and then the intrinsic and extrinsic parameters of the camera are obtained through camera calibration.

[0026] By transforming the radar coordinate system to the camera coordinate system, the projection of the radar point cloud onto the visual image is obtained.

[0027] Furthermore, a fusion network is constructed to predict each bounding box in the bounding box proposal information and the confidence score corresponding to each bounding box, including:

[0028] A 490-channel position-sensitive scoring map was constructed using a 1×1 convolutional layer based on the internal feature map output by the image-based object detector.

[0029] Use a location-sensitive region of interest alignment layer to crop the score map based on the suggested location for each bounding box;

[0030] Unstructured radar point clouds are encoded into two-dimensional images, thus obtaining a 3-channel heatmap;

[0031] The 3-channel heatmap is input into a three-layer CNN to extract the occupancy feature map, and the probability of the target's presence at each location is embedded in it; the features are aligned, cropped, and generated as radar features for each bounding box; two fully connected (FC) layers are used for box regression and classification to obtain the radar confidence score.

[0032] The confidence scores of the two modes are added together and sent to the sigmoid layer to obtain the final score.

[0033] Furthermore, encoding unstructured radar point clouds into two-dimensional images includes:

[0034] Project the point cloud onto a two-dimensional image coordinate system;

[0035] Calculate the two-dimensional histogram of the projected point cloud on the three channels, including the number of points on the z-axis, average depth, and average velocity;

[0036] The values ​​on each channel are normalized to the range [0,1].

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] 1. This invention proposes a millimeter-wave radar and camera fusion system for target detection, requiring only a small amount of labeled image / radar data for new scenes, as it can fully utilize large public image datasets for extensive training. This significant feature enables it to adapt to highly complex real-world environments. It addresses the problem of limited publicly available camera-radar synchronization datasets.

[0039] 2. This invention not only improves the accuracy of target detection in unconventional environments, but also saves computational costs. Its main contribution is the realization of a radar-camera fusion system for reliable target detection. Its learning-based fusion optimization design can significantly improve the performance of existing image-based target detectors in challenging environments.

[0040] For the reasons mentioned above, multimodal target detection based on wireless and vision fusion can improve the accuracy of target detection in unconventional environments and is applicable to various fields such as autonomous driving, intelligent robots, and security monitoring. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart of a target detection method that combines wireless and visual methods in an embodiment.

[0043] Figure 2 This is a flowchart of point cloud processing in an embodiment. Detailed Implementation

[0044] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0045] like Figure 1 As shown in the figure, this invention discloses a target detection method that integrates wireless and visual technologies, comprising the following steps:

[0046] S1. Acquire first image data containing human targets based on a camera; correspondingly, acquire second image data containing human targets based on radar.

[0047] In this embodiment, a practical multimodal dataset for radar and camera fusion was collected.

[0048] S2. Target detection is performed on the first image data using an image-based target detector; target tracking is performed on the second image data using a radar-based target tracker, wherein the radar-based target tracker is used to generate a 3D bounding box after clustering point cloud data using the DBSCAN algorithm.

[0049] S3. Generate bounding box suggestion information based on the bounding boxes output by the image-based target detector and the 3D bounding boxes produced by the radar-based target tracker.

[0050] S4. Perform global feature extraction on the first image data and the second image data to obtain a global multimodal feature map.

[0051] S5. The global multimodal feature map is cropped according to the bounding box suggestion information to obtain the local features of the image.

[0052] S6. Construct a fusion network to predict each bounding box in the bounding box proposal information and the corresponding confidence score for each bounding box, and determine whether each bounding box should be retained based on the predicted confidence score.

[0053] This invention follows a two-stage fusion paradigm. The first stage aims to aggregate bounding box proposals from an image-based object detector and a radar-based tracker. Simultaneously, global feature extraction is performed on the entire frame from both the image and radar branches to obtain a global feature map. Then, two cropping operations (PS-RoI Align and RoI Align) are performed, cropping the global feature map based on the location of the bounding box proposals to obtain local features. In the second stage, a fusion architecture is proposed to predict a new bounding box and a confidence score for each bounding box through a fusion network. Based on the new confidence scores, a threshold can be easily used to determine whether each box proposal should be retained. In summary, this invention mainly consists of three modules: an image-based object detector, a radar-based object tracker, and a fusion framework. These three modules are assembled in a loosely coupled manner, enabling support for different image-based object detectors. Furthermore, separate weight training for the image correlation module is implemented, reducing the dependence on large amounts of labeled multimodal data.

[0054] Image-based object detector: A CNN-based object detector is used to obtain bounding boxes, class scores, and confidence scores. Specifically, given an image I, the feature extractor (body) F... body First, extract the internal feature map f = F. body (I). A typical feature extractor consists of several convolutional layers, activation layers, and pooling layers. Then, another network (head) F... headThe feature map is processed to generate a set of output boxes B. Therefore, the entire process of an object detector can be summarized as B = F. head (F body (I)).

[0055] In the overall architecture, the bounding box detection results and the internal feature map f are used by the subsequent fusion framework. Specifically, B is first filtered by a confidence threshold and then combined with the radar bounding box proposal as the overall candidate bounding box proposal. f is used to construct a position-sensitive scoring map through a 1×1 convolutional layer. The reuse of feature maps from image-based detectors saves a significant amount of computation, and the overall network architecture is compatible with different object detectors, including YOLO and SSD. This compatibility leverages the latest advancements in detection algorithms, allowing users to easily upgrade object detectors without redesigning the entire fusion model.

[0056] Radar-based target detectors: First, point cloud data is processed. Each frame of the secure radar point cloud is a set of points, each represented by a 4-dimensional vector consisting of x (from left to right), y (from top to bottom), z (from back to front) axis coordinates and radial velocity (velocity along the z-axis). However, clutter and noise severely pollute the radar point cloud, leading to unwanted false keypoints. Therefore, a density-based clustering method, DBSCAN, is used to identify foreground targets from clutter. Points from foreground objects can be grouped into clusters, while points from unwanted noise are typically scattered at low density. Furthermore, unlike K-means, DBSCAN does not require prior information about the number of clusters, making it well-suited for object detection tasks where the number of objects is unknown. After DBSCAN, each point is labeled with a cluster index or outlier flag. After filtering out outliers, the center position and z-axis velocity of each cluster are estimated by averaging the corresponding values ​​of all points in that cluster. Additionally, for each cluster, the outermost points belonging to it are searched, and these points are used to approximate the size of the 3D bounding box.

[0057] Spatiotemporal synchronization of radar and vision: A 3D bounding box is generated in radar coordinates and with a timestamp through clustering and tracking steps. To achieve fusion between the two sensors, a unified coordinate system and timestamps are required. Spatially, the 3D bounding box is sliced ​​along the z-axis to obtain cross-sections. These cross-sections are then projected onto a 2D image. Camera intrinsic and extrinsic parameters are obtained through camera calibration. Coordinate transformation is then performed to convert the radar coordinate system to the camera coordinate system and then to the pixel coordinate system, resulting in a projection of the radar point cloud onto the visual image. For temporal synchronization, each image frame is associated with the most recent radar frame.

[0058] Fusion Framework: For image features, the internal feature maps of the image-based detector are separated and represented using a 1×1 convolutional layer. Then, a position-sensitive region of interest alignment layer is used to crop the score map according to the location of each bounding box proposal. For each bounding box, we obtain a 7×7×10 feature map.

[0059] For radar features, it is difficult to directly extract spatial patterns from radar point clouds due to their sparse distribution and uncertain length within the field of view. To leverage the powerful spatial feature extraction capabilities of convolutional layers, we encode unstructured radar point clouds into two-dimensional images through three preprocessing steps:

[0060] (1) Project the point cloud onto the two-dimensional image coordinates.

[0061] (2) Calculate the two-dimensional histogram of the projected point cloud on the three channels: the number of points on the z-axis, the average depth, and the average velocity.

[0062] (3) Normalize the value on each channel to the range [0,1].

[0063] These steps yield a 3-channel heatmap. This heatmap is then fed into a three-layer CNN to extract an occupancy feature map, embedding the probability of an object's presence at each location. Alignment and cropping are performed to generate radar features for each bounding box, also 7×7×10 in size. The image feature map is then flattened, followed by a 256-channel fully connected (FC) layer, and finally two FC layers for box regression and classification. For the radar features, two convolutional layers with kernel sizes of 7×7 and 1×1 are used to abstract a global representation, which is also the radar confidence score. The confidence scores of the two modes are then summed and fed into a sigmoid layer to obtain the final score. The outputs of the R-CNN and fusion module construct a feature vector with the same dimension as the image-based object detector's output. To merge the two feature vector results for more reliable optimization, a learning-based ensemble module is introduced, consisting of two fully connected layers. The first fully connected layer fuses information from each class from both inputs, while the second fully connected layer captures global correlations between classes. The final softmax layer outputs a 2-dimensional vector, forcing the network to make a decision between foreground and background.

[0064] Traditional multimodal object detection based on the fusion of wireless and visual data often involves placing the two modalities into different feature extractors and concatenating internal image and radar feature maps. This fusion model requires training from scratch using a multimodal dataset, but acquiring image feature extraction capabilities requires a large amount of labeled data. However, publicly available camera-radar datasets are scarce. Therefore, users typically need to collect and label their own datasets based on object categories and deployment environments, which is very costly. This invention proposes a training method with separate weights, supporting separate training of image-related and radar-related modules. It primarily pre-trains the image-related layer using a large public image dataset, while fine-tuning the radar feature extraction and fusion layers using a small amount of labeled multimodal datasets. This preserves image feature extraction capabilities while adapting to the distribution of multimodal data in new scenarios, solving the problem of limited simultaneous visual and radar datasets. Finally, the fusion of visual and radar data improves the performance of image-based object detectors in challenging environments.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for wireless and visual fusion target detection, characterized in that, The method comprises the following steps: obtaining first image data with a human target based on a camera; correspondingly, obtaining second image data with a human target based on a radar; performing target detection on the first image data by an image-based target detector; performing target tracking on the second image data by a radar-based target tracker, the radar-based target tracker being configured to generate a three-dimensional bounding box after clustering point cloud data by a DBSCAN algorithm; generating bounding box suggestion information according to a bounding box output by the image-based target detector and the three-dimensional bounding box generated by the radar-based target tracker; extracting global multi-modal feature maps from the first image data and the second image data; cropping the global multi-modal feature maps according to the bounding box suggestion information to obtain local features of the image; constructing a fusion network to predict each bounding box in the bounding box suggestion information and a confidence score corresponding to each bounding box, and determining whether each bounding box should be retained based on the predicted confidence score, wherein the construction of the fusion network comprises: encoding unstructured radar point cloud represented by the second image data into a two-dimensional image to obtain a 3-channel heat map, wherein the encoding into a two-dimensional image comprises: calculating a two-dimensional histogram of the projected point cloud in three channels, the three channels including the number of points on the z-axis, the average depth, and the average speed; extracting an image confidence score and a radar confidence score respectively based on the local features and radar features extracted from the 3-channel heat map; adding the confidence scores of the two modes together and sending them to a sigmoid layer to obtain a final score as the predicted confidence score.

2. The method of claim 1, wherein, The target detection on the first image data by the image-based target detector comprises outputting a bounding box, a class score and a confidence score by a CNN network, specifically: Given an image I, by a feature extractor F body First extract the internal feature map f = F body (I); Another network F head The feature map is processed to generate a set of output boxes B: B = F head (F body (I)) In the overall architecture, the detection result of the bounding box and the internal feature map f are used by the subsequent fusion framework, the B is filtered by a confidence threshold to output the bounding box.

3. The method of claim 1, wherein, The radar-based target tracker is configured to generate a three-dimensional bounding box after clustering point cloud data by a DBSCAN algorithm, comprising: obtaining a 4-dimensional vector composed of x, y, z axis coordinates and radial velocity for each point; using the DBSCAN algorithm to identify foreground targets from the image, each point of the foreground target being marked by an index of a cluster or an outlier flag; for each cluster, searching for the outermost point belonging to it, and approximating the size of the three-dimensional bounding box based on the outermost point.

4. The method of claim 3, wherein, Using the DBSCAN algorithm to identify foreground targets from the image, each point of the foreground target being marked by an index of a cluster or an outlier flag, further comprising: estimating the center position and the velocity along the z-axis of each cluster by averaging the corresponding values of all points in the cluster.

5. The method of claim 3, wherein the method further comprises: The radar-based target tracker is configured to generate a three-dimensional bounding box after clustering point cloud data by a DBSCAN algorithm, further comprising: slicing the three-dimensional bounding box in the z-axis to obtain cross sections; projecting the cross sections into a 2D image, and then obtaining the intrinsic and extrinsic matrices of the camera through camera calibration; The radar coordinate system is converted to the camera coordinate system by coordinate system conversion to obtain the projection of the radar point cloud to the visual image.

6. The method of claim 1, wherein, A fusion network is constructed to predict each bounding box in the bounding box proposal information and the confidence score corresponding to each bounding box, including: A 490-channel position-sensitive score map is constructed using a 1x1 convolution layer according to the internal feature map output by the image-based target detector; A position-sensitive region of interest alignment layer is used to crop the score map according to the position of each bounding box proposal; The unstructured radar point cloud is encoded into a two-dimensional image, thereby obtaining a 3-channel heat map; The 3-channel heat map is input into a three-layer CNN to extract an occupancy feature map and embed the probability of the target existing at each position therein; alignment is then performed, and the radar feature of each bounding box is cropped; two FC layers are used for box regression and classification, thereby obtaining the confidence score of the radar; The confidence scores of the two modes are added together and sent to a sigmoid layer to obtain the final score.

7. The method of claim 6, wherein the method further comprises: Encoding the unstructured radar point cloud into a two-dimensional image includes: Projecting the point cloud into a two-dimensional image coordinate; Calculating a two-dimensional histogram of the projected point cloud in three channels, including the number of points on the z-axis, the average depth, and the average speed; Normalizing the values in each channel to the range [0, 1].

Images