Target detection method and device based on wireless sensing and image fusion

Abstract

The embodiment of the invention provides a target detection method and device based on wireless sensing and image fusion. Comprising the steps that an image and perception data are acquired, the image and the perception data are processed respectively, a first bounding box and a second bounding box corresponding to at least one detection target, global features corresponding to the image and global features corresponding to the perception data are obtained, the first bounding box corresponds to the image, and the second bounding box corresponds to the second bounding box; the second bounding box corresponds to the sensing data; based on a global feature corresponding to the image and a global feature corresponding to the sensing data, obtaining a local image feature and a local sensing feature of the at least one detection target in a candidate box set, the candidate box set being obtained by performing fusion processing based on the first bounding box and the second bounding box; and processing the local image feature and the local sensing feature of at least one detection target in the candidate box set to obtain the target. Therefore, the accuracy of target detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of detection technology, and in particular to a target detection method and device based on wireless sensing and image fusion. Background Art

[0002] Image-based detectors are susceptible to image quality degradation under adverse environmental conditions such as heavy rain or insufficient light. In extreme environments, traditional image-based detection modules fail, while perception-based detectors can still generate corresponding detection candidate frames. However, wireless perception cannot obtain clear images, and solutions must be sought from the aspects of technical reuse complementarity and data fusion. Based on wireless perception and image data fusion, the design of a target detector based on perception and image fusion can improve detection accuracy. For example, traditional cameras will have problems such as false alarms and missed alarms in harsh environments. Traditional wireless perception is applicable in harsh environments but cannot see clear images, nor can it identify information such as vehicle license plates, models, logos, and body colors. Therefore, wireless perception technology can be integrated with image detection to create a wireless perception and image fusion holographic road perception system with important breakthroughs in signal processing, target recognition, and event detection algorithms.

[0003] Most existing wireless sensing and image fusion methods can be divided into three categories: data fusion, feature fusion, and result fusion. 1. Data fusion methods tend to follow the core idea of ​​using wireless sensing data to guide target detection in image space. The detection of wireless sensing data is converted into a region of interest (ROI) in the image and then used for further image processing. The core idea is to generate candidate boxes of target objects from one type of wireless sensing data and then verify them on another type of data. A major disadvantage of data fusion is that the detection pipeline completely ignores areas without wireless sensing data, resulting in the possibility of missed detection. 2. Feature fusion is a recently popular fusion concept that enables the detection system to learn cross-sensing patterns with different feature representations. With the emergence of advanced deep learning networks for image classification and regression, most feature fusion methods utilize complex image-based detection networks and modify them to integrate wireless sensing data. Compared with data fusion, feature layer fusion is more low-level and more conducive to the neural network to learn the complementarity between different sensors, but the complexity of the algorithm design is also relatively high. The general practice is to map the wireless sensing point cloud data to the image coordinate system to form a point cloud image similar to the camera image. 3. Result fusion is performed at the end of two independent processing branches to integrate the respective modal detection results. This method needs to consider how to integrate results of different credibility, and most of them adopt traditional filtering-based strategies. Since the complementarity at the feature level is not fully considered, this method has limited improvement on system performance.

[0004] Since the data sources, data structures and data features are different in data fusion, the fusion effect is poor; in feature fusion, the fusion of the feature layer is more basic, which is more conducive to the neural network to learn the complementarity between different sensors, but the complexity of the algorithm design is relatively high, and the fusion method needs further study; in result fusion, the complementarity at the feature level is not fully considered, and the improvement of system performance is limited.

[0005] Therefore, it is necessary to provide a new target detection method. Summary of the invention

[0006] The present application discloses a target detection method and device based on wireless sensing and image fusion, which can improve the accuracy of target detection.

[0007] In a first aspect, an embodiment of the present application provides a target detection method based on wireless sensing and image fusion, including:

[0008] Acquire an image and perception data, and process the image and the perception data respectively to obtain a first bounding box and a second bounding box corresponding to at least one detection target, as well as a global feature corresponding to the image and a global feature corresponding to the perception data, wherein the first bounding box corresponds to the image and the second bounding box corresponds to the perception data;

[0009] Based on the global features corresponding to the image and the global features corresponding to the perception data, obtaining local image features and local perception features of the at least one detection target in a candidate box set, wherein the candidate box set is obtained by fusion processing based on the first bounding box and the second bounding box;

[0010] The local image features and local perception features of at least one detection target in the candidate frame set are processed to obtain the target.

[0011] In the embodiment of the present application, by processing the image and the perception data respectively, a first bounding box and a second bounding box corresponding to at least one detection target, as well as global features corresponding to the image and global features corresponding to the perception data are obtained; then, based on the global features corresponding to the image and the global features corresponding to the perception data, local image features and local perception features of at least one detection target in the candidate frame set are obtained; then, based on the local image features and local perception features of at least one detection target, processing is performed to obtain the target. By adopting this method, the candidate frame set obtained based on the image and the perception data is processed, which effectively improves the accuracy of target detection based on perception and image fusion.

[0012] In a possible implementation, the processing of the local image features and the local perception features of the at least one detection target in the candidate frame set to obtain the target includes:

[0013] Performing feature extraction processing on the local image features and the local perception features of each detection target in the at least one detection target in the candidate frame set, respectively, to obtain a first confidence and a second confidence of each detection target, wherein the first confidence corresponds to the local image features, and the second confidence corresponds to the local perception features;

[0014] The first confidence and the second confidence of each detected target are fused to obtain a fused confidence of at least one detected target in the candidate frame set, wherein the fused confidence of the target is greater than a first threshold.

[0015] In this example, confidence scores are obtained and fused based on candidate boxes, which can effectively improve detection accuracy.

[0016] In a possible implementation manner, the fusing the first confidence and the second confidence of each detection target to obtain a fused confidence of the at least one detection target in the candidate frame set includes:

[0017] Adding the first confidence level and the second confidence level of each detected target respectively to obtain the sum of the confidence levels of each detected target;

[0018] Preset processing is performed on the sum of the confidences of the at least one detection target respectively to obtain a fused confidence of the at least one detection target.

[0019] In a possible implementation, the method further includes:

[0020] It is confirmed that the confidence level of the first bounding box is not less than a second threshold.

[0021] This example first performs confidence screening on the bounding box corresponding to the image to ensure the reliability of its confidence, further improving the accuracy of target detection.

[0022] In a possible implementation, the image includes at least a first frame image, a second frame image, and a third frame image in sequence, the number of the detection targets in the first frame image and the second frame image is greater than the number of the detection targets in the third frame image, and the method further includes:

[0023] Based on the position of the missing detection target in the first frame image and the position in the second frame image and a preset image sampling period, obtaining a predicted position of the missing detection target in the third frame image;

[0024] If the predicted position of the missing detection target in the third frame image is consistent with the position of the detection target detected in the third frame image, it is confirmed that the missing detection target is blocked in the third frame image.

[0025] The multi-frame detection algorithm in this example can effectively solve the occlusion problem, improve the accuracy and robustness of multi-target detection, and more accurately detect the positions of targets such as the human body.

[0026] In a second aspect, an embodiment of the present application provides a target detection device based on wireless sensing and image fusion, including:

[0027] a first processing module, configured to acquire an image and perception data, and process the image and the perception data respectively to obtain a first bounding box and a second bounding box corresponding to at least one detection target, and a global feature corresponding to the image and a global feature corresponding to the perception data, wherein the first bounding box corresponds to the image, and the second bounding box corresponds to the perception data;

[0028] A second processing module, configured to obtain local image features and local perception features of at least one detection target in a candidate box set based on global features corresponding to the image and global features corresponding to the perception data, wherein the candidate box set is obtained by fusion processing based on the first bounding box and the second bounding box;

[0029] A determination module is used to process local image features and local perception features of at least one detection target in the candidate frame set to obtain the target.

[0030] In a possible implementation manner, the determining module is used to:

[0031] Performing feature extraction processing on the local image features and the local perception features of each detection target in the at least one detection target in the candidate frame set, respectively, to obtain a first confidence and a second confidence of each detection target, wherein the first confidence corresponds to the local image features, and the second confidence corresponds to the local perception features;

[0032] The first confidence and the second confidence of each detected target are fused to obtain a fused confidence of at least one detected target in the candidate frame set, wherein the fused confidence of the target is greater than a first threshold.

[0033] In a possible implementation manner, the determining module is further configured to:

[0034] Adding the first confidence level and the second confidence level of each detected target respectively to obtain the sum of the confidence levels of each detected target;

[0035] Preset processing is performed on the sum of the confidences of the at least one detection target respectively to obtain a fused confidence of the at least one detection target.

[0036] In a possible implementation manner, the determining module is further configured to:

[0037] It is confirmed that the confidence level of the first bounding box is not less than a second threshold.

[0038] In a possible implementation manner, the image at least includes a first frame image, a second frame image, and a third frame image in sequence, and the determining module is further used to:

[0039] Based on the position of the missing detection target in the first frame image and the position in the second frame image and a preset image sampling period, obtaining a predicted position of the missing detection target in the third frame image;

[0040] If the predicted position of the missing detection target in the third frame image is consistent with the position of the detection target detected in the third frame image, it is confirmed that the missing detection target is blocked in the third frame image.

[0041] In a third aspect, an embodiment of the present application provides a target detection device based on wireless sensing and image fusion, comprising a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute a method provided in any possible implementation manner of the first aspect.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement a method provided in any possible implementation manner of the first aspect.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a computer, the computer executes a method provided in any possible implementation manner of the first aspect.

[0044] It can be understood that the device described in the second aspect, the device described in the third aspect, the computer storage medium described in the fourth aspect, or the computer program product described in the fifth aspect provided above are all used to execute any method provided in the first aspect. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The following is an introduction to the drawings used in the embodiments of the present application.

[0046] Figure 1ais a schematic diagram of a target detection system provided in an embodiment of the present application;

[0047] Figure 1b is a schematic diagram of another target detection system provided in an embodiment of the present application;

[0048] Figure 2 It is a flowchart of a target detection method based on wireless sensing and image fusion provided in an embodiment of the present application;

[0049] Figure 3 It is a flowchart of another target detection method based on wireless sensing and image fusion provided in an embodiment of the present application;

[0050] Figure 4 is a schematic diagram of target detection provided by an embodiment of the present application;

[0051] Figure 5 It is a structural schematic diagram of a target detection device based on wireless sensing and image fusion provided in an embodiment of the present application;

[0052] Figure 6 It is a structural schematic diagram of another target detection device based on wireless sensing and image fusion provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0054] The following will describe the system architecture of the embodiment of the present application in detail with reference to the accompanying drawings. Figure 1a , Figure 1a It is a schematic diagram of a target detection system applicable to an embodiment of the present application, the system comprising an image acquisition device 101, a wireless sensing device 102 and a server 103.

[0055] The image acquisition device 101 is used to acquire images. The image acquisition device 101 may be, for example, a camera device. The camera device may be used to capture image information of an environment. The camera device may include a monocular camera, a binocular camera, a structured light camera, a panoramic camera, etc. The image information acquired by the camera device may include a static image or video stream information.

[0056] The wireless sensing device 102 is used to detect various data. The wireless sensing device 102 can be, for example, various wireless sensors. For example, a laser radar can use lasers to sense objects in the environment. In some embodiments, the laser radar may include one or more laser sources, a laser scanner, and one or more detectors, as well as other system components. A millimeter wave radar can use radio signals to sense objects in the surrounding environment. An ultrasonic radar can use ultrasonic signals to sense surrounding objects.

[0057] The server 103 is a device with centralized computing capabilities. Exemplarily, the server 103 can be implemented by a server, a virtual machine, a cloud or a robot, a base station, or other devices.

[0058] When the server 103 includes a server, the type of the server includes but is not limited to a general-purpose computer, a dedicated server computer, a blade server, etc. This application does not strictly limit the number of servers included in the server 103, and the number can be one or more (such as a server cluster, etc.).

[0059] A virtual machine refers to a computing module that is simulated by software and has complete hardware system functions and runs in a completely isolated environment. Of course, in addition to virtual machines, the server 103 can also be implemented through other computing instances, such as containers.

[0060] The cloud is a software platform that uses application virtualization technology, which enables one or more software and applications to be developed and run in an independent virtualized environment. Optionally, when the server 103 is implemented through the cloud, the cloud can be deployed on a public cloud, a private cloud, or a hybrid cloud.

[0061] The server can also be a base station, an evolved NodeB (eNodeB), a transmission reception point (TRP), a next generation base station (next generation NodeB, gNB) in the fifth generation (5G) mobile communication system, a next generation base station in the sixth generation (6G) mobile communication system, a base station in a future mobile communication system, or an access node in a WiFi system, etc.; it can also be a module or unit that completes part of the functions of a base station, for example, it can be a centralized unit (central unit, CU) or a distributed unit (distributed unit, DU), etc.

[0062] See also Figure 1b , Figure 1bIt is a schematic diagram of another target detection system applicable to the embodiment of the present application, which includes an image acquisition device 101, a wireless sensing device 102 and a terminal 104. Among them, the terminal 104 can also be called a terminal device, a user equipment (UE), a mobile station, a mobile terminal, etc. The terminal can be widely used in various scenarios, for example, device-to-device (D2D), vehicle to everything (V2X) communication, machine-type communication (MTC), Internet of Things (IOT), virtual reality, augmented reality, industrial control, automatic driving, telemedicine, smart grid, smart furniture, smart office, smart wear, smart transportation, smart city, etc. The terminal can be a mobile phone, a tablet computer, a computer with wireless transceiver function, a wearable device, a vehicle, a drone, a helicopter, an airplane, a ship, a robot, a mechanical arm, a smart home device, etc. The embodiment of the present application does not limit the specific technology and specific device form adopted by the terminal.

[0063] The architecture of the embodiment of the present application is described above, and the method of the embodiment of the present application is described in detail below.

[0064] Reference Figure 2 FIG. 1 is a flow chart of a target detection method based on wireless sensing and image fusion provided in an embodiment of the present application. Optionally, the method can be applied to the aforementioned target detection system, for example Figure 1a , Figure 1b The target detection system shown in Figure 1 is Figure 2 The target detection method shown may include steps 201-203. It should be understood that for the convenience of description, this application is described in the order of 201-203, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the execution order, execution time, execution number, etc. of the above one or more steps. The following description takes the execution subject of steps 201-203 of the target detection method as a server as an example, and this application is also applicable to other execution subjects such as terminals. Steps 201-203 are as follows:

[0065] 201. Acquire an image and perception data, and process the image and the perception data respectively to obtain a first bounding box and a second bounding box corresponding to at least one detection target, as well as a global feature corresponding to the image and a global feature corresponding to the perception data.

[0066] It is understandable that the image and the perception data both correspond to at least one detection target in the same scene. For example, the server obtains an image of location A at a certain time from a camera, and obtains perception data (such as point cloud data, etc.) at the same time at location A from a perception device such as a laser radar.

[0067] The first bounding box corresponds to the image, and the second bounding box corresponds to the perception data. That is, the image is processed to obtain a first bounding box corresponding to at least one detection target. The perception data is processed to obtain a second bounding box corresponding to at least one detection target.

[0068] The number of the first bounding box and the second bounding box may be one or more. For example, when there is only one detection target in the image or perception data, the number of the first bounding box and the second bounding box are 1 respectively. When there are multiple detection targets in the image or perception data, the number of the first bounding box and the second bounding box may be more than one respectively.

[0069] For example, when the detection target is a scene of a car, there may be multiple detection targets. Of course, it is also possible to detect a specific car, and this solution does not limit this.

[0070] In a possible implementation, feature extraction processing is performed on the image to obtain an extracted feature map, and then the extracted feature map is processed to obtain a first bounding box corresponding to at least one detection target. Perception tracking processing is performed on the perception data to obtain a second bounding box corresponding to at least one detection target.

[0071] The image and the perception data are processed respectively to obtain global features corresponding to the image and global features corresponding to the perception data. For example, the image and the perception data can be input into a preset neural network model for feature extraction processing to obtain corresponding global features.

[0072] 202. Based on the global features corresponding to the image and the global features corresponding to the perception data, obtain local image features and local perception features of at least one detection target in the candidate frame set.

[0073] The candidate box set is obtained by merging the first bounding box and the second bounding box. The fusion process can be understood as taking the union of the first bounding box and the second bounding box, or obtaining a new candidate box according to the union area division.

[0074] Optionally, the number of candidate boxes in the candidate box set may be the sum of the number of the first bounding box and the number of the second bounding box, or may be less than the sum of the number of the first bounding box and the second bounding box, etc. This solution does not impose any limitation on this.

[0075] Based on the global features corresponding to the image and the global features corresponding to the perception data, the local features (ie, local image features and local perception features) of each detection target can be obtained by using a candidate box cropping operation and a local feature extraction operation.

[0076] In a possible implementation, the confidence of the first bounding box corresponding to the at least one detection target is not less than a preset value. That is, the confidence of the bounding box corresponding to the image is first screened to ensure the reliability of its confidence. Then, the first bounding box and the second bounding box are fused.

[0077] 203. Process local image features and local perception features of at least one detection target in the candidate frame set to obtain a target.

[0078] The target may be one or more of the at least one detection target.

[0079] In a possible implementation, feature extraction processing is performed on local image features and local perception features of each detection target in the at least one detection target in the candidate frame set to obtain a first confidence level and a second confidence level of each detection target.

[0080] The first confidence level corresponds to the local image feature, and the second confidence level corresponds to the local perceptual feature.

[0081] Then, the first confidence and the second confidence of each detection target are fused to obtain a fused confidence of at least one detection target in the candidate frame set.

[0082] Optionally, the first confidence and the second confidence of each detection target are added together to obtain the sum of the confidences of each detection target; and then the sum of the confidences of at least one detection target is subjected to preset processing to obtain a fused confidence of the at least one detection target.

[0083] The preset processing may be, for example, processing through an activation function sigmod.

[0084] Among them, the confidence of the fusion of the target is greater than the first threshold. That is to say, based on the confidence of the fusion of each detection target obtained above, the detection target with a confidence greater than a preset value is screened out, which is the determined target. In this example, the confidence score is obtained based on the candidate box and fused, which can effectively improve the detection accuracy.

[0085] In the embodiment of the present application, by processing the image and the perception data respectively, a first bounding box and a second bounding box corresponding to at least one detection target, as well as global features corresponding to the image and global features corresponding to the perception data are obtained; then, based on the global features corresponding to the image and the global features corresponding to the perception data, local image features and local perception features of at least one detection target in the candidate frame set are obtained; then, based on the local image features and local perception features of at least one detection target, processing is performed to obtain the target. By adopting this method, the candidate frame set obtained based on the image and the perception data is processed, which effectively improves the accuracy of target detection based on perception and image fusion.

[0086] Reference Figure 3 FIG. 1 is a flow chart of another target detection method based on wireless sensing and image fusion provided in an embodiment of the present application. Optionally, the method can be applied to the aforementioned target detection system, for example Figure 1a , Figure 1b The target detection system shown in Figure 1 is Figure 3 The target detection method shown may include steps 301-305. It should be understood that for the convenience of description, this application is described in the order of 301-305, and is not intended to limit the execution to the above order. The embodiment of the present application does not limit the execution order, execution time, execution number, etc. of the above one or more steps. The following description takes the execution subject of steps 301-305 of the target detection method as a server as an example, and this application is also applicable to other execution subjects. Steps 301-305 are as follows:

[0087] 301. Process the image and the perception data to obtain a set of candidate boxes.

[0088] Combination Figure 4 In the first stage shown, the image is processed based on the image detector to obtain a first bounding box with a confidence level greater than a threshold for at least one detected target, and the perception tracker is used to process the perception data to obtain a second bounding box for at least one detected target. Then, the first bounding box and the second bounding box are fused to obtain a set of candidate boxes.

[0089] Specifically, in the image detection process, the image I first passes through the feature extractor F body Extract the internal feature map f = F body (I). Exemplarily, the feature extractor may include a convolution layer, an activation layer, and a pooling layer. The feature map extracted above is passed through the network F head Further processing produces a series of bounding boxes B I The processing flow of the entire image detector can be summarized as B I =F head (Fbody (I)).

[0090] The bounding box B generated by the image detector is further filtered by confidence score, and a bounding box with a confidence score greater than a second threshold is determined as the first bounding box.

[0091] For the perception data, the perception tracker (R) is used to process the second bounding box B R .

[0092] Then, the image detector (I) and the perceptual tracker (R) are fused to obtain the candidate box set B, which can be expressed as:

[0093]

[0094] Among them, K = |B|, which means the number of candidate boxes in the candidate box set B, K is the total number of candidate boxes in the candidate box set, and b k is the kth candidate box in the candidate box set.

[0095] The first stage also processes the image to obtain the global feature G of the image I . Process the perception data to obtain the global feature G of the perception data R .

[0096] That is to say, both the image and perception branches will extract the global features of each branch to obtain the multimodal feature map G I and G R .

[0097] 302. Obtain local image features and local perception features of at least one detection target in the candidate frame set based on the candidate frame set.

[0098] Combination Figure 4 The subnetwork of the candidate box set A in the second stage shown processes the global features of the image to obtain local image features of at least one detection target. Among them, the subnetwork of the candidate box set uses the information learned from a large amount of image data to regress and classify the detection box. For example, in the R-CNN subnet, the feature map of the image is first flattened and then a fully connected layer is applied, and its output results are then used for regression and classification of the bounding box through two fully connected layers. First, the image feature map of each region of interest (RoI) is flattened, and then a fully connected (FC) layer with 256 channels is applied and then two fully connected layers are connected in parallel for regression and classification of the detection box.

[0099] The global features of the perception data are also processed to obtain local perception features of at least one detection target.

[0100] For example, after the candidate box cropping operation, the local image features and local perception features of each region of interest (i.e., the detection target) can be obtained, which can be expressed as:

[0101] L I ,L R =Cropping(G I ,G R ; B);

[0102] in They are respectively a collection of local feature maps of the image and perception modalities.

[0103] Exemplarily, for the perception branch, the point cloud data is projected into two-dimensional image coordinates, and the two-dimensional histogram of the projected point cloud on three channels is calculated. Then, the values ​​on each channel are normalized to the range of [0,1], and a three-channel heat map is obtained. It is input into, for example, a three-layer convolution (Convolutional Neural Network, CNN) to extract features, in which the probability of the target existing at each position is embedded. By cropping it and generating the features of each candidate box, the above-mentioned local perception features are obtained.

[0104] 303. Perform processing based on local image features and local perception features of at least one detection target, respectively, to obtain an image confidence score and a perception confidence score of at least one detection target.

[0105] For the features of each candidate set box, for example, two convolutional layers with kernel sizes of 7×7 and 1×1 can be used to abstract the global representation and obtain a confidence score.

[0106] For an introduction to this part, please refer to the above Figure 2 The introduction of step 203 in the illustrated embodiment will not be repeated here.

[0107] 304. Perform fusion processing based on the image confidence score and the perception confidence score of at least one detected target to obtain a fused confidence score of the at least one detected target.

[0108] like Figure 4 The B feature fusion in the second stage shown, based on the fusion process, obtains a fused confidence score of at least one detected target. The confidence scores of two different modalities are added together and passed through an activation function sigmod to obtain the fused confidence score.

[0109] Among them, feature fusion can predict a new position and confidence score for each candidate box. The predicted bounding box can be expressed as:

[0110]

[0111] The predicted new bounding box is larger than the initial bounding box b because it combines information from two different modalities. k More reliable.

[0112] Among them, although the perceived point cloud is sparse and low-resolution, it can strongly indicate the presence of an object, which is an important supplement to the confidence score of the R-CNN subnet. Specifically, for the radar features of each region of interest, two convolutional layers with kernel sizes of 7×7 and 1×1 are used to abstract the global representation, which is also the confidence score from the radar. The confidence scores of the two different modalities are then added together and passed through an activation function sigmod to obtain the final fused confidence score.

[0113] Furthermore, the second stage also includes a C-integration module. This module is to further explore the ability of the image-based detector to make more reliable predictions.

[0114] The (C+1)-dimensional vectors generated by the R-CNN subnet and the perceptual fusion are of the same dimension as the output of the image detector. In order to merge the two (C+1)-dimensional vectors to obtain more reliable classification decisions, the model introduces a learning-based integration module to integrate the output of the R-CNN joint perceptual fusion with the output of the original image detector. The integration module may include two fully connected layers and an activation layer, and the output results can be used to distinguish foreground from background. The user can determine the list of bounding boxes to be retained by setting a threshold. Intuitively, the greater the variance between categories, the more consistent the classification results between the two input vectors, and the higher the output confidence score. Among them, this module does not utilize the bounding boxes generated by the perceptual tracker.

[0115] 305. Obtain a target based on a fused confidence score of at least one detected target.

[0116] The target can be obtained based on the screening.

[0117] In a possible implementation, the image includes at least a first frame image, a second frame image, and a third frame image in sequence, the number of the detection targets in the first frame image and the second frame image is greater than the number of the detection targets in the third frame image, and the method further includes:

[0118] Based on the position of the missing detection target in the first frame image and the position in the second frame image and a preset image sampling period, obtaining a predicted position of the missing detection target in the third frame image;

[0119] If the predicted position of the missing detection target in the third frame image is consistent with the position of the detection target detected in the third frame image, it is confirmed that the missing detection target is blocked in the third frame image.

[0120] It can be understood that the above-mentioned preset image sampling period is, for example, the difference between the acquisition time of the first frame image and the acquisition time of the second frame image. When the boundary box of a detection target exists in both the first frame image and the second frame image, but is not detected in the continuous third image, the moving speed can be determined based on the position information and time information, and then its position in the third frame image can be predicted. Then it is determined whether the target is blocked.

[0121] That is to say, this solution can also be applied to multi-frame detection, so as to improve the problem of detection frame loss due to occlusion based on detection frame tracking.

[0122] In actual scenes, it is easy for multiple targets to be occluded by each other during detection. At this time, neither the image detector nor the radar tracker can give a good occlusion judgment. To solve this problem, this scheme proposes a target detection method based on multi-frame detection. For example, based on the characteristic that people will not disappear out of thin air in actual scenes, a multi-frame detection algorithm is designed. If the number of people detected in the current frame is less than that in the previous frame, the multi-frame detection module will compare the positions of the detection frames of the previous and next two frames, and combine the moving speeds of different detection frames to determine the lost detection frame. By matching the distance and speed, the position of the detection frame of the occluded part is determined.

[0123] Furthermore, considering that in actual situations, the target may be blocked first, then move away and disappear, thus losing the detection frame for a long time. To solve this problem, when the detection frame is lost for N consecutive frames, it is determined that the target has disappeared.

[0124] The multi-frame detection algorithm in this example can effectively solve the occlusion problem, improve the accuracy and robustness of multi-target detection, and more accurately detect the positions of targets such as the human body.

[0125] For example, a set of image acquisition and wireless sensing data acquisition systems were installed, and a batch of samples containing multimodal data of human body detection were collected. During the experiment, experimental personnel, robots and other moving objects were arranged to walk randomly in front of the sensor, and data containing multiple lighting conditions and multiple targets were collected.

[0126] Among them, (1) under normal lighting conditions, the detection results of the radar and camera fusion model are compared with those of the radar model alone. It is found that the confidence score of the detection results of the radar and camera fusion model is higher than that of the model using only radar; and because the image contains a large number of pixels, the detection box position provided by the image-based detector is more accurate. This is mainly because the camera has advantages in the resolution and semantic information of the image scene, while the millimeter-wave radar has better anti-interference and depth information acquisition capabilities. Therefore, fusing the data of these two sensors can better improve the performance of the target detector, making it have better detection accuracy and robustness.

[0127] (2) The results of the camera and radar fusion model and the image detection model alone in a dark environment were compared. It was found that under low light conditions, the traditional image-based detector could not correctly detect all targets due to the lack of effective information under light conditions. However, the model that fused radar data can still detect targets more accurately, which significantly improves the application effect of the fusion model in dark environments. In actual engineering applications, target detection in dark environments is very important, and traditional image-based methods are affected in this case and have low detection rates. Therefore, the fusion model proposed in this scheme has extremely high practical application value. In short, the above experimental results fully demonstrate the reliability and accuracy of the target detection method based on camera and radar fusion in dark environments, and provide strong support for the realization of intelligent scenes such as efficient transportation.

[0128] In the embodiment of the present application, target detection includes two stages. The first stage is the fusion of the candidate box set: the bounding box generated by the image detector will be aggregated with the bounding box generated by the perception after filtering by the confidence score to form a total candidate box set, which is used to assist in screening the feature map of the perception and image data. The second stage is the confidence score fusion: using the fused candidate box set, the confidence scores of different modalities are obtained, the confidence scores of the two different modalities are added together and the fused confidence score is obtained through an activation function sigmod. By adopting this method, the accuracy of target detection can be improved.

[0129] It should be noted that in the various embodiments of the present application, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the various embodiments are consistent and can be referenced mutually, and the technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0130] The above detailed description of the method of the embodiment of the present application, the following provides a device of the embodiment of the present application. It can be understood that in the various device embodiments of the present application, the division of multiple units or modules is only a logical division according to function, and is not used as a limitation on the specific structure of the device. In a specific implementation, some functional modules may be subdivided into more small functional modules, and some functional modules may also be combined into one functional module, but no matter whether these functional modules are subdivided or combined, the general process performed by the device is the same. For example, some devices contain a receiving unit and a sending unit. In some designs, the sending unit and the receiving unit can also be integrated into a communication unit, which can implement the functions implemented by the receiving unit and the sending unit. Usually, each unit corresponds to its own program code (or program instruction), and when the program code corresponding to each of these units is run on the processor, the unit is controlled by the processing unit to execute the corresponding process to implement the corresponding function.

[0131] An embodiment of the present application also provides a device for implementing any of the above methods. For example, a target detection device based on wireless sensing and image fusion is provided, which includes a module (or means) for implementing each step performed by the server in any of the above methods.

[0132] For example, refer to Figure 5 FIG. 1 is a schematic diagram of a target detection device based on wireless sensing and image fusion provided in an embodiment of the present application. The target detection device is used to implement the aforementioned target detection method, for example Figure 2 , Figure 3 The target detection method shown.

[0133] like Figure 5 As shown, the device may include a first processing module 501, a second processing module 502 and a determination module 503, which are specifically as follows:

[0134] A first processing module 501 is used to acquire an image and perception data, and process the image and the perception data respectively to obtain a first bounding box and a second bounding box corresponding to at least one detection target, as well as a global feature corresponding to the image and a global feature corresponding to the perception data, wherein the first bounding box corresponds to the image and the second bounding box corresponds to the perception data;

[0135] A second processing module 502 is used to obtain local image features and local perception features of the at least one detection target in a candidate box set based on global features corresponding to the image and global features corresponding to the perception data, wherein the candidate box set is obtained by fusion processing based on the first bounding box and the second bounding box;

[0136] The determination module 503 is used to process the local image features and local perception features of the at least one detection target in the candidate frame set to obtain the target.

[0137] In a possible implementation, the determining module 503 is configured to:

[0138] Performing feature extraction processing on the local image features and the local perception features of each detection target in the at least one detection target in the candidate frame set, respectively, to obtain a first confidence and a second confidence of each detection target, wherein the first confidence corresponds to the local image features, and the second confidence corresponds to the local perception features;

[0139] The first confidence and the second confidence of each detected target are fused to obtain a fused confidence of at least one detected target in the candidate frame set, wherein the fused confidence of the target is greater than a first threshold.

[0140] In a possible implementation, the determining module 503 is further configured to:

[0141] Adding the first confidence level and the second confidence level of each detected target respectively to obtain the sum of the confidence levels of each detected target;

[0142] Preset processing is performed on the sum of the confidences of the at least one detection target respectively to obtain a fused confidence of the at least one detection target.

[0143] In a possible implementation, the determining module 503 is further configured to:

[0144] It is confirmed that the confidence level of the first bounding box is not less than a second threshold.

[0145] In a possible implementation, the image at least includes a first frame image, a second frame image, and a third frame image in sequence, and the determining module 503 is further configured to:

[0146] Based on the position of the missing detection target in the first frame image and the position in the second frame image and a preset image sampling period, obtaining a predicted position of the missing detection target in the third frame image;

[0147] If the predicted position of the missing detection target in the third frame image is consistent with the position of the detection target detected in the third frame image, it is confirmed that the missing detection target is blocked in the third frame image.

[0148] For the introduction of the above modules, please refer to the description of the above embodiments, which will not be repeated here.

[0149] It should be understood that the division of each module in each of the above devices is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. In addition, the modules in the target detection device can be implemented in the form of a processor calling software; for example, the target detection device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each module of the device, wherein the processor is, for example, a general-purpose processor, such as a central processing unit (CPU) or a microprocessor, and the memory is a memory inside the device or a memory outside the device. Alternatively, the modules in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits, and the hardware circuits can be understood as one or more processors; for example, in one implementation, the hardware circuit is an application-specific integrated circuit (ASIC), and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be realized by a programmable logic device (PLD), taking a field programmable gate array (FPGA) as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by configuring the configuration file, so as to realize the functions of some or all of the above units. All modules of the above devices can be realized in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the rest by hardware circuits.

[0150] Reference Figure 6 As shown in FIG. 1 , it is a schematic diagram of the hardware structure of another target detection device based on wireless sensing and image fusion provided in an embodiment of the present application. Figure 6 The target detection device 600 shown (the device 600 may be a computer device) includes a memory 601, a processor 602, a communication interface 603 and a bus 604. The memory 601, the processor 602 and the communication interface 603 are connected to each other through the bus 604.

[0151] The memory 601 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM).

[0152] The memory 601 can store programs. When the program stored in the memory 601 is executed by the processor 602, the processor 602 and the communication interface 603 are used to execute the various steps of the target detection method of the embodiment of the present application.

[0153] The processor 602 is a circuit with signal processing capability. In one implementation, the processor 602 may be a circuit with instruction reading and running capability, such as a central processing unit CPU, a microprocessor, a graphics processing unit (GPU) (which may be understood as a microprocessor), or a digital signal processor (DSP); in another implementation, the processor 602 may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit may be fixed or reconfigurable, such as the processor 602 being a hardware circuit implemented by an ASIC or a programmable logic device PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration may be understood as the process of the processor loading instructions to implement the functions of some or all of the above modules. In addition, it may also be a hardware circuit designed for artificial intelligence, which may be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc. The processor 602 is used to execute relevant programs to implement the functions that need to be performed by the units in the target detection device of the embodiment of the present application, or to execute the target detection method of the method embodiment of the present application.

[0154] It can be seen that each module in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0155] In addition, the modules in the above device can be fully or partially integrated together, or can be implemented independently. In one implementation, these modules are integrated together and implemented in the form of a system-on-a-chip (SOC). The SOC may include at least one processor for implementing any of the above methods or implementing the functions of the modules of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0156] The communication interface 603 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the device 600 and other devices or a communication network. For example, data can be acquired through the communication interface 603 .

[0157] The bus 604 may include a path for transmitting information between various components of the device 600 (eg, the memory 601 , the processor 602 , and the communication interface 603 ).

[0158] It should be noted that although Figure 6 The device 600 shown only shows a memory, a processor, and a communication interface, but in the specific implementation process, those skilled in the art should understand that the device 600 also includes other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 600 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 600 may also only include the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 6 All devices shown in .

[0159] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0160] The embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed on a computer or a processor, the computer or the processor executes one or more steps in any of the above methods.

[0161] It should be understood that in the description of the present application, unless otherwise specified, " / " indicates that the objects associated before and after are in an "or" relationship, for example, A / B can represent A or B; wherein A and B can be singular or plural. Also, in the description of the present application, unless otherwise specified, "multiple" refers to two or more than two. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, wherein a, b, c can be single or multiple. In addition, in order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be necessarily different. Meanwhile, in the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete manner for ease of understanding.

[0162] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the division of the unit is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling, direct coupling, or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0163] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. The available medium may be a read-only memory (ROM), or a random access memory (RAM), or a magnetic medium, such as a floppy disk, a hard disk, a tape, a disk, or an optical medium, such as a digital versatile disc (DVD), or a semiconductor medium, such as a solid state disk (SSD), etc.

[0165] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A target detection method based on wireless sensing and image fusion, characterized in that: include: Acquire an image and perception data, and process the image and the perception data respectively to obtain a first bounding box and a second bounding box corresponding to at least one detection target, as well as a global feature corresponding to the image and a global feature corresponding to the perception data, wherein the first bounding box corresponds to the image and the second bounding box corresponds to the perception data; Based on the global features corresponding to the image and the global features corresponding to the perception data, obtaining local image features and local perception features of the at least one detection target in a candidate box set, wherein the candidate box set is obtained by fusion processing based on the first bounding box and the second bounding box; The local image features and local perception features of at least one detection target in the candidate frame set are processed to obtain the target.

2. The method according to claim 1, characterized in that The processing of the local image features and the local perception features of the at least one detection target in the candidate frame set to obtain the target includes: Performing feature extraction processing on the local image features and the local perception features of each detection target in the at least one detection target in the candidate frame set, respectively, to obtain a first confidence and a second confidence of each detection target, wherein the first confidence corresponds to the local image features, and the second confidence corresponds to the local perception features; The first confidence and the second confidence of each detected target are fused to obtain a fused confidence of at least one detected target in the candidate frame set, wherein the fused confidence of the target is greater than a first threshold.

3. The method according to claim 2, characterized in that The fusing the first confidence level and the second confidence level of each detection target to obtain a fused confidence level of at least one detection target in the candidate frame set includes: Adding the first confidence level and the second confidence level of each detected target respectively to obtain the sum of the confidence levels of each detected target; Preset processing is performed on the sum of the confidences of the at least one detection target respectively to obtain a fused confidence of the at least one detection target.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: It is confirmed that the confidence level of the first bounding box is not less than a second threshold.

5. The method according to any one of claims 1 to 4, characterized in that: The image at least includes a first frame image, a second frame image and a third frame image in sequence, the number of the detection targets in the first frame image and the second frame image is greater than the number of the detection targets in the third frame image, and the method further includes: Based on the position of the missing detection target in the first frame image and the position in the second frame image and a preset image sampling period, obtaining a predicted position of the missing detection target in the third frame image; If the predicted position of the missing detection target in the third frame image is consistent with the position of the detection target detected in the third frame image, it is confirmed that the missing detection target is blocked in the third frame image.

6. A target detection device based on wireless sensing and image fusion, characterized in that: include: a first processing module, configured to acquire an image and perception data, and process the image and the perception data respectively to obtain a first bounding box and a second bounding box corresponding to at least one detection target, and a global feature corresponding to the image and a global feature corresponding to the perception data, wherein the first bounding box corresponds to the image, and the second bounding box corresponds to the perception data; A second processing module, configured to obtain local image features and local perception features of at least one detection target in a candidate box set based on global features corresponding to the image and global features corresponding to the perception data, wherein the candidate box set is obtained by fusion processing based on the first bounding box and the second bounding box; A determination module is used to process local image features and local perception features of at least one detection target in the candidate frame set to obtain the target.

7. The device according to claim 6, characterized in that The determining module is used to: Performing feature extraction processing on the local image features and the local perception features of each detection target in the at least one detection target in the candidate frame set, respectively, to obtain a first confidence and a second confidence of each detection target, wherein the first confidence corresponds to the local image features, and the second confidence corresponds to the local perception features; The first confidence and the second confidence of each detected target are fused to obtain a fused confidence of at least one detected target in the candidate frame set, wherein the fused confidence of the target is greater than a first threshold.

8. The device according to claim 7, characterized in that The determining module is further used for: Adding the first confidence level and the second confidence level of each detected target respectively to obtain the sum of the confidence levels of each detected target; Preset processing is performed on the sum of the confidences of the at least one detection target respectively to obtain a fused confidence of the at least one detection target.

9. The device according to any one of claims 6 to 8, characterized in that The determining module is further used for: It is confirmed that the confidence level of the first bounding box is not less than a second threshold.

10. The device according to any one of claims 6 to 9, characterized in that The image at least includes a first frame image, a second frame image and a third frame image in sequence, and the determination module is further used to: Based on the position of the missing detection target in the first frame image and the position in the second frame image and a preset image sampling period, obtaining a predicted position of the missing detection target in the third frame image; If the predicted position of the missing detection target in the third frame image is consistent with the position of the detection target detected in the third frame image, it is confirmed that the missing detection target is blocked in the third frame image.

11. A target detection device based on wireless sensing and image fusion, characterized in that: It comprises a processor and a memory; wherein the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method according to any one of claims 1 to 5.

13. A computer program product, characterized in that When the computer program product runs on a computer, the computer is caused to execute the method according to any one of claims 1 to 5.

Images