Target detection method, equipment, storage medium and product based on radar and vision fusion

Abstract

The present invention discloses a target detection method, device, storage medium and product based on radar-visual fusion, wherein the target detection method comprises obtaining radar target detection information and visual target detection information; generating heterogeneous images according to the radar target detection information and the visual target detection information; extracting features from the heterogeneous images to obtain feature vectors of radar targets and feature vectors of visual targets; matching the feature vectors of the radar targets and the feature vectors of visual targets to obtain matching results. Compared with the traditional rule-based fusion method, the present invention performs radar-visual fusion based on the metric of feature vectors, which has better simplicity, learnability and generalization, improves the target detection efficiency, and solves the problem that the traditional method needs to manually design association rules, which is time-consuming and labor-intensive, and lacks generalization.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, and in particular, relates to a target detection method, device, storage medium and product based on radar and vision fusion. Background Art

[0002] Radar and visual sensors are currently more mature, economical and stable sensors, and have been widely used in perimeter security, traffic monitoring, autonomous driving and other fields. Based on their complementary characteristics, the outputs of radar detection and visual perception are usually fused to generate the best perception results.

[0003] Generally speaking, radar-vision fusion algorithms can be divided into three categories: data-level fusion, feature-level fusion, and decision-level fusion. Both data-level fusion and feature-level fusion combine radar and visual sensor information to directly detect objects of interest. These methods often require high-precision ground truth information, which is not sufficient for most scenarios. As for decision-level fusion, traditional target object fusion methods tend to manually design various distance metrics to represent the similarity between different sensor outputs, and apply distance minimization or other heuristic rules to perform data association; in order to cope with environmental complexity and uncertainty, probabilistic models are sometimes used in the association process. However, these methods have the disadvantages of limited information provided by manually designed rules, cumbersome design, time-consuming and labor-intensive, and insufficient generalization, making it difficult to cope with various complex situations or growing data.

[0004] For traditional decision-level fusion methods, radar and vision fusion is generally achieved by combining rule-based association algorithms with kinematic model-based tracking. The key is the data association between radar and visual detection. However, the clutter, noise interference and point cloud sparsity of radar detection information, as well as the depth ambiguity of a single camera, make this association problem very challenging. The association process of traditional methods is generally manually designed based on minimizing certain distance metrics and some heuristic rules. This not only requires a lot of human engineering and tuning, but also is difficult to adapt to the growing data, and the generalization is difficult to meet actual requirements.

[0005] Another newer type of decision-level fusion method is to use deep learning-based methods to replace rule-based radar-vision fusion. It mainly focuses on using the combined data of radar and vision as the input of the neural network to directly detect targets. However, these methods all rely on high-precision ground-truth information such as lidar information to establish a high-precision connection between radar and vision. For public data, the corresponding information may have been provided, making it convenient to conduct experiments. However, for actual application scenarios, similar conditions are usually not available due to various restrictions. Summary of the invention

[0006] The purpose of the present invention is to provide a target detection method, device, storage medium and product based on radar and vision fusion to solve at least one of the problems that traditional methods require manual design of association rules, which is time-consuming and labor-intensive and lacks generalization, and the problem that traditional deep learning methods require high-precision ground or scene information.

[0007] The present invention solves the above technical problems through the following technical solutions: a target detection method based on radar-visual fusion, comprising:

[0008] Obtain radar target detection information and visual target detection information;

[0009] generating a heterogeneous image based on the radar target detection information and the visual target detection information;

[0010] Extracting features from the heterogeneous images to obtain feature vectors of radar targets and feature vectors of visual targets;

[0011] The feature vector of the radar target and the feature vector of the visual target are matched to obtain a matching result.

[0012] Furthermore, the specific process of acquiring the visual target detection information includes:

[0013] Adjusting the parameters of the visual sensor according to the radar target detection information so that the radar target is completely within the field of view of the visual sensor;

[0014] Acquire first image data collected by the visual sensor, perform region detection on the first image data, and obtain a mask map of the visual target activity area;

[0015] Adjusting the parameters of the visual sensor according to the visual target activity area mask map so that the visual target activity area is within the visual field of the visual sensor to the greatest extent;

[0016] Acquire second image data collected by the visual sensor, perform target detection on the second image data, and obtain visual target detection information.

[0017] Further, a pre-trained multi-task detection model is used to perform region detection on the first image data and target detection on the second image data; wherein the multi-task detection model includes a backbone network, a feature pyramid network, an upsampling layer, a multi-detection head network and a segmentation head network;

[0018] The backbone network is used to extract features from the first image data or the second image data;

[0019] The feature pyramid network is used to extract features of different scales from the features extracted by the backbone network;

[0020] The multi-detection head network is used to detect features of different scales extracted by the feature pyramid network to obtain target detection results;

[0021] The upsampling layer is used to upsample the features of different scales extracted by the feature pyramid network so that the feature size is the same as the input size;

[0022] The segmentation head network is used to segment the features output by the upsampling layer to obtain mask images of the target active area and other areas.

[0023] Furthermore, the training process of the multi-task detection model includes:

[0024] Constructing a sample data set; wherein the sample data set includes a plurality of first samples and a plurality of second samples, each of the first samples includes a first image sample and its target bounding box and target category, and each of the second samples includes a second image sample and its target activity area;

[0025] The multi-task detection model is trained in three stages using the sample data set, specifically including:

[0026] First, freeze the upsampling layer and segmentation head network, and use all the first samples to train the backbone network, feature pyramid network and multi-detection head network;

[0027] Then freeze the backbone network and the multi-detection head network, and use all the second samples to train the feature pyramid network, upsampling layer, and segmentation head network;

[0028] Finally, all the first samples and the second samples are used to fine-tune the multi-task detection model to obtain a trained multi-task detection model.

[0029] Further, generating a heterogeneous image according to the radar target detection information and the visual target detection information specifically includes:

[0030] Mapping the radar target detection information into an image according to a calibrated coordinate system conversion matrix;

[0031] Based on the radar target detection information and the visual target detection information in the image, a single-channel heat map is generated at the corresponding position of the image;

[0032] Generate heterogeneous images based on multiple single-channel heat maps and the RGB three-channel data of the original image.

[0033] Further, a pre-trained neural network model is used to extract features from the heterogeneous image; wherein the neural network model includes a feature extraction module, a feature fusion module and a feature generation module;

[0034] The feature extraction module is used to extract features from the heterogeneous image;

[0035] The feature fusion module is used to fuse the features extracted by the feature extraction module to obtain a feature map;

[0036] The feature generation module is used to evenly divide the feature map into two sub-feature maps along the channel direction, and extract the feature vector of the position corresponding to the radar target detection information from one of the sub-feature maps to obtain the feature vector of the radar target; and extract the feature vector of the position corresponding to the visual target detection information from the other sub-feature map to obtain the feature vector of the visual target.

[0037] Furthermore, during the training process of the neural network model, the loss function is designed as:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] in, represents the total loss value, represents the positive sample loss value, represents the negative sample loss value, represents the weight parameter used to balance the loss, represents the ordinal loss, , Respectively represent the positive sample set and negative sample set in each frame image, , Represents the positive sample set The number of samples and the set of negative samples The number of samples, represents the association pair consisting of the i-th radar target and the j-th visual target, represents the feature vector of the i-th radar target, represents the feature vector of the jth visual target, , All represent pre-set hyperparameters. represents the predicted positive correlation sample set, Represents a positive correlation sample set The number of samples, Represents a random associated pair The distance of radar target i, Represents a random associated pair The target distance after the visual target j is converted to the radar coordinate system, Represents a random associated pair The y-axis pixel coordinate of the target bounding box after the radar target i is converted to the visual coordinate system, represents the y-axis pixel coordinate of the bounding box of visual target j, Represents the activation function.

[0043] Furthermore, matching the feature vector of the radar target with the feature vector of the visual target specifically includes:

[0044] Calculating a matching matrix based on the feature vector of the radar target and the feature vector of the visual target; wherein each element in the matching matrix represents the similarity between the feature vector of the radar target and the feature vector of the visual target;

[0045] Selecting the radar target and the visual target corresponding to the minimum element in each row or column of the matching matrix to match and obtain a matching pair;

[0046] Calculating the distance difference between the radar target and the visual target in each of the matching pairs;

[0047] When the distance difference is less than the distance threshold, the matching pair is a credible match and matching target detection information is obtained; otherwise, the matching pair is an untrustworthy match and is filtered.

[0048] Furthermore, the target detection method further includes performing secondary detection on the unmatched radar target and the unmatched visual target respectively;

[0049] Perform secondary detection on unmatched radar targets, including:

[0050] According to the calibrated coordinate system transformation matrix, the unmatched radar target detection information is mapped into the image;

[0051] Determine a local area according to a mapping position of unmatched radar target detection information in the image;

[0052] Cut out the local area from the original image, and perform target detection on the cut out local area image to obtain an output result of whether there is a radar target in the local area image;

[0053] Whether the radar target exists is determined based on the output result of whether the radar target exists in the local area image and the original confidence of the radar target.

[0054] Perform secondary detection on unmatched visual targets, including:

[0055] According to the bounding box of the unmatched visual target, a local area image corresponding to the bounding box is intercepted from the original image;

[0056] Performing target detection on the captured local area image to obtain an output result of whether there is a visual target in the local area image;

[0057] Whether the visual target exists is determined based on the output result of whether the visual target exists in the local area image and the original confidence of the visual target.

[0058] Based on the same concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the target detection method based on radar-vision fusion as described above.

[0059] Based on the same concept, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the target detection method based on radar-visual fusion as described above is implemented.

[0060] Based on the same concept, the present invention also provides a computer program product, including a computer program / instruction, which implements the target detection method based on radar and vision fusion as described above when executed by a processor. Beneficial Effects

[0061] Compared with the prior art, the advantages of the present invention are:

[0062] The present invention generates heterogeneous images according to radar target detection information and visual target detection information, extracts feature vectors of radar targets and feature vectors of visual targets from the heterogeneous images respectively, and performs radar-vision fusion based on the measurement of feature vectors. Compared with the traditional rule-based fusion method, the present invention has better simplicity, learnability and generalization, improves the target detection efficiency, and solves the problem that the traditional method needs to manually design association rules, which is time-consuming and labor-intensive and lacks generalization.

[0063] The present invention guides the visual sensor to adjust the azimuth and magnification based on the radar target detection information, and guides the visual sensor to adjust the pitch based on the visual image segmentation result (i.e., the regional mask image), so that the radar target can appear completely in the field of view of the visual sensor with a suitable size and position, and be displayed on the image, so as to better perform radar and visual fusion and improve the target detection accuracy.

[0064] The present invention designs a multi-task detection model that simultaneously realizes target detection and semantic segmentation. While realizing the detection of targets of interest, it also completes the segmentation of the area where the target may appear in the image, thereby guiding the visual sensor to focus on this area and improving the target detection accuracy.

[0065] For unmatched radar targets or visual targets, the present invention performs secondary target detection on the corresponding local area image, reconstructs the target confidence information based on the radar-visual fusion result (i.e., the matching result) and the secondary target detection result, and further reduces the possibility of false alarms and missed detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0067] Figure 1 is a flow chart of a target detection method based on radar-visual fusion in an embodiment of the present invention;

[0068] Figure 2 1 is a diagram of the architecture of a multi-task detection model in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following is a clear and complete description of the technical solution in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0070] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Example

[0071] like Figure 1 As shown, a target detection method based on radar-visual fusion provided by an embodiment of the present invention comprises the following steps:

[0072] Step 1: Get radar target detection information.

[0073] The radar is used to scan the detection area to obtain radar target detection information in the detection area; wherein the radar target detection information S i radar Including the distance R of the radar target i radar 、Direction A i radar and confidence C i radar , the superscript radar represents the radar target, the subscript i represents the radar target number, and the radar target refers to the target detected by the radar.

[0074] Step 2: Based on the radar target detection information, obtain the visual target detection information.

[0075] In order to improve the target detection accuracy, the radar target detection information is used to guide the visual sensor to adjust the parameters. The visual sensor of this embodiment is a high-definition camera. In a specific embodiment of the present invention, based on the radar target detection information, obtaining the visual target detection information specifically includes:

[0076] Step 2.1: Adjust the parameters of the visual sensor according to the radar target detection information so that the radar target is completely within the field of view of the visual sensor;

[0077] Step 2.2: Acquire the first image data collected by the visual sensor, perform area detection on the first image data, and obtain a mask map of the visual target activity area;

[0078] Step 2.3: Adjust the parameters of the visual sensor according to the visual target activity area mask map so that the visual target activity area is within the visual sensor's field of view to the greatest extent;

[0079] Step 2.4: Acquire the second image data collected by the visual sensor, perform target detection on the second image data, and obtain visual target detection information.

[0080] In this embodiment, the magnification value of the visual sensor is adjusted according to the distance of the radar target: when the distance of the radar target is small, the magnification value of the visual sensor is reduced; when the distance of the radar target is large, the magnification value of the visual sensor is increased. The correspondence between the distance and the magnification value can be obtained by pre-calibration on site, so that the magnification value that needs to be adjusted can be known according to the distance of the radar target, and the magnification value can be adjusted. The azimuth angle of the visual sensor is adjusted according to the azimuth of the radar target: during installation, the relationship between the zero angle of the radar and the zero angle of the visual sensor gimbal is known. Through calibration and compensation, the zero angle of the radar and the zero angle of the visual sensor gimbal can be adjusted to be consistent. At this time, the azimuth of the radar target detected by the radar is the angle to which the gimbal of the visual sensor should be rotated.

[0081] By adjusting the magnification value and azimuth angle of the visual sensor, the radar target can appear as completely as possible within the field of view of the visual sensor and be displayed in the image.

[0082] In step 2.2, a pre-trained multi-task detection model is used to perform region detection on the first image data; in step 2.4, a pre-trained multi-task detection model is used to perform object detection on the second image data. Figure 2As shown, the multi-task detection model includes a backbone network, a feature pyramid network, an upsampling layer, a multi-detection head network and a segmentation head network; the backbone network is connected to the feature pyramid network, the feature pyramid network is connected to the upsampling layer and the multi-detection head network, and the upsampling layer is connected to the segmentation head network.

[0083] The backbone network is used to extract features from the first image data or the second image data; the feature pyramid network is used to extract features of different scales from the features extracted by the backbone network; the multi-detection head network is used to detect the features of different scales extracted by the feature pyramid network to obtain target detection results; the upsampling layer is used to upsample the features of different scales extracted by the feature pyramid network to make the feature size the same as the size of the input image; the segmentation head network is used to segment the features output by the upsampling layer to obtain mask images of the target active area and other areas. The target active area corresponds to the foreground, and the other areas correspond to the background. For example, for ground targets, the foreground is the ground area and the background is the sky.

[0084] In a specific embodiment of the present invention, the training process of the multi-task detection model includes:

[0085] Step A1: construct a sample data set; wherein the sample data set includes multiple first samples and multiple second samples, each first sample includes a first image sample and its target bounding box and target category, and each second sample includes a second image sample and its target activity area.

[0086] The annotation information of the target bounding box includes the pixel coordinates, width and height of the center point of the bounding box, and the annotation information of the target active area includes the pixel point set at the edge of the area.

[0087] Step A2: Use the sample dataset to train the multi-task detection model in three stages, including:

[0088] Step A21: Freeze the upsampling layer and the segmentation head network, and use all the first samples to train the backbone network, feature pyramid network, and multi-detection head network;

[0089] Step A22: Freeze the backbone network and the multi-detection head network, and use all the second samples to train the feature pyramid network, the upsampling layer, and the segmentation head network;

[0090] Step A23: Use all the first samples and the second samples to fine-tune the multi-task detection model based on a small training rate to obtain a trained multi-task detection model.

[0091] The multi-task detection model needs to perform multi-task detection (i.e., target activity area detection and target detection). Therefore, during training, it is necessary to first train for different tasks and then fine-tune for balance.

[0092] In step 2.2, the first image data is detected using a pre-trained multi-task detection model, which can not only obtain the visual target activity area mask map, but also obtain the visual target detection information. The visual target detection information at this time is not used in subsequent steps, but the visual target detection information obtained in step 2.4 is used. In step 2.4, the second image data is detected using a pre-trained multi-task detection model, which can not only obtain the visual target detection information, but also obtain the visual target activity area mask map. Since the visual target activity area is already within the field of view of the visual sensor to the greatest extent, the visual target activity area mask map is not used at this time.

[0093] The parameters of the visual sensor are adjusted according to the visual target activity area mask map: the pitch angle of the visual sensor is adjusted in a feedback manner by adjusting the proportion of the visual target activity area mask map in the image collected by the visual sensor until the proportion reaches an optimal value, such as 3 / 4, which can be set according to the on-site environment. When the proportion reaches the optimal value, the visual target activity area can be within the visual sensor's field of view to the greatest extent and displayed in the image.

[0094] In this embodiment, the visual target detection information includes the pixel coordinates of the center point of the bounding box of the visual target (x j camera ,y j camera )、Width w j camera and height h j camera , target category CLA j camera and confidence C j camera , the superscript camera represents the visual target, the subscript j represents the visual target number, and the visual target refers to the target collected by the visual sensor.

[0095] Step 3: Generate heterogeneous images based on radar target detection information and visual target detection information.

[0096] In a specific embodiment of the present invention, generating a heterogeneous image according to radar target detection information and visual target detection information specifically includes:

[0097] Step 3.1: Map the radar target detection information into the image according to the calibrated coordinate system transformation matrix.

[0098] Through calibration, the coordinate system conversion matrix (i.e., the conversion matrix from the radar coordinate system to the world coordinate system, the matrix from the world coordinate system to the camera coordinate system, and the matrix from the camera coordinate system to the pixel coordinate system) can be obtained. The radar can not only detect the distance R of the radar targeti radar and direction A i radar , the spatial position of the radar target can also be detected. Specifically, the spatial position of the radar target is calculated according to the distance and azimuth of the radar target. The specific formula is:

[0099] x ri =r ri cosθ azii sinθ elei (1)

[0100] y ri =r ri sinθ azii sinθ elei (2)

[0101] z ri =r ri cosθ elei (3)

[0102] Among them, (x ri ,y ri ,z ri ) represents the spatial position of the ith radar target in the radar coordinate system, and the coordinate axes of the radar coordinate system follow the standard three-dimensional Cartesian coordinates; θ azii represents the horizontal azimuth angle of the i-th radar target in the radar coordinate system, where the rightmost horizontal end of the detectable area is the horizontal azimuth angle of zero degrees; θ elei represents the vertical pitch angle of the ith radar target in the radar coordinate system, where vertical upward is the vertical pitch angle of zero degrees; r ri It represents the distance of the i-th radar target from the coordinate origin (i.e., radar device) in the radar coordinate system.

[0103] When mapping radar target detection information to an image, the spatial position of the radar target is first converted to the world coordinate system according to the coordinate system conversion matrix, then converted to the camera coordinate system (i.e., the visual coordinate system), and finally converted to the image pixel coordinate system, thus obtaining the pixel coordinates of the radar target in the pixel coordinate system, that is, mapping the radar target detection information to the image.

[0104] Step 3.2: Based on the radar target detection information and visual target detection information in the image, generate a single-channel heat map at the corresponding position of the image.

[0105] Radar target detection information S i radar Including the distance R of the radar target i radar 、Direction A i radar and confidence Ci radar , a single-channel heat map is generated according to the value of each detection information at the corresponding position where the radar target is mapped to the image. Exemplarily, a single-channel heat map is generated according to the distance value of the radar target at the corresponding position where the radar target is mapped to the image. The larger the distance value, the brighter the color. Thus, a single-channel heat map of the detection information of three radar targets can be obtained.

[0106] Visual object detection information includes the width w of the bounding box of the visual object j camera and height h j camera , target category CLA j camera and confidence C j camera , then a single-channel heat map is generated according to the value of each detection information at the corresponding position of the visual target in the image. Thus, a single-channel heat map of the detection information of four visual targets can be obtained.

[0107] Step 3.3: Generate a heterogeneous image based on multiple single-channel heat maps and the RGB three-channel data of the original image.

[0108] A 10-channel heterogeneous image is composed of three single-channel heat maps of radar target detection information, four single-channel heat maps of visual target detection information, and the RGB three-channel data of the original image. That is, the size of the heterogeneous image is Cin×H×W, where Cin represents the number of channels of the heterogeneous image, H represents the height of the heterogeneous image, and W represents the width of the heterogeneous image.

[0109] Step 4: Extract features from heterogeneous images to obtain feature vectors of radar targets and feature vectors of visual targets.

[0110] In a specific embodiment of the present invention, a pre-trained neural network model is used to extract features from heterogeneous images; wherein the neural network model includes a feature extraction module, a feature fusion module, and a feature generation module. In this embodiment, the neural network model uses a backbone network as a feature extraction module, and extracts features from heterogeneous images through the backbone network; and uses a feature pyramid network as a feature fusion module. Unlike the feature pyramid network in the multi-task detection model, the feature pyramid network of the neural network model only outputs a feature map of one scale, and the features extracted by the feature extraction module are fused through the feature pyramid network to obtain a feature map. The feature map Cout×H×W output by the feature fusion module has the same size as the heterogeneous image Cin×H×W input to the feature extraction module, and Cout represents the number of channels of the feature map.

[0111] In order to obtain the feature vector, the feature generation module is used to evenly divide the feature map into two sub-feature maps along the channel direction, and the feature vector corresponding to the radar target detection information is extracted from one of the sub-feature maps C1×H×W to obtain the feature vector V of the radar target. Ri , with a size of 1×C1; extract the feature vector of the corresponding position of the visual target detection information from another sub-feature map C2×H×W to obtain the feature vector V of the visual target Cj , the size is 1×C2, C1=C2=1 / 2×Cout.

[0112] The neural network model is used to extract features from heterogeneous images to obtain the feature vector V of the radar target. Ri and the feature vector V of the visual target Cj Previously, it was necessary to first train the neural network model using heterogeneous image samples, and use the designed damage function and back propagation to enable the neural network model to perform feature learning. In a specific embodiment of the present invention, the designed loss function is:

[0113] (4)

[0114] (5)

[0115] (6)

[0116] (7)

[0117] in, represents the total loss value, represents the positive sample loss value, represents the negative sample loss value, represents an adjustable weight parameter used to balance the loss, represents the ordinal loss, , Respectively represent the positive sample set and negative sample set in each frame image, , Represents the positive sample set The number of samples and the set of negative samples The number of samples, represents the association pair consisting of the i-th radar target and the j-th visual target, represents the feature vector of the i-th radar target, represents the feature vector of the jth visual target, , All represent pre-set hyperparameters. represents the predicted positive correlation sample set, Represents a positive correlation sample set The number of samples, Represents a random associated pair The distance of radar target i, Represents a random associated pair The target distance after the visual target j is converted to the radar coordinate system, Represents a random associated pair The y-axis pixel coordinate of the target bounding box after the radar target i is converted to the visual coordinate system, represents the y-axis pixel coordinate of the bounding box of visual target j, Represents the activation function and is used to smooth the loss value. , They respectively represent the threshold of the "expected representation distance" of positively correlated samples and the threshold of the "expected representation distance" of negatively correlated samples. The "expected representation distance" means that the feature vectors obtained after training have a certain representation distance. It is expected that the representation distance between positively correlated samples is smaller, while the representation distance between negatively correlated samples is larger.

[0118] Positive sample loss and negative sample loss It is based on the loss of Ray-view matching, and then an ordinal loss is introduced , that is, given the radar target detection information and visual target detection information to be associated, generally speaking, a farther radar target will be associated with a farther visual target, a closer radar target will be associated with a closer visual target, and radar targets with a far-near relationship should have the same far-near relationship when mapped to the image.

[0119] Step 5: Match the feature vector of the radar target with the feature vector of the visual target to obtain a matching result.

[0120] In a specific embodiment of the present invention, matching the feature vector of the radar target with the feature vector of the visual target specifically includes:

[0121] Step 5.1: Calculate the matching matrix based on the feature vector of the radar target and the feature vector of the visual target.

[0122] Each element in the matching matrix represents the similarity between the feature vector of the radar target and the feature vector of the visual target (i.e., the distance between the feature vectors). The matching matrix is ​​obtained by calculating the cosine similarity between the feature vector of the i-th radar target and the feature vector of the j-th visual target.

[0123] Step 5.2: Select the radar target and visual target corresponding to the minimum element in each row or column of the matching matrix to match and obtain a matching pair.

[0124] During the matching process, it may happen that each visual target is associated with information of multiple radar targets or each radar target is associated with information of multiple visual targets. In order to achieve a one-to-one matching between visual targets and radar targets, the radar target corresponding to the minimum element in each row or column of the matching matrix is ​​selected to match the visual target. For each element M in the matching matrix, ij , which represents the similarity between the feature vector of the i-th radar target and the feature vector of the j-th visual target. The i-th row of the matching matrix represents the similarity between the feature vector of the i-th radar target and the feature vector of each visual target. The minimum element in the i-th row is selected, and the minimum element is used to determine which visual target matches the i-th radar target. Similarly, the j-th column of the matching matrix represents the similarity between the feature vector of each radar target and the feature vector of the j-th visual target. The minimum element in the j-th column is selected, and the minimum element is used to determine which radar target matches the j-th visual target.

[0125] Step 5.3: Calculate the distance difference between the radar target and the visual target in each matching pair; when the distance difference is less than the distance threshold, the matching pair is a credible match and the matching target detection information is obtained; otherwise, the matching pair is an untrustworthy match and is filtered.

[0126] In order to ensure the accuracy of matching, after matching based on the similarity between the feature vectors, judgment is also made based on the physical distance to ensure that the physical distance difference between the radar target and the visual target in each matching pair is less than the distance threshold.

[0127] Step 6: Perform secondary detection on the unmatched radar targets and unmatched visual targets respectively.

[0128] After matching the radar target with the visual target according to the feature vector of the radar target and the feature vector of the visual target, three matching results are obtained, the first is the radar target that is not matched with the visual target, the second is the visual target that is not matched with the radar target, and the third is the radar target and the visual target are successfully matched to obtain a matching pair. In order to prevent the phenomenon of false alarm and missed detection of the target, the present invention also performs secondary detection on the unmatched radar target and the unmatched visual target respectively.

[0129] Before the secondary detection, a target binary classification model is designed and trained to obtain a trained target binary classification model. The image to be detected is input into the trained target binary classification model to predict whether there is a target in the image to be detected. The target binary classification model can be based on traditional machine learning models such as support vector machines, or a classification model based on a neural network.

[0130] In a specific embodiment of the present invention, performing secondary detection on unmatched radar targets includes:

[0131] Step 6.11: Map the unmatched radar target detection information into the image according to the calibrated coordinate system transformation matrix;

[0132] Step 6.12: Determine the local area according to the mapping position of the unmatched radar target detection information in the image;

[0133] Step 6.13: intercept the local area from the original image, and input the intercepted local area image into the trained target binary classification model to obtain an output result of whether the local area image contains a radar target;

[0134] Step 6.14: Determine whether the radar target exists based on the output result of whether the radar target exists in the local area image and the original confidence of the radar target.

[0135] A rectangular box with fixed width and height is set as the width and height of the local area, and the y-axis coordinate of the maximum pixel of the mapping position is used as the bottom edge of the local area. The output result of whether there is a radar target in the local area image includes the confidence C of the radar target i class , the confidence C of the radar target i class (i.e., the confidence predicted by the target binary classification model) and the original confidence C of the radar target i radar (i.e., the confidence level detected by the radar) is weighted and summed to obtain the final confidence level C of the radar target. i total When the final confidence level of the radar target C i total When it is greater than or equal to the confidence threshold, it indicates that the radar target exists. The weighting coefficient of the weighted summation can be set according to the statistical information of the radar detection accuracy. If the actual radar detection result is more reliable, the weighting coefficient of the original confidence is larger; if it cannot be determined, the two weighting coefficients are equal, both of which are 0.5.

[0136] In a specific embodiment of the present invention, performing secondary detection on the unmatched visual target includes:

[0137] Step 6.21: According to the bounding box of the unmatched visual target, a local area image corresponding to the bounding box is intercepted from the original image;

[0138] Step 6.22: Input the captured local area image into the trained target binary classification model to obtain an output result of whether the local area image contains a visual target;

[0139] Step 6.23: Determine whether the visual target exists based on the output result of whether the visual target exists in the local area image and the original confidence of the visual target.

[0140] The output result of whether there is a visual target in the local area image includes the confidence C of the visual target j class , the confidence C of the visual target j class (i.e., the confidence of the target binary classification model prediction) and the original confidence C of the visual target j camera (i.e., the confidence output by the multi-task detection model) is weighted summed to obtain the final confidence C of the visual target j total When the final confidence C of the visual target j total When it is greater than or equal to the confidence threshold, it indicates that the visual target exists. The weighting coefficient of the weighted sum can be set according to the statistical information of the detection accuracy of the multi-task detection model. If the detection result of the multi-task detection model is more reliable, the weighting coefficient of the original confidence is larger; if it cannot be determined, the two weighting coefficients are equal, both of which are 0.5.

[0141] The confidence of the matching pair is high, so no secondary detection is performed. It is directly assumed that the confidence C of the target binary classification model prediction ij class 1, and the confidence of the matching pair (C i radar , C j camera ) and the predicted C ij class Obtain the final confidence C in an average weighted manner ij total , that is: C ij total =0.5*C ij class +0.25*C i radar +0.25*C j camera .

[0142] Example 2

[0143] An embodiment of the present invention also provides an electronic device, which includes: a memory, a processor, and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the target detection method based on radar-vision fusion in the embodiment of the present application.

[0144] Although not shown, the electronic device includes a processor that can perform various appropriate operations and processes according to the programs and / or data stored in the read-only memory (ROM) or the programs and / or data loaded from the storage part into the random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general-purpose main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for device operation are also stored. The processor, ROM, and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0145] The processor and the memory are used together to execute the program / instructions stored in the memory. When the program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0146] Although not shown, an embodiment of the present invention further provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the target detection method based on radar-visual fusion in an embodiment of the present application.

[0147] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0148] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the target detection method based on radar-vision fusion in the embodiment of the present application.

[0149] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A target detection method based on radar and vision fusion, characterized in that: The target detection method comprises: Obtain radar target detection information and visual target detection information; generating a heterogeneous image based on the radar target detection information and the visual target detection information; Extracting features from the heterogeneous images to obtain feature vectors of radar targets and feature vectors of visual targets; Matching the feature vector of the radar target with the feature vector of the visual target to obtain a matching result; The generating of heterogeneous images according to the radar target detection information and the visual target detection information specifically includes: Mapping the radar target detection information into an image according to a calibrated coordinate system conversion matrix; Based on the radar target detection information and the visual target detection information in the image, a single-channel heat map is generated at the corresponding position of the image; Generate heterogeneous images based on multiple single-channel heat maps and RGB three-channel data of the original image; Using a pre-trained neural network model to extract features from the heterogeneous image; wherein the neural network model includes a feature extraction module, a feature fusion module and a feature generation module; The feature extraction module is used to extract features from the heterogeneous image; The feature fusion module is used to fuse the features extracted by the feature extraction module to obtain a feature map; The feature generation module is used to evenly divide the feature map into two sub-feature maps along the channel direction, and extract the feature vector of the position corresponding to the radar target detection information from one of the sub-feature maps to obtain the feature vector of the radar target; and extract the feature vector of the position corresponding to the visual target detection information from the other sub-feature map to obtain the feature vector of the visual target.

2. The target detection method based on radar and visual fusion according to claim 1, characterized in that: The specific process of obtaining the visual target detection information includes: Adjusting the parameters of the visual sensor according to the radar target detection information so that the radar target is completely within the field of view of the visual sensor; Acquire first image data collected by the visual sensor, perform region detection on the first image data, and obtain a mask map of the visual target activity area; Adjusting the parameters of the visual sensor according to the visual target activity area mask map so that the visual target activity area is within the visual field of the visual sensor to the greatest extent; Acquire second image data collected by the visual sensor, perform target detection on the second image data, and obtain visual target detection information.

3. The target detection method based on radar and visual fusion according to claim 2 is characterized in that: Performing region detection on the first image data and object detection on the second image data using a pre-trained multi-task detection model; wherein the multi-task detection model includes a backbone network, a feature pyramid network, an upsampling layer, a multi-detection head network, and a segmentation head network; The backbone network is used to extract features from the first image data or the second image data; The feature pyramid network is used to extract features of different scales from the features extracted by the backbone network; The multi-detection head network is used to detect features of different scales extracted by the feature pyramid network to obtain target detection results; The upsampling layer is used to upsample the features of different scales extracted by the feature pyramid network so that the feature size is the same as the input size; The segmentation head network is used to segment the features output by the upsampling layer to obtain mask images of the target active area and other areas.

4. The target detection method based on radar and visual fusion according to claim 3 is characterized in that: The training process of the multi-task detection model includes: Constructing a sample data set; wherein the sample data set includes a plurality of first samples and a plurality of second samples, each of the first samples includes a first image sample and its target bounding box and target category, and each of the second samples includes a second image sample and its target activity area; The multi-task detection model is trained in three stages using the sample data set, specifically including: First, freeze the upsampling layer and segmentation head network, and use all the first samples to train the backbone network, feature pyramid network and multi-detection head network; Then freeze the backbone network and the multi-detection head network, and use all the second samples to train the feature pyramid network, upsampling layer, and segmentation head network; Finally, all the first samples and the second samples are used to fine-tune the multi-task detection model to obtain a trained multi-task detection model.

5. The target detection method based on radar and visual fusion according to claim 1, characterized in that: During the training process of the neural network model, the designed loss function is: ； ； ； ； in, represents the total loss value, represents the positive sample loss value, represents the negative sample loss value, represents the weight parameter used to balance the loss, represents the ordinal loss, , Respectively represent the positive sample set and negative sample set in each frame image, , Represents the positive sample set The number of samples and the set of negative samples The number of samples, represents the association pair consisting of the i-th radar target and the j-th visual target, represents the feature vector of the i-th radar target, represents the feature vector of the jth visual target, , All represent pre-set hyperparameters. represents the predicted positive correlation sample set, Represents a positive correlation sample set The number of samples, Represents a random associated pair The distance of radar target i, Represents a random associated pair The target distance after the visual target j is converted to the radar coordinate system, Represents a random associated pair The y-axis pixel coordinate of the target bounding box after the radar target i is converted to the visual coordinate system, represents the y-axis pixel coordinate of the bounding box of visual target j, Represents the activation function.

6. The target detection method based on radar and visual fusion according to claim 1, characterized in that: Matching the feature vector of the radar target with the feature vector of the visual target specifically includes: Calculating a matching matrix based on the feature vector of the radar target and the feature vector of the visual target; wherein each element in the matching matrix represents the similarity between the feature vector of the radar target and the feature vector of the visual target; Selecting the radar target and the visual target corresponding to the minimum element in each row or column of the matching matrix to match and obtain a matching pair; Calculating the distance difference between the radar target and the visual target in each of the matching pairs; When the distance difference is less than the distance threshold, the matching pair is a credible match and matching target detection information is obtained; otherwise, the matching pair is an untrustworthy match and is filtered.

7. The target detection method based on radar and visual fusion according to any one of claims 1 to 6, characterized in that: The target detection method further comprises performing secondary detection on the unmatched radar target and the unmatched visual target respectively; Perform secondary detection on unmatched radar targets, including: According to the calibrated coordinate system transformation matrix, the unmatched radar target detection information is mapped into the image; Determine a local area according to a mapping position of unmatched radar target detection information in the image; Cut out the local area from the original image, and perform target detection on the cut out local area image to obtain an output result of whether there is a radar target in the local area image; Determine whether the radar target exists according to the output result of whether the radar target exists in the local area image and the original confidence of the radar target; Perform secondary detection on unmatched visual targets, including: According to the bounding box of the unmatched visual target, a local area image corresponding to the bounding box is intercepted from the original image; Performing target detection on the captured local area image to obtain an output result of whether there is a visual target in the local area image; Whether the visual target exists is determined based on the output result of whether the visual target exists in the local area image and the original confidence of the visual target.

8. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the target detection method based on radar and vision fusion as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the target detection method based on radar and vision fusion as described in any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the target detection method based on radar and vision fusion as described in any one of claims 1 to 7 is implemented.

Images