Target detection method and device based on millimeter wave radar and visible light fusion

Abstract

The application discloses a target detection method and device based on millimeter wave radar and visible light fusion, the method comprises the following steps: converting radar data into a radar image, so that the convolution network can better extract effective features from the sparse form of millimeter wave radar data. A radar feature extraction method based on deformable convolution is used to reduce the feature fusion interference caused by the registration error of the millimeter wave radar image and the visible light image; a normalized attention mechanism is used to effectively reduce the influence of radar image virtual scene interference through a sparse weight penalty mechanism; in view of the problems of difficult effective feature extraction, clutter false alarm interference and the like in the current water surface target detection algorithm, the application can extract effective features from the complex water surface environment and more accurately extract target objects.

Description

TECHNICAL FIELD

[0001] The present application relates to target monitoring technology in water surface environment, and particularly relates to a target detection method and device based on millimeter wave radar and visible light fusion in water surface harsh environment. BACKGROUND

[0002] As one of the core problems of computer vision, target detection aims to find the category and location of specific targets in an image, and has been widely applied in various fields, such as automatic driving, remote sensing image, video monitoring and medical detection. Water surface target detection is mainly applied to water surface detection equipment or unmanned surface equipment, among which unmanned surface vessels (USV) play an important role in water surface processing fields such as water surface rescue, garbage detection and terrain detection. Laser radar is generally used for target recognition in target detection, but due to the poor semantic information of laser radar point cloud and too many false alarm points of water waves, the recognition accuracy of this kind of algorithm is not enough. In addition, the existing target detection also uses a target recognition method based on three-dimensional laser radar, which has too high data dimension and great operation difficulty, and it is difficult to distinguish the target type. In addition, the water surface target detection method using visible light RGB image may cause missed detection of the target when the effective features of the target are blocked in the case that the visible light image is blocked. SUMMARY

[0003] The present application provides a target detection method and device based on millimeter wave radar and visible light fusion to at least solve the foregoing technical problems.

[0004] According to a first aspect of an embodiment of the present application, a target detection method based on millimeter wave radar and visible light fusion is provided, comprising:

[0005] Based on the millimeter wave radar, a first image of a target region in a water surface environment is collected, three-dimensional data in a radar coordinate system in the first image is converted into two-dimensional image data in a camera coordinate, and the distance, Doppler intensity and energy intensity in the radar image data are respectively corresponding to the RGB of the image channel;

[0006] A visible light image of the target region is collected, and a de-distortion processing is performed on the visible light image to obtain a to-be-processed image;

[0007] First feature data of the radar two-dimensional image data is extracted, and second feature data of the to-be-processed image is extracted, a normalization algorithm is used to suppress the extracted features, and a proportional factor measuring the importance of the features is added to the first feature data and the second feature data respectively;

[0008] The first feature data and the second feature data are fused, and the fused feature data is subjected to target confidence loss, class confidence loss and coordinate regression loss processing to obtain certain feature data;

[0009] Based on the certain feature data, the target object is detected and recognized, and the target object in the water surface environment is extracted.

[0010] In an implementable manner, after converting the three-dimensional data in the radar coordinate system in the image into radar image data in the camera coordinate system, the method further comprises:

[0011] The two-dimensional data in the radar coordinate system in the first image is subjected to variable convolution, that is, a displacement variable of the two-dimensional data is added to the convolution layer of the two-dimensional data, and the receptive field range of the two-dimensional data is changed through the displacement variable, so that the receptive field range is a polygon;

[0012] The displacement variable is obtained through online learning according to the two-dimensional data, or the displacement variable is obtained according to the preset value corresponding to different application scenarios.

[0013] In an implementable manner, the method further comprises adding a scale factor for measuring the importance of the data in the following manner:

[0014]

[0015] Wherein, BN() is a batch normalization operation, B in is the input three-dimensional data, and are the mean and standard deviation of the small batch β respectively, γ and β are trainable affine transformation parameters, and ∈ is a preset constant.

[0016] In an implementable manner, the method further comprises processing the fused feature data through target confidence loss, class confidence loss and coordinate regression loss in the following manner:

[0017]

[0018]

[0019] Wherein, and represent the binary cross-entropy loss, P0 represents the target confidence score in the prediction box, P IOU represents the IOU value of the target box corresponding to the prediction box, w obj and w cls are the weights of positive samples, IoU represents the intersection over union between two boundary rectangular boxes, α represents the weight coefficient, ν represents the consistency of the relative proportion of two rectangular boxes, and b and b gtrepresents the center points of the two boundary rectangular frames, p represents the Euclidean distance between the two boundary rectangular frames, and c represents the distance of the diagonal line of the closure region of the two boundary rectangular frames.

[0020] In an implementation, the de-distortion processing of the visible light image comprises:

[0021] The radial distortion of the image is determined in the following manner:

[0022] x dr = x (1 + k1r 2 + k2r 4 + k3r 6 )

[0023] y dr = y (1 + k1r 2 + k2r 4 + k3r 6 )

[0024] r 2 = x 2 +y 2

[0025] where (x, y) is the ideal non-distorted image coordinate system coordinate, (x dr , y dr ) is the coordinate of the distorted image pixel point, k1, k2, and k3 are distortion coefficients, which are obtained during camera calibration;

[0026] r 2 = x 2 +y 2

[0027] The tangential distortion of the image is determined in the following manner:

[0028] x dt = 2p1xy + p2(r 2 + 2x 2 ) + 1

[0029] y dt = 2p1(r 2 + 2y 2 ) + 2p2xy + 1

[0030] where (x dt , y dt ) is the coordinate of the distorted image pixel point, p1 and p2 are distortion coefficients, which are obtained during camera calibration;

[0031] The sum of the radial distortion and the tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined in the following manner:

[0032] xd = x dr + x dt

[0033] y d = y dr + y dt

[0034] undistort the visible light image according to the determined distortion parameters to obtain an undistorted visible light image.

[0035] According to a second aspect of the present application, a target detection device based on millimeter wave radar and visible light fusion is provided, comprising:

[0036] a radar image processing unit, configured to collect a first image of a target region in a water surface environment based on a millimeter wave radar, convert three-dimensional data in a radar coordinate system in the first image into two-dimensional image data in a camera coordinate, and make the distance, Doppler intensity and energy intensity in the radar image data correspond to RGB of an image channel respectively;

[0037] a collection unit, configured to collect a visible light image of the target region, and perform undistortion processing on the visible light image to obtain a to-be-processed image;

[0038] a feature processing unit, configured to extract first feature data of the radar two-dimensional image data, and extract second feature data of the to-be-processed image, suppress the extracted features using a normalization algorithm, and add a proportion factor that measures the importance of the features to the first feature data and the second feature data respectively;

[0039] a loss processing unit, configured to fuse the first feature data and the second feature data, and perform target confidence loss, class confidence loss and coordinate regression loss processing on the fused feature data to obtain confident feature data;

[0040] a recognition unit, configured to detect and recognize a target object based on the confident feature data, and extract the target object in the water surface environment.

[0041] In an implementation manner, the device further comprises:

[0042] a convolution unit, configured to perform variability convolution on two-dimensional data in the radar coordinate system in the first image, that is, add a displacement variable of the two-dimensional data on a convolution layer of the two-dimensional data, change a receptive field range of the two-dimensional data through the displacement variable, and make the receptive field range of the two-dimensional data be a polygon;

[0043] wherein the displacement variable is obtained through online learning according to the two-dimensional data, or the displacement variable is obtained according to a preset value corresponding to different application scenarios.

[0044] In an implementable mode, the feature processing unit is further configured to:

[0045] The data is added with a scale factor of importance of the measured feature in the following manner:

[0046]

[0047] wherein BN() is a batch normalization operation, B in is the input three-dimensional data, and are the mean and standard deviation of the small batch β respectively, γ and β are trainable affine transformation parameters, and ∈ is a preset constant.

[0048] In an implementable mode, the loss processing unit is further configured to process the fused feature data in the following manner:

[0049]

[0050] wherein, and denote the binary cross-entropy loss, P0 denotes the target confidence score in the prediction box, P IOU denotes the IOU value of the target box corresponding to the prediction box, w obj and w cls are the weights of positive samples, IoU denotes the intersection over union between two bounding rectangular boxes, α denotes the weight coefficient, ν denotes the consistency of the relative proportion of two rectangular boxes, b and b gt denote the center points of two bounding rectangular boxes, ρ denotes the Euclidean distance between two bounding rectangular boxes, and c denotes the distance of the diagonal line of the closure region of two bounding rectangular boxes.

[0051] In an implementable mode, the acquisition unit is further configured to:

[0052] The radial distortion of the image is determined in the following manner:

[0053] x dr = x (1 + k1r 2 + k2r 4 + k3r 6 )

[0054] y dr = y (1 + k1r 2 + k2r 4 + k3r 6 )

[0055] r 2 = x 2 +y 2

[0056] wherein (x, y) is the ideal non-distorted image coordinate system coordinate, (x dr ,y dr ) is the coordinate of the image pixel point after distortion, k1, k2, k3 are distortion coefficients, which are obtained when calibrating the camera;

[0057] r 2 =x 2 +y 2

[0058] The tangential distortion of the image is determined in the following manner:

[0059] x dt =2p1xy+p2(r 2 +2x 2 )+1

[0060] y dt =2p1(r 2 +2y 2 )+2p2xy+1

[0061] wherein (x dt ,y dt ) is the coordinate of the image pixel point after distortion, p1, p2 are distortion coefficients, which are obtained when calibrating the camera;

[0062] The sum of the radial distortion and the tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined in the following manner:

[0063] x d =x dr +x dt

[0064] y d =y dr +y dt

[0065] The visible light image is de-distorted by using the determined distortion parameters, to obtain a de-distorted visible light image.

[0066] The technical scheme provided by the embodiments of the present application can have the following beneficial effects:

[0067] The application converts the millimeter wave image data collected by the radar into two-dimensional radar image data, and completes registration with the visible light image, improves the target monitoring and recognition accuracy in the water surface environment, reduces the error influence existing in the registration of the millimeter wave radar image and the visible light image by adding a scale factor to the feature data, provides a deformable convolution parameter through online learning, and extracts accurate and reliable radar features. Through the special processing method of normalized attention, the false alarm interference of the water surface clutter radar image is reduced, the selected radar information is more reliable and effective, the feature weight of the virtual scene in the image is reduced, and the recognition accuracy of the target object in the water surface harsh environment such as motion blur, water mist interference, target shielding and the like is greatly improved.

[0068] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the application. BRIEF DESCRIPTION OF DRAWINGS

[0069] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the application, and together with the specification, serve to explain the principles of the application.

[0070] Figure 1 The flowchart of the target detection method based on millimeter wave radar and visible light fusion shown by an embodiment of the application;

[0071] Figure 2 The radar image generation diagram of the target object shown by an embodiment of the application;

[0072] Figure 3 The image processing method diagram of the radar module shown by an embodiment of the application;

[0073] Figure 4 The image processing result diagram shown by an embodiment of the application;

[0074] Figure 5 The composition structure diagram of the target detection device based on millimeter wave radar and visible light fusion shown by an embodiment of the application;

[0075] Figure 6 The structure diagram of the electronic device of an embodiment of the application. DETAILED DESCRIPTION

[0076] The exemplary embodiments will be described in detail herein with reference to the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the application. Instead, they are merely examples of apparatuses and methods consistent with some aspects of the application as detailed in the appended claims.

[0077] The exemplary embodiments will be described in detail below with reference to the drawings. In the following description, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments are not meant to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0078] Figure 1 A flowchart of a target detection method based on millimeter wave radar and visible light fusion according to an embodiment of the present disclosure is shown in FIG. 1. The target detection method based on millimeter wave radar and visible light fusion according to an embodiment of the present disclosure includes the following steps: Figure 1

[0079] In step 101, a first image of a target region in a water surface environment is acquired based on millimeter wave radar, and three-dimensional data in a radar coordinate system in the first image is converted into radar image data in a camera coordinate, so that the distance, Doppler intensity, and energy intensity in the radar image data correspond to RGB of an image channel, respectively.

[0080] In the embodiment of the present disclosure, due to the imaging characteristics of the millimeter wave radar, the returned data is three-dimensional data, which cannot be directly input into a two-dimensional convolution network and is difficult to be fused with image data. In the present disclosure, a projection conversion method of radar image is adopted to project the three-dimensional data to a pixel coordinate system through the internal and external parameters of the camera. First, the radar point in the three-dimensional radar coordinate is converted to the corresponding position in the camera coordinate by formula (1). Wherein, X r is the 3D position information in the radar coordinate, R is the rotation matrix, T is the translation matrix, and X i is the position of the radar point cloud in the pixel coordinate system.

[0081] X i = X r R + T (1)

[0082] The radar image data contains three dimensions, i.e. the distance of the target (range), the radial velocity of the target (doppler), and the signal strength of the target (power), whose amplitude sizes are different, which cannot make the network better understand. And the amplitude difference with the visible light image is large, which is not conducive to the subsequent feature fusion.

[0083] The embodiment of the present disclosure proposes a calculation method of matrix mapping, and the calculation method is shown in formula (2).

[0084]

[0085] ​Wherein, the matrix A is a 3*3 mapping matrix, the matrix C is a 1*3 parameter matrix, according to the amplitude range of the radar data, the radar image is mapped into the radar image with the amplitude comparable to the visible light RGB amplitude.

[0086] After generating the radar image, in order to highlight the importance difference of different target points, according to the Euclidean distance value of the radar data, the radar points are expanded to different degrees to form solid circles of different sizes, as shown in the following formula (3).

[0087]

[0088] Wherein, r dis The radius of the generated solid circle is represented by r, and a k The data gain is represented by a, dis represents the Euclidean distance of the radar data, and x, y and z are the positions of the target point in the three-dimensional space in the radar coordinate system.

[0089] At the same time, since the radar center point has high confidence, a Gaussian filtering algorithm is added, as shown in the following formula (4):

[0090]

[0091] Wherein, R, G and B are the channel values of the generated solid circle image. G(R, G, B) is the channel value obtained after Gaussian filtering transformation; δ 2 The variance is represented by δ.

[0092] In step 102, the visible light image of the target region is collected, and the visible light image is de-distorted to obtain a to-be-processed image.

[0093] In the embodiment of the application, the visible light image can be collected by an industrial camera, and the target region in the water surface environment can be a water area where the target object to be identified is located. Due to the movement of the water surface boat, wind or the influence of the sea tide, some bad environments often appear on the water surface, resulting in motion blur, water mist interference, target shielding and the like in the collected image. Therefore, the identification result will be very poor by using the conventional target object identification method. The embodiment of the application is proposed for the identification of the target object in the water surface environment. By using the related technical means of the embodiment of the application, the identification error of the target object can be greatly reduced, and the identification accuracy of the target object can be improved.

[0094] Specifically, the imaging process is essentially the conversion of several coordinate systems. First, a point in space is converted from the world coordinate system to the camera coordinate system, then it is projected onto the imaging plane (image physical coordinate system), and finally the data on the imaging plane is converted to the image plane (image pixel coordinate system). Due to the distortion of the camera in a series of processes such as camera manufacturing: radial distortion and tangential distortion.

[0095] The radial distortion is determined as shown in equation (5): for a lens, taking the center of the lens as the origin, and going outward is the direction of the radius of the lens, the closer the light ray is to the center, the smaller the distortion is, and the farther the light ray is from the center along the radial direction, the larger the distortion is.

[0096]

[0097] where (x, y) is the ideal non-distorted coordinate (image coordinate system), (x dr ,y dr ) is the coordinate of the image pixel point after distortion. k1, k2, k3 are distortion coefficients.

[0098] The tangential distortion of the image is determined as shown in equation (6): the tangential distortion is the distortion generated when the lens and the imaging plane are not in line, which is similar to the perspective transformation.

[0099] x dt =2p1xy+p2(r 2 +2x 2 )+1

[0100] y dt =2p1(r 2 +2y 2 )+2p2xy+1 (6)

[0101] where (x, y) is the ideal non-distorted coordinate (image coordinate system), (x dt ,y dt ) is the coordinate of the image pixel point after distortion. p1, p2 are distortion coefficients.

[0102] The sum of the radial distortion and the tangential distortion is taken as the overall distortion, and the overall visible light image distortion is determined by the following method:

[0103] x d =x dr +x dt

[0104] y d =y dr +y dt

[0105] Those skilled in the art should understand that steps 101 and 102 are steps executed in parallel, which can be started at the same time when image acquisition is performed on the target area. Of course, within a certain time difference range, they can also be executed in sequence. Steps 101 and 102 are only for identification convenience, and are not a time sequence related limitation.

[0106] In step 103, first feature data of the radar two-dimensional image data is extracted, and second feature data of the image to be processed is extracted, the extracted features are suppressed using a normalization algorithm, and a proportional factor measuring the importance of the features is added to the first feature data and the second feature data, respectively.

[0107] In the embodiments of the present application, since there is a certain error in the internal parameter calibration data in the camera calibration process, and the camera distortion types are various and cannot be completely removed by calibration, a certain deviation exists in the process of converting the millimeter wave point cloud image to the radar RGB image. Considering the above problems, deformable convolution is introduced. The deformable convolution adds a displacement variable on the convolution layer, and the variable is learned online according to the data. After introducing the variable, the range of the receptive field is changed, and the receptive field becomes a polygon.

[0108] Specifically, as shown in Figure 3 , the two-dimensional data in the radar coordinate system in the first image is subjected to deformable convolution, that is, a displacement variable of the two-dimensional data is added to the convolution layer of the two-dimensional data, the range of the receptive field of the two-dimensional data is changed through the displacement variable, and the range of the receptive field is a polygon; wherein the displacement variable is obtained through online learning according to the two-dimensional data; or the displacement variable is obtained according to the preset value corresponding to different application scenarios. Figure 3 In the deformable convolution, Conv represents convolution operation, Maxpooling represents maximum pooling operation, the idea of moving window is used to move on the image, the maximum value in the window is selected and recorded, and finally the corresponding output is obtained.

[0109] The radar image feature extraction module of the embodiment of the present application is composed of three parts of a convolution module, a pooling module, and a feature enhancement module. For the input 3-channel RGB image, first, the feature is preliminarily extracted through four convolutions, normalization, and an activation layer, and the size of the feature map is gradually reduced through this way. Since the information amount in the image feature is large, the expression ability of the network for complex features is enhanced through the multi-channel separated convolution. The channel separated convolution way is composed of two branches, one branch uses 1x1 convolution to change the channel, and the other branch first uses 1x1 convolution to change the channel and then passes through four 3x3 convolutions to extract the feature. The size of the feature map is reduced and the parameters of the degradation model are reduced using the maximum pooling way to ensure that rich feature information can be extracted at different scales. The maximum pooling module is composed of two branches, one branch first passes through maximum pooling and then passes through a 1x1 convolution to change the number of channels, and the other branch first passes through a 1x1 convolution to change the number of channels and then passes through a 3x3 convolution kernel with a step of 2 to down-sample. The image feature extraction module finally generates three feature maps of different scales. These feature maps capture different levels and scales of information in the input image, providing a rich feature basis for the subsequent target detection task.

[0110] wherein, 1x1Conv: a convolution kernel with a size of 1x1 is used for convolution, mainly used for the number of layers of the feature image.

[0111] Maxpooling3x3: the convolution kernel of the convolution operation has data weight, and the pooling directly calculates the original data in the pooling window, and the calculation process can be to select the maximum value.

[0112] wherein, Deformable conv represents deformable convolution processing, BN is a batch normalization operation, that is, BatchNormalization operation, the BN layer standardizes the distribution of each batch of data, so that each layer of the neural network can learn a relatively stable input distribution, thereby alleviating the internal covariate shift problem, and thus more suitable for the model of the present application embodiment in which the input data distribution changes constantly during training. Act represents an activation function, which improves the nonlinear fitting ability of the model by introducing a nonlinear variable. The embodiment of the present application improves the operation structure of the BN layer by introducing convolution operation to improve the local feature extraction and calculation efficiency.

[0113] Specifically, the proportion factor measuring the importance of the feature is added to the data in the following way:

[0114]

[0115] wherein, BN() is a batch normalization operation, Bin is the input three-dimensional data, and are the mean and standard deviation of the small batch β respectively, γ and β are trainable affine transformation parameters, ∈ is a preset constant, which is a minimum value, used to prevent the denominator from being zero without affecting the overall calculation result.

[0116] In step 104, the first feature data and the second feature data are fused, and the fused feature data is processed for target confidence loss, class confidence loss and coordinate regression loss to obtain confident feature data.

[0117] In the embodiments of the present application, the fused feature data is processed for target confidence loss, class confidence loss and coordinate regression loss in the following manner:

[0118]

[0119] wherein, and is the binary cross-entropy loss, P0 is the target confidence score in the prediction box, P IOU is the IOU value of the target box corresponding to the prediction box, w obj and w cls is the weight of the positive sample, IoU represents the intersection over union between two boundary rectangular boxes, α represents the weight coefficient, ν represents the consistency of the relative proportion of two rectangular boxes, b and b gt represent the center points of two boundary rectangular boxes, ρ represents the Euclidean distance between two boundary rectangular boxes, and c represents the distance of the diagonal line of the closure region of two boundary rectangular boxes.

[0120] In step 105, the target object is detected and recognized based on the confident feature data, and the target object under the water surface environment is extracted.

[0121] The essence of the technical solutions of the embodiments of the present application is further illustrated by specific examples.

[0122] The water surface environment target detection method based on a millimeter wave radar and a visible light camera provided in the embodiments of the present application is used for input 4D millimeter wave radar data and visible light camera images. First, the 4D millimeter wave radar data is projected to a camera coordinate system through a radar multi-channel mapping image generation algorithm, and is converted into a radar RGB image through camera intrinsic parameters. Subsequently, the registered radar RGB image and the visible light camera image are respectively processed through a visible light data processing branch and a radar feature extraction branch of a neural network such as YOLOv7. Specifically, the embodiments of the present application precisely adjust the processing details, which are described in detail below, so as to obtain corresponding radar RGB feature maps and visible light feature maps. Then, the radar RGB feature maps are further extracted through a normalization attention mechanism. Finally, the two types of feature maps are subjected to a detection head of YOLOv7 for target detection and identification.

[0123] Specifically, the embodiments of the present application use four convolution operations to preliminarily extract low-level features while gradually reducing the size of the feature maps. Considering the relationship between the shortest and longest gradient paths of each layer and the longest gradient path of the entire network during feature extraction, an ELAN (Efficient Layer Aggregation Network) module is used to enhance the representation ability of the network for complex features. An MP module is used to ensure that rich feature information can be extracted at different scales. The ELAN module consists of two branches. One branch uses a 1x1 convolution to change the number of channels, and the other branch first uses a 1x1 convolution to change the number of channels and then uses four 3x3 convolutions to extract features. The MP module also consists of two branches. One branch first undergoes maximum pooling and then a 1x1 convolution to change the number of channels, and the other branch first undergoes a 1x1 convolution to change the number of channels and then undergoes a convolution block with a 3x3 kernel and a stride of 2 for down-sampling. Through the feature extraction module, three feature maps of different scales are finally generated. These feature maps capture different levels and scales of information in the input image, providing a rich feature basis for subsequent target detection tasks.

[0124] Due to errors in the intrinsic calibration data during the camera calibration process and the variety of camera distortion types that cannot be completely removed through calibration, there is a certain offset in the process of converting the millimeter wave point cloud image to the radar RGB image. Considering the above problems, the embodiments of the present application calibrate the image data by setting a radar rectification feature extraction module (Radar Rectify Feature Block) and introducing deformable convolution to process the converted data. Specifically, a displacement variable is added to the convolution layer in deformable convolution, which is learned online according to the data. After introducing the variable, the range of the receptive field is changed, making the receptive field a polygon.

[0125] The radar feature extraction module is used to extract feature points of the image, and the processing manner is as follows:

[0126] In order to reduce the interference of noise and improve the anti-interference ability of the radar data processing branch, the normalization attention module is added in the processing model. The normalization attention mechanism is a high-performance and lightweight attention mechanism. The channel and spatial attention sub-modules are integrated from the CBAM module and redesigned. That is, the batch normalization scaling factor (BN) is added on the basis of the original, as shown in formula (7), which uses the scaling factor to measure the variance of the channel and indicates the importance of the feature.

[0127]

[0128] Among them, and are the mean and standard deviation of the small batch β, γ and β are trainable affine transformation parameters, ∈ is a preset constant, which is a very small value, which is used to prevent the denominator from being zero without affecting the overall calculation result.

[0129] The batch normalization scaling factor is added in the original channel attention module, and the output function is as shown in formula (8). Among them represents the weight, F1 is the input, M C is the output of the channel attention module. Similarly, the batch normalization scaling factor is added in the original spatial attention module to obtain the improved spatial attention mechanism, and the output formula is as shown in formula (9). Among them represents the weight, F2 is the input, M s is the output of the channel attention module.

[0130] M c =sigmoid(W γ (BN(F1))) (8)

[0131] M s =sigmoid(W λ (BN s (F2))) (9)

[0132] In order to evaluate the accuracy of the detection algorithm, the loss function is used to improve the detection precision. The total loss function is usually composed of the following parts:

[0133] Classification loss: used to measure the difference between the predicted class label and the true label.

[0134] Bounding box regression loss: measures the difference between the predicted bounding box and the ground truth bounding box.

[0135] Confidence loss: measures the prediction accuracy of the model's confidence in the presence of the target

[0136] Target confidence loss Loss obj : The target confidence loss refers to the accuracy of the model in predicting whether a target exists in a certain grid cell. This loss reflects the degree of confidence of the model in the presence or absence of a target. The confidence score is a value between 0 and 1, representing the probability that the model considers the bounding box to contain a certain target.

[0137] Class confidence loss Loss cls : Each grid cell may predict multiple bounding boxes, and each bounding box will have a class probability distribution. These probability distributions represent the probability of the bounding box containing each class. The class confidence loss calculates the loss by comparing the predicted class probability distribution with the true class label.

[0138] Coordinate regression Loss bbox : Each predicted bounding box is represented by four parameters: center coordinates (x, y) and width, height (w, h). The coordinate regression loss is used to calculate the difference between the model's predicted parameters and the true values. As shown in equation (10).

[0139] Loss = Loss obj × α obj + Loss cls × α cls + Loss bbox × α box (10)

[0140] where α obj , α cls , α box are preset parameters.

[0141] Both the target confidence loss and the class confidence loss are binary cross-entropy losses. The target confidence loss is calculated by the sample pair obtained by matching the positive samples. P0 represents the confidence score of the target in the predicted box, and P IOU represents the IOU value of the target box corresponding to the predicted box. The binary cross-entropy is calculated to obtain the final target confidence loss. As shown in equation (11). The class confidence loss calculates the class loss by the class score c p of the predicted box and the target box class c gt . As shown in equation (12).

[0142]

[0143] wherein, with denoted as binary cross-entropy loss, w obj with w cls denoted as the weight of positive samples.

[0144] The coordinate regression loss uses CIoU Loss. CIoU Loss considers the overlapping area, the center point distance and the aspect ratio, and penalizes the inconsistency between the predicted shape and the ground truth, as shown in equation (13).

[0145]

[0146] wherein, a represents a weight coefficient, v represents the consistency of the relative proportion of the two rectangular frames, b and b gt represent the center points of the two rectangular frames, p represents the Euclidean distance between the two rectangular frames, and c represents the distance of the diagonal line of the closure region of the two rectangular frames.

[0147] In order to verify the effectiveness and advancement of the technical solutions of the embodiments of the present application, the embodiments of the present application are compared with the Achelous algorithm, and the average accuracy (mAP) and the target recognition accuracy (AP) are used as evaluation indexes, and the indexes are the larger the better. The experimental results are shown in Table 1. When the selected IOU=0.5 is used as the standard for testing the performance of the target recognition algorithm, the average accuracy of the Achelous algorithm under this index is 76.2%, while the average accuracy (MAP) of the embodiments of the present application reaches 89.3%, and the accuracy is improved by 13.1%, which shows that compared with the existing research algorithm, the algorithm proposed in the embodiments of the present application has leading target detection performance.

[0148] Table 1 Comparison of algorithm type accuracy

[0149] Algorithm Sensor Mean Average Precision (mAP) Achelous Millimeter wave radar + camera 76.2％ RIF-YOLO Millimeter wave radar + camera 89.3％

[0150] In addition, the embodiments of the present application also compare the detection accuracy (AP) of the two algorithms under different categories of the data set, and the results are shown in Table 2. From the table, it can be concluded that compared with the Achelous algorithm, the RIF-YOLO algorithm has a significant improvement in the detection accuracy of each category, among which the wharf is improved by 11%, the fisherman is improved by 24%, and the buoy is improved by 13%. This effectively proves the effectiveness of the RIF-YOLO algorithm compared with the same type of algorithm.

[0151] Table 2 Comparison of algorithm recognition accuracy

[0152]

[0153]

[0154] To prove the importance of the normalization attention mechanism and the radar feature extraction module adopted by the embodiments of the present application, the above two methods are subjected to an ablation experiment. The ablation experiment results are shown in Table 3.

[0155] Table 3 Ablation experiment results

[0156]

[0157] As can be seen from Table 3 above, after introducing the radar feature extraction module alone, the average accuracy of the model is improved by 5% compared with YOLOv7. After adding the normalization attention mechanism, the average accuracy of the model of the embodiments of the present application is improved by 1%. After adding the two, the accuracy is improved by 6%. The experiment shows that the radar feature extraction module and the normalization attention mechanism can improve the target detection accuracy in the water surface environment, and the average accuracy reaches 89%, proving that the model with the above two modules is the optimal structure.

[0158] To further directly prove the effectiveness of the embodiments of the present application in common water surface target detection difficult scenarios, including target occlusion, motion blur, water mist, etc., the visible light target detection algorithm YOLOv7 algorithm, the millimeter wave radar and visible light fusion Achelous detection algorithm and the (RIF-YOLO) algorithm of the embodiments of the present application are compared, and the comparison results are shown in Figure 4 .

[0159] From scene (a), it can be found that in the case of target occlusion, the YOLOv7 algorithm has a missed detection phenomenon, the Achelous algorithm with the radar branch and the algorithm of the embodiments of the present application can identify the object, and the embodiments of the present application have higher identification accuracy. Under scene (b), due to the rapid movement of the target, motion blur is caused. The YOLOv7 algorithm fails to find the target ship. Under scene (c), affected by the water mist, the target recognition is disturbed. The YOLOv7 algorithm and the Achelous algorithm both fail to successfully identify all the fishermen. However, the technical solution of the embodiments of the present application can find as many targets as possible, greatly improving the detection accuracy. The experiment shows that the technical solution proposed in the embodiments of the present application is higher than the YOLOv7 algorithm and the Achelous algorithm in terms of detection accuracy and recognition rate.

[0160] Figure 5 The composition structure schematic diagram of the target detection device based on millimeter wave radar and visible light fusion of the embodiments of the present application is shown in Figure 5 , the target detection device based on millimeter wave radar and visible light fusion of the embodiments of the present application comprises:

[0161] The radar image processing unit 50 is used to acquire a first image of a target area in a water environment based on millimeter-wave radar, and convert the three-dimensional data in the radar coordinate system in the first image into two-dimensional image data in the camera coordinate system, so that the range, Doppler intensity and energy intensity in the radar image data correspond to the RGB of the image channel respectively.

[0162] Acquisition unit 51 is used to acquire visible light images of the target area, perform distortion correction processing on the visible light images, and obtain the image to be processed;

[0163] The feature processing unit 52 is used to extract first feature data from radar two-dimensional image data and second feature data from the image to be processed, and to suppress the extracted features using a normalization algorithm, and to add a scaling factor to measure the importance of the features to the first feature data and the second feature data respectively.

[0164] The loss processing unit 53 is used to fuse the first feature data and the second feature data, and to process the fused feature data with target confidence loss, category confidence loss and coordinate regression loss to obtain confident feature data.

[0165] The identification unit 54 is used to detect and identify target objects based on reliable feature data, and to extract target objects in the water surface environment.

[0166] In one embodiment, the target detection device based on millimeter-wave radar and visible light fusion according to the present disclosure further includes:

[0167] Convolutional unit ( Figure 4 (not shown in the image) is used to perform variable convolution on two-dimensional data in the radar coordinate system of the first image, that is, to add a displacement variable of the two-dimensional data to the convolution layer of the two-dimensional data, and to change the receptive field of the two-dimensional data through the displacement variable so that its receptive field is a polygon.

[0168] The displacement variables are obtained through online learning based on two-dimensional data; or the displacement variables are obtained based on preset values ​​corresponding to different application scenarios.

[0169] In one embodiment, the feature processing unit 52 is further configured to:

[0170] Add a scaling factor to the data to measure the importance of the features in the following ways:

[0171]

[0172] Where BN() is the batch normalization operation. Bin For the input 3D data, and respectively, and are trainable affine transformation parameters, and is a preset constant.

[0173] In an implementation, the loss processing unit 53 is further configured to perform target confidence loss, class confidence loss and coordinate regression loss processing on the fused feature data by the following manner:

[0174]

[0175] wherein, and represents a binary cross-entropy loss, P0 represents a target confidence score in a prediction box, P IOU represents an IOU value of a target box corresponding to the prediction box, w obj and w cls is a weight of a positive sample, IoU represents an intersection over union between two bounding rectangular boxes, a represents a weight coefficient, v represents a consistency of a relative proportion of two rectangular boxes, b and b gt represent the center points of two bounding rectangular boxes, p represents a Euclidean distance between two bounding rectangular boxes, and c represents a distance of a diagonal line of a closure region of two bounding rectangular boxes.

[0176] In an implementation, the acquisition unit 51 is further configured to:

[0177] determine the radial distortion of the image by the following manner:

[0178] x dr = x (1 + k1r 2 + k2r 4 + k3r 6 )

[0179] y dr = y (1 + k1r 2 + k2r 4 + k3r 6 )

[0180] r 2 = x 2 +y 2

[0181] wherein, (x, y) is a coordinate of an ideal non-distorted image coordinate system, (x dr , y dr ) is a coordinate of a distorted image pixel point, k1, k2 and k3 are distortion coefficients, which are obtained when calibrating the camera.

[0182] r 2 = x 2 +y 2

[0183] The tangential distortion of the image is determined by the following manner:

[0184] x dt = 2p1xy + p2(r 2 + 2x 2 ) + 1

[0185] y dt = 2p1(r 2 + 2y 2 ) + 2p2xy + 1

[0186] wherein (x dt , y dt ) is the coordinate of the pixel point of the distorted image, p1 and p2 are distortion coefficients, which are obtained when calibrating the camera;

[0187] The sum of the radial distortion and the tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined by the following manner:

[0188] x d = x dr + x dt

[0189] y d = y dr + y dt

[0190] The visible light image is de-distorted by using the determined distortion parameters, to obtain a de-distorted visible light image.

[0191] In the example embodiments, the foregoing units and the like can be implemented by one or more central processing units (CPUs), graphics processing units (GPUs), application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, micro controllers (MCUs), microprocessors (Microprocessors), or other electronic elements.

[0192] Regarding the apparatus in the above embodiments, the specific manner in which each module and unit performs its operations has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0193] Figure 6 A schematic block diagram of an example network element 800 that can be used to implement embodiments of the present disclosure is shown. Figure 6 As shown, network element 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 can also store various programs and data required for the operation of network element 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0194] Multiple components in network element 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, data transceiver, etc. Communication unit 809 allows network element 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0195] The computing unit 801 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs various methods and processes described above, such as the target detection method based on millimeter wave radar and visible light fusion. For example, in some embodiments, the target detection method based on millimeter wave radar and visible light fusion can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the network element 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded onto the RAM 803 and executed by the computing unit 801, one or more steps of the target detection method based on millimeter wave radar and visible light fusion described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the target detection method based on millimeter wave radar and visible light fusion by any other suitable means, such as by means of firmware.

[0196] Various implementations of the systems and techniques described above in embodiments of the application can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0197] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces a means for implementing the functions / acts specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0198] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0199] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0200] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0201] The computer system can include clients and servers. This relationship can be. The servers are typically remote from the clients with the interactions typically occurring over a communication network. The relationship between client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers incorporating blockchain.

[0202] It should be understood that the various forms of flow shown above can be used to reorder, add, or remove steps. For example, the steps described in the embodiments of the present application can be executed in parallel, in sequence, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

[0203] In addition, the terms "first", "second", "third", etc. are used only for descriptive purposes and should not be construed as indicating or implying relative importance or an indicated number of technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality of" is two or more, unless otherwise specifically limited.

[0204] The above is merely a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A target detection method based on millimeter wave radar and visible light fusion, characterized in that, The method comprises: Based on the millimeter wave radar, the first image of the target area in the water surface environment is collected, the three-dimensional data in the radar coordinate system in the first image is converted into two-dimensional image data in the camera coordinate system, the distance, Doppler intensity and energy intensity in the radar image data are respectively corresponding to the RGB of the image channel; Collecting a visible light image of the target area, and performing distortion removal processing on the visible light image to obtain a to-be-processed image; Extracting first feature data of the radar two-dimensional image data, and extracting second feature data of the to-be-processed image, using a normalization algorithm to suppress the extracted features, and adding a proportion factor for measuring the importance of the features to the first feature data and the second feature data respectively; Fusing the first feature data and the second feature data, and processing the fused feature data through target confidence loss, category confidence loss and coordinate regression loss to obtain confident feature data; Based on the confident feature data, the target object is detected and recognized, and the target object in the water surface environment is extracted; The method further comprises adding a proportion factor for measuring the importance of the features to the data in the following manner: in, For batch normalization calculations, For the input 3D data, and They are small batches The mean and standard deviation, and These are trainable affine transformation parameters. This is a preset constant; The fused feature data is processed through target confidence loss, category confidence loss and coordinate regression loss in the following manner: in, and This is represented as the binary cross-entropy loss. Represented as the target confidence score in the prediction box. This is represented as the IOU value of the target bounding box corresponding to the predicted bounding box. and The weights are the positive samples, and IoU represents the intersection-union ratio between two bounding boxes. Indicates the weighting coefficient. This indicates the consistency of the relative proportions of two rectangles. and This indicates the center point of the two bounding rectangles. This represents the Euclidean distance between two bounding rectangles. This represents the distance between the diagonals of the enclosing regions of two bounding rectangles. The category score of the predicted bounding box. For the target bounding box category.

2. The method of claim 1, wherein, After converting the three-dimensional data in the radar coordinate system in the image into radar image data in the camera coordinate system, the method further comprises: Performing variable convolution on the two-dimensional data in the radar coordinate system in the first image, that is, adding a displacement variable of the two-dimensional data to the convolution layer of the two-dimensional data, changing the receptive field range of the two-dimensional data through the displacement variable, so that the receptive field range is a polygon; Wherein, the displacement variable is obtained through online learning according to the two-dimensional data, or the displacement variable is obtained according to the preset value corresponding to different application scenarios.

3. The method of claim 1, wherein, The distortion removal processing on the visible light image comprises: The radial distortion of the image is determined in the following manner: wherein, is the ideal undistorted image coordinate system coordinate, is the coordinate of the distorted image pixel, is the distortion coefficient, obtained when calibrating the camera; The tangential distortion of the image is determined in the following manner: wherein, is the coordinate of the pixel point of the distorted image, is the distortion coefficient, which is obtained when calibrating the camera; The sum of the radial distortion and the tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined in the following manner: The visible light image is de-distorted with the determined distortion parameters to obtain a de-distorted visible light image.

4. A target detection device based on fusion of millimeter wave radar and visible light, characterized by, The device comprises: A radar image processing unit, configured to collect, based on a millimeter wave radar, a first image of a target area in a water surface environment, convert three-dimensional data in a radar coordinate system in the first image into two-dimensional image data in a camera coordinate system, and make the distance, Doppler intensity and energy intensity in the radar image data respectively corresponding to the RGB of the image channel; A collecting unit, configured to collect a visible light image of the target area, and perform distortion removal processing on the visible light image to obtain a to-be-processed image; A feature processing unit, configured to extract first feature data of the radar two-dimensional image data, and extract second feature data of the to-be-processed image, use a normalization algorithm to suppress the extracted features, and add a proportion factor for measuring the importance of the features to the first feature data and the second feature data respectively; The loss processing unit is configured to fuse the first feature data and the second feature data, and perform target confidence loss, class confidence loss, and coordinate regression loss processing on the fused feature data to obtain certain feature data. The recognition unit is configured to detect and recognize the target object based on the certain feature data, and extract the target object in the water surface environment. The feature processing unit is further configured to: add a scale factor for measuring the importance of the data in the following manner: wherein, is a batch normalization operation, is the input three-dimensional data, and are the mean and standard deviation of the mini-batch, respectively, and are trainable affine transformation parameters, is a preset constant; The feature processing unit is further configured to: perform target confidence loss, class confidence loss, and coordinate regression loss processing on the fused feature data in the following manner: wherein, wherein is expressed as a binary cross-entropy loss, is expressed as a target confidence score in a prediction box, is expressed as an IOU value of a target box corresponding to a prediction box, wherein is a weight of a positive sample, and IoU represents an intersection over union between two bounding rectangular boxes, is expressed as a weight coefficient, is expressed as a consistency of a relative proportion of two rectangular boxes, and is expressed as a center point of two bounding rectangular boxes, is expressed as an Euclidean distance between two bounding rectangular boxes, is expressed as a distance of a diagonal line of a closure area of two bounding rectangular boxes, is a class score of a prediction box, is a target box class.

5. The apparatus of claim 4, wherein, The device further includes: The convolution unit performs variability convolution on the two-dimensional data in the radar coordinate system in the first image, that is, a displacement variable of the two-dimensional data is added to the convolution layer of the two-dimensional data, the receptive field range of the two-dimensional data is changed through the displacement variable, and the receptive field range of the two-dimensional data is a polygon. The displacement variable is obtained through online learning according to the two-dimensional data, or the displacement variable is obtained according to a preset value corresponding to different application scenarios.

6. The apparatus of claim 4, wherein, The acquisition unit is further configured to: determine the radial distortion of the image in the following manner: wherein, is the ideal undistorted image coordinate system coordinate, is the coordinate of the distorted image pixel, is the distortion coefficient, obtained when calibrating the camera; determine the tangential distortion of the image in the following manner: wherein, is the coordinate of the pixel point of the distorted image, is the distortion coefficient, which is obtained when calibrating the camera; determine the overall distortion of the visible light image in the following manner by taking the sum of the radial distortion and the tangential distortion as the overall distortion: perform de-distortion on the visible light image according to the determined distortion parameter to obtain a de-distorted visible light image.

Images