Target detection method and device based on millimeter wave radar and visible light fusion

Abstract

The invention discloses a target detection method and device based on millimeter-wave radar and visible light fusion, and the method comprises the steps: enabling a convolutional network to better extract effective features from the millimeter-wave radar data in a sparse form through employing a method for converting radar data into a radar image. A radar feature extraction method based on deformable convolution is adopted, and feature fusion interference caused by registration errors of millimeter wave radar images and visible light images is reduced; a normalized attention mechanism is adopted, and the influence of radar image virtual scene interference is effectively reduced through a sparse weight punishment mechanism; aiming at the problems of difficulty in effective feature extraction, clutter false alarm interference and the like in a current water surface target detection algorithm, the method can extract effective features from a complex water surface environment, and can extract a target object more accurately.

Description

Technical Field

[0001] The present application relates to target monitoring technology in a water surface environment, and in particular to a target detection method and device based on millimeter wave radar and visible light fusion in a harsh water surface environment. Background Art

[0002] As one of the core problems of computer vision, target detection aims to find the category and location of specific targets in an image. It is now widely used in various fields, such as autonomous driving, remote sensing images, video surveillance, and medical testing. Surface target detection is mainly used in surface detection equipment or unmanned surface equipment. Among them, unmanned surface vessels (USVs) play an important role in surface processing fields such as surface rescue, garbage detection, and terrain detection. Laser radar is generally used for target recognition in target detection, but due to the poor semantic information of the laser radar point cloud and too many false alarm points of surface waves, the recognition accuracy of this type of algorithm is not enough. In addition, the existing target detection also uses a target recognition method based on three-dimensional laser radar. This method has too high data dimensions, high computational difficulty, and difficulty in distinguishing target types. When the visible light RGB image is used for surface target detection, the effective features of the target are blocked when the visible light image is blocked, which easily leads to missed detection of the target. Summary of the invention

[0003] The present application provides a target detection method and device based on millimeter-wave radar and visible light fusion to at least solve the aforementioned technical problems.

[0004] According to a first aspect of an embodiment of the present application, a target detection method based on millimeter wave radar and visible light fusion is provided, including:

[0005] A first image of a target area in a water surface environment is collected based on a millimeter-wave radar, and three-dimensional data in a radar coordinate system in the first image is converted into two-dimensional image data in a camera coordinate system, so that the distance, Doppler intensity, and energy intensity in the radar image data correspond to the RGB of an image channel, respectively;

[0006] Collecting a visible light image of the target area, and performing dedistortion processing on the visible light image to obtain an image to be processed;

[0007] Extracting first feature data of the radar two-dimensional image data and extracting second feature data of the image to be processed, suppressing the extracted features using a normalization algorithm, and adding a proportional factor for measuring the importance of the features to the first feature data and the second feature data respectively;

[0008] The first feature data and the second feature data are fused, and the fused feature data are processed with target confidence loss, category confidence loss, and coordinate regression loss to obtain confident feature data;

[0009] Detect and identify target objects based on confident feature data and extract target objects in water surface environments.

[0010] In one possible implementation, after converting the three-dimensional data in the radar coordinate system in the image into radar image data in the camera coordinate system, the method further includes:

[0011] Performing variable convolution on the two-dimensional data in the radar coordinate system in the first image, that is, adding a displacement variable of the two-dimensional data to the convolution layer of the two-dimensional data, and changing the receptive field range of the two-dimensional data by the displacement variable so that the receptive field range is a polygon;

[0012] The displacement variables are obtained by online learning based on two-dimensional data; or the displacement variables are obtained based on preset values ​​corresponding to different application scenarios.

[0013] In one embodiment, the method further includes adding a scaling factor measuring the importance of the feature to the data in the following manner:

[0014]

[0015] Among them, BN() is the batch normalization operation, B in is the input three-dimensional data, and are the mean and standard deviation of the mini-batch β, γ and β are trainable affine transformation parameters, and ∈ is a preset constant.

[0016] In one embodiment, the method further includes processing the fused feature data with target confidence loss, category confidence loss, and coordinate regression loss in the following manner:

[0017]

[0018]

[0019] in, and It is expressed as the binary cross entropy loss, P0 is expressed as the target confidence score in the prediction box, and P IOU It is expressed as the IOU value of the target box corresponding to the predicted box, w obj With w cls is the weight of the positive sample, IoU represents the intersection-over-union ratio between two bounding rectangles, α represents the weight coefficient, ν represents the consistency of the relative proportions of the two rectangles, b and b gtrepresents the center points of the two bounding rectangular boxes, ρ represents the Euclidean distance between the two bounding rectangular boxes, and c represents the distance between the diagonals of the closure areas of the two bounding rectangular boxes.

[0020] In one possible implementation manner, the dedistortion processing of the visible light image includes:

[0021] The radial distortion of an image is determined by:

[0022] x dr =x(1+k1r 2 +k2r 4 +k3r 6 )

[0023] y dr =y(1+k1r 2 +k2r 4 +k3r 6 )

[0024] r 2 =x 2 +y 2

[0025] Among them, (x, y) is the ideal undistorted image coordinate system coordinate, (x dr ,y dr ) is the coordinate of the pixel point of the distorted image, k1, k2, k3 are the distortion coefficients, which are obtained when calibrating the camera;

[0026] r 2 =x 2 +y 2

[0027] The tangential distortion of the image is determined by:

[0028] x dt =2p1xy+p2(r 2 +2x 2 )+1

[0029] y dt =2p1(r 2 +2y 2 )+2p2xy+1

[0030] Among them, (x dt ,y dt ) is the coordinate of the pixel point of the distorted image, p1 and p2 are the distortion coefficients, which are obtained when calibrating the camera;

[0031] The sum of radial distortion and tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined by the following method:

[0032] xd =x dr +x dt

[0033] y d =y dr +y dt

[0034] The visible light image is dedistorted using the determined distortion parameters to obtain a dedistorted visible light image.

[0035] According to a second aspect of the present application, a target detection device based on millimeter wave radar and visible light fusion is provided, comprising:

[0036] A radar image processing unit, for collecting a first image of a target area in a water surface environment based on a millimeter wave radar, converting three-dimensional data in a radar coordinate system in the first image into two-dimensional image data in a camera coordinate system, so that the distance, Doppler intensity, and energy intensity in the radar image data correspond to the RGB of an image channel respectively;

[0037] An acquisition unit is used to acquire a visible light image of a target area and perform a dedistortion process on the visible light image to obtain an image to be processed;

[0038] A feature processing unit, used to extract first feature data of the radar two-dimensional image data and second feature data of the image to be processed, suppress the extracted features using a normalization algorithm, and add a proportional factor measuring the importance of the features to the first feature data and the second feature data respectively;

[0039] A loss processing unit, used to fuse the first feature data and the second feature data, and process the fused feature data with target confidence loss, category confidence loss, and coordinate regression loss to obtain confident feature data;

[0040] The recognition unit is used to detect and recognize the target object based on the confirmed feature data, and extract the target object in the water surface environment.

[0041] In one embodiment, the device further comprises:

[0042] A convolution unit performs variable convolution on the two-dimensional data in the radar coordinate system in the first image, that is, adds a displacement variable of the two-dimensional data to the convolution layer of the two-dimensional data, and changes the receptive field range of the two-dimensional data through the displacement variable so that the receptive field range is a polygon;

[0043] The displacement variables are obtained by online learning based on two-dimensional data; or the displacement variables are obtained based on preset values ​​corresponding to different application scenarios.

[0044] In one possible implementation manner, the feature processing unit is further used for:

[0045] Add a scaling factor to the data that measures the importance of the feature by:

[0046]

[0047] Among them, BN() is the batch normalization operation, B in is the input three-dimensional data, and are the mean and standard deviation of the mini-batch β, γ and β are trainable affine transformation parameters, and ∈ is a preset constant.

[0048] In one possible implementation manner, the loss processing unit is further used to process the target confidence loss, the category confidence loss and the coordinate regression loss on the fused feature data in the following manner:

[0049]

[0050] in, and It is expressed as the binary cross entropy loss, P0 is expressed as the target confidence score in the prediction box, and P IOU It is expressed as the IOU value of the target box corresponding to the predicted box, w obj With w cls is the weight of the positive sample, IoU represents the intersection-over-union ratio between two bounding rectangles, α represents the weight coefficient, ν represents the consistency of the relative proportions of the two rectangles, b and b gt represents the center points of the two bounding rectangular boxes, ρ represents the Euclidean distance between the two bounding rectangular boxes, and c represents the distance between the diagonals of the closure areas of the two bounding rectangular boxes.

[0051] In one possible implementation, the acquisition unit is further used for:

[0052] The radial distortion of an image is determined by:

[0053] x dr =x(1+k1r 2 +k2r 4 +k3r 6 )

[0054] y dr =y(1+k1r 2 +k2r 4 +k3r 6 )

[0055] r 2 =x 2 +y 2

[0056] Among them, (x, y) is the ideal undistorted image coordinate system coordinate, (x dr ,y dr ) is the coordinate of the pixel point of the distorted image, k1, k2, k3 are the distortion coefficients, which are obtained when calibrating the camera;

[0057] r 2 =x 2 +y 2

[0058] The tangential distortion of the image is determined by:

[0059] x dt =2p1xy+p2(r 2 +2x 2 )+1

[0060] y dt =2p1(r 2 +2y 2 )+2p2xy+1

[0061] Among them, (x dt ,y dt ) is the coordinate of the pixel point of the distorted image, p1 and p2 are the distortion coefficients, which are obtained when calibrating the camera;

[0062] The sum of radial distortion and tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined by the following method:

[0063] x d =x dr +x dt

[0064] y d =y dr +y dt

[0065] The visible light image is dedistorted using the determined distortion parameters to obtain a dedistorted visible light image.

[0066] The technical solution provided by the embodiments of the present application may have the following beneficial effects:

[0067] This application improves the accuracy of target monitoring and recognition in water surface environments by converting the millimeter wave image data collected by radar into two-dimensional radar image data and completing registration with visible light images. By adding a scale factor to the feature data, the error effect of the registration of millimeter wave radar images and visible light images is reduced. The deformable convolution parameters provided by online learning are accurate and reliable in extracting radar features. Through the special processing method of normalized attention, the false alarm interference of surface clutter radar images is reduced, the selected radar information is more reliable and effective, and the feature weight of virtual scenes in the image is reduced, which greatly improves the recognition accuracy of target objects in harsh water surface environments such as motion blur, water fog interference, and target occlusion.

[0068] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0070] Figure 1 A schematic diagram of a flow chart of a target detection method based on millimeter wave radar and visible light fusion according to an embodiment of the present application;

[0071] Figure 2 A schematic diagram of generating a radar chart of a target object shown in an embodiment of the present application;

[0072] Figure 3 This is a schematic diagram of an image processing method of a radar module according to an embodiment of the present application;

[0073] Figure 4 This is a schematic diagram of an image processing result shown in an embodiment of the present application;

[0074] Figure 5 This is a schematic diagram of the structure of a target detection device based on millimeter wave radar and visible light fusion according to an embodiment of the present application;

[0075] Figure 6 It is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0076] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0077] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0078] Figure 1 This is a flow chart of a target detection method based on millimeter wave radar and visible light fusion according to an embodiment of the present application, as shown in FIG. Figure 1 As shown, the target detection method based on millimeter wave radar and visible light fusion in the embodiment of the present application includes the following steps:

[0079] Step 101, based on the millimeter wave radar, collects a first image of the target area in the water surface environment, converts the three-dimensional data in the radar coordinate system in the first image into radar image data in the camera coordinate system, and makes the distance, Doppler intensity, and energy intensity in the radar image data correspond to the RGB of the image channel respectively.

[0080] In the embodiment of the present application, due to the imaging characteristics of the millimeter wave radar, the data returned by it is three-dimensional data, which cannot be directly input into the two-dimensional convolutional network and is difficult to fuse with the image data. The present application adopts the projection conversion method of the radar image to project the three-dimensional data into the pixel coordinate system through the intrinsic and extrinsic parameters of the camera. First, the radar point in the three-dimensional radar coordinate is converted to the corresponding position in the camera coordinate by formula (1). Among them, X r is the 3D position information in radar coordinates, R is the rotation matrix, T is the translation matrix, X i is the position of the radar point cloud in the pixel coordinate system.

[0081] X i =X r R+T (1)

[0082] Radar image data contains three dimensions, namely, the target's range, the target's radial velocity (doppler), and the target's signal strength (power). The amplitudes of these dimensions vary, which makes it difficult for the network to understand them better. In addition, the amplitudes of these dimensions differ greatly from those of visible light images, which is not conducive to subsequent feature fusion.

[0083] The embodiment of the present application proposes a calculation method for matrix mapping, and the calculation method is shown in formula (2).

[0084]

[0085] Among them, matrix A is a 3×3 mapping matrix, and matrix C is a 1×3 parameter matrix, which is mapped into a radar image equivalent to the visible light RGB amplitude according to the amplitude range of the radar data.

[0086] After the radar image is generated, in order to highlight the importance differences of different target points, the radar points are expanded to different degrees according to the Euclidean distance values ​​of the radar data to form solid circles of different sizes, as shown in the following formula (3).

[0087]

[0088] Among them, r dis Represents the radius of the generated solid circle, α k It is expressed as data gain, dis represents the Euclidean distance of radar data, and x, y, and z are the positions of the target point in the three-dimensional space under the radar coordinate system.

[0089] At the same time, due to the high confidence of the radar center point, the Gaussian filtering algorithm is added. As shown in the following formula (4):

[0090]

[0091] Among them, R, G, and B are the image channel values ​​after generating the solid circle. G(R, G, B) is the channel value obtained after Gaussian filtering transformation; δ 2 Expressed as variance.

[0092] Step 102 , collect a visible light image of the target area, and perform dedistortion processing on the visible light image to obtain an image to be processed.

[0093] In the embodiment of the present application, the visible light image can be collected by an industrial camera, etc., and the target area in the water surface environment can be the water area where the target object to be identified is located. Due to the movement of the surface boat, wind or seawater tide, the water surface of the water area often has some harsh environments, resulting in motion blur, water fog interference, target occlusion, etc. in the collected image. In this way, the conventional target object recognition method will have very poor recognition results. The embodiment of the present application is proposed for the recognition of target objects in this water surface environment. Through the relevant technical means of the embodiment of the present application, the recognition error of the target object can be greatly reduced and the recognition accuracy of the target object can be improved.

[0094] Specifically, the imaging process is essentially the transformation of several coordinate systems. First, a point in space is transformed from the world coordinate system to the camera coordinate system, then projected to the imaging plane (image physical coordinate system), and finally the data on the imaging plane is transformed to the image plane (image pixel coordinate system). Due to a series of processes such as camera manufacturing, camera distortion is generated: radial distortion and tangential distortion.

[0095] The radial distortion is determined as shown in formula (5): For a lens, the center of the lens is taken as the origin, and the outward direction is the radius of the lens. The closer the light is to the center, the smaller the distortion is, and the farther away from the center along the radius, the larger the distortion is.

[0096]

[0097] Among them, (x, y) is the ideal undistorted coordinate system (image coordinate system), (x dr ,y dr ) is the coordinate of the pixel point of the distorted image. k1, k2, k3 are the distortion coefficients.

[0098] Determine the tangential distortion of the image, as shown in formula (6): Tangential distortion is the distortion produced when the lens is not aligned with the imaging plane, which is similar to perspective transformation.

[0099] x dt =2p1xy+p2(r 2 +2x 2 )+1

[0100] y dt =2p1(r 2 +2y 2 )+2p2xy+1 (6)

[0101] Among them, (x, y) is the ideal undistorted coordinate system (image coordinate system), (x dt ,y dt ) is the coordinate of the pixel point of the distorted image. p1 and p2 are the distortion coefficients.

[0102] The sum of radial distortion and tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined by the following method:

[0103] x d =x dr +x dt

[0104] y d =y dr +y dt

[0105] Those skilled in the art should understand that step 101 and step 102 are steps executed in parallel, and can be started at the same time when capturing images of the target area. Of course, within a certain time difference range, they can also be executed one after another. Steps 101 and 102 are only for the convenience of identification and are not related to the timing.

[0106] Step 103, extracting first feature data of the radar two-dimensional image data and second feature data of the image to be processed, suppressing the extracted features using a normalization algorithm, and adding a proportional factor for measuring the importance of the features to the first feature data and the second feature data respectively.

[0107] In the embodiment of the present application, due to the presence of certain errors in the intrinsic calibration data during the camera calibration process, and the various types of camera distortion, which cannot be completely removed by calibration, a certain offset occurs during the conversion of the millimeter-wave point cloud image to the radar RGB image. Considering the above problems, deformable convolution is introduced. Deformable convolution adds a displacement variable to the convolution layer, and this variable is learned online according to the data. After introducing this variable, it is equivalent to changing the range of the receptive field, making the receptive field a polygon.

[0108] Specifically, Figure 3 As shown, variable convolution is performed on the two-dimensional data in the radar coordinate system in the first image, that is, a displacement variable of the two-dimensional data is added to the convolution layer of the two-dimensional data, and the receptive field range of the two-dimensional data is changed by the displacement variable so that the receptive field range is a polygon; wherein the displacement variable is obtained by online learning based on the two-dimensional data; or the displacement variable is obtained according to preset values ​​corresponding to different application scenarios. Figure 3 In the figure, Conv represents the convolution operation, and Maxpooling represents the maximum pooling operation. It uses the idea of ​​moving the window to move on the image, selects the largest value in the window and records it, and finally gets the corresponding output.

[0109] The radar image feature extraction module of the embodiment of the present application is composed of three parts: a convolution module, a pooling module, and a feature enhancement module. For the input 3-channel RGB image, the initial feature extraction is first performed through four convolution, normalization, and activation layers, and the size of the feature map is gradually reduced in this way. Due to the large amount of information in the image features, the network's ability to express complex features is enhanced by multi-channel separate convolution. The channel separation convolution method consists of two branches, one branch uses 1×1 convolution to change the channel, and the other branch first uses 1×1 convolution to change the channel, and then undergoes four 3×3 convolutions to extract features. The size of the feature map is reduced by using the maximum pooling method, and the parameters of the model are downgraded to ensure that rich feature information can be extracted at different scales. The maximum pooling module consists of two branches, one branch first undergoes maximum pooling, and then undergoes a 1×1 convolution to change the number of channels, and the other branch first undergoes a 1×1 convolution to change the number of channels, and then passes through a 3×3 convolution block with a kernel step size of 2 for downsampling. The image feature extraction module finally generates three feature maps of different scales. These feature maps capture information of different levels and scales in the input image, providing a rich feature basis for subsequent object detection tasks.

[0110] Among them, 1×1Conv: uses a convolution kernel of size 1×1 for convolution, which is mainly used for the layers of feature images.

[0111] Maxpooling3×3: The convolution kernel of the convolution operation has data weights, while pooling directly calculates the original data in the pooling window. This calculation process can select the maximum value.

[0112] Among them, Deformable conv represents deformable convolution processing, BN is batch normalization operation, that is, BatchNormalization operation. The BN layer standardizes the distribution of each batch of data so that each layer of the neural network can learn a relatively stable input distribution, thereby alleviating the problem of internal covariate shift, and is more suitable for the model in which the input data distribution changes continuously during the training process of the embodiment of the present application. Act is represented by the activation function, which improves the nonlinear fitting ability of the model by introducing nonlinear variables. The embodiment of the present application improves the operation structure of the BN layer by introducing convolution operation to improve local feature extraction and calculation efficiency.

[0113] Specifically, a scaling factor that measures the importance of a feature is added to the data in the following way:

[0114]

[0115] Among them, BN() is a batch normalization operation, Bin is the input three-dimensional data, and are the mean and standard deviation of the small batch β, γ and β are trainable affine transformation parameters, ∈ is a preset constant, which is a minimum value used to prevent the denominator from being zero without affecting the overall calculation result.

[0116] Step 104, the first feature data and the second feature data are fused, and the fused feature data are processed with target confidence loss, category confidence loss and coordinate regression loss to obtain confirmed feature data.

[0117] In the embodiment of the present application, the fused feature data is processed with target confidence loss, category confidence loss and coordinate regression loss in the following manner:

[0118]

[0119] in, and It is expressed as the binary cross entropy loss, P0 is expressed as the target confidence score in the prediction box, and P IOU It is expressed as the IOU value of the target box corresponding to the predicted box, w obj With w cls is the weight of the positive sample, IoU represents the intersection-over-union ratio between two bounding rectangles, α represents the weight coefficient, ν represents the consistency of the relative proportions of the two rectangles, b and b gt represents the center points of the two bounding rectangular boxes, ρ represents the Euclidean distance between the two bounding rectangular boxes, and c represents the distance between the diagonals of the closure areas of the two bounding rectangular boxes.

[0120] Step 105 , detecting and identifying the target object based on the confirmed feature data, and extracting the target object in the water surface environment.

[0121] The essence of the technical solution of the embodiment of the present application is further illustrated by specific examples below.

[0122] The surface environment target detection method based on millimeter-wave radar and visible light camera proposed in the embodiment of the present application, for the input 4D millimeter-wave radar data and visible light camera image, firstly, the 4D millimeter-wave radar data is passed through the radar multi-channel mapping image generation algorithm, so that the radar point cloud is projected into the camera coordinate system, and converted into a radar RGB image through the camera internal parameters. Subsequently, the aligned radar RGB image and the visible light camera image are respectively passed through the visible light data processing branch and the radar feature extraction branch of a neural network such as YOLOv7. Specifically, the embodiment of the present application has made precise adjustments to the processing details, which are described in detail below, so that the corresponding radar RGB feature map and visible light feature map can be obtained. Then, the radar RGB feature map is subjected to the normalized attention mechanism to further extract features. Finally, the two types of feature maps are passed through the detection head of YOLOv7 for target detection and recognition.

[0123] Specifically, the embodiment of the present application uses four convolution operations to preliminarily extract low-level features, while gradually reducing the size of the feature map. Considering the relationship between the shortest and longest gradient path of each layer and the longest gradient path of the entire network during the feature extraction process, the ELAN (Efficient Layer Aggregation Network) module is used to enhance the network's ability to represent complex features. The MP module is used to ensure that rich feature information can be extracted at different scales. The ELAN module consists of two branches, one branch uses 1×1 convolution to change the channel, and the other branch first uses 1×1 convolution to change the channel, and then undergoes four 3×3 convolutions to extract features. The MP module also consists of two branches, one branch first undergoes maximum pooling, then undergoes a 1×1 convolution to change the number of channels, and the other branch first undergoes 1×1 convolution to change the number of channels, and then undergoes a 3×3 convolution kernel with a step size of 2 convolution blocks for downsampling. Through the feature extraction module, three feature maps of different scales are finally generated. These feature maps capture information of different levels and scales in the input image, respectively, and provide a rich feature basis for subsequent target detection tasks.

[0124] Since there are certain errors in the intrinsic calibration data during the camera calibration process, and the camera distortion types are diverse and cannot be completely removed by calibration, a certain offset occurs in the process of converting the millimeter-wave point cloud image to the radar RGB image. Taking the above problems into consideration, the embodiment of the present application calibrates the image data by setting a radar correction feature extraction module (Radar RectifyFeature Block) and introduces deformable convolution to process the converted data. Specifically, a displacement variable is added to the convolution layer of the deformable convolution, and this variable is learned online according to the data. After introducing this variable, it is equivalent to changing the range of the receptive field, so that the receptive field becomes a polygon.

[0125] The embodiment of the present application uses a radar feature extraction module to extract feature points from an image. Specifically, the processing method is as follows:

[0126] In order to reduce the interference of noise and improve the anti-interference ability of the radar data processing branch, the embodiment of the present application adds a normalized attention module to the processing model. The normalized attention mechanism (Normalization-based Attention Module) is a high-performance, lightweight attention mechanism. Its module integrates and redesigns the channel and spatial attention submodules from the CBAM module. That is, a batch scaling factor (BN) is added on the original basis, as shown in formula (7), and the scale factor is used to measure the variance of the channel and indicate the importance of the feature.

[0127]

[0128] in, and are the mean and standard deviation of the small batch β, γ and β are trainable affine transformation parameters, ∈ is a preset constant, which is a minimum value used to prevent the denominator from being zero without affecting the overall calculation result.

[0129] The batch scaling factor is added to the original channel attention module, and its output function is shown in formula (8). represents weight, F1 is input, M C is the output of the channel attention module. Similarly, adding a batch scaling factor to the original spatial attention module results in an improved spatial attention mechanism, whose output formula is shown in formula (9). represents weight, F2 is input, M s is the output of the channel attention module.

[0130] M c =sigmoid(W γ (BN(F1))) (8)

[0131] M s =sigmoid(W λ (BN s (F2))) (9)

[0132] In order to judge the accuracy of the detection algorithm, the embodiment of the present application adopts a loss function to improve the detection accuracy. The total loss function usually consists of the following parts:

[0133] Classification loss: It is used to measure the difference between the predicted class label and the true label.

[0134] Bounding box regression loss: It is used to measure the difference between the predicted bounding box and the true bounding box.

[0135] Confidence loss: used to measure the prediction accuracy of the model's confidence in the existence of the target

[0136] Target confidence loss Loss obj : Object confidence loss refers to the accuracy of the model in predicting whether an object exists in a certain grid cell. This part of the loss reflects the model's confidence in whether the object exists or not. The confidence score is a value between 0 and 1, indicating the probability that the model believes that the bounding rectangle contains a certain object.

[0137] Category confidence loss Loss cls : Each grid cell may predict multiple bounding boxes, and each bounding box will have a class probability distribution. These probability distributions represent the probability that the bounding box contains each class. Class confidence loss calculates the loss by comparing the predicted class probability distribution with the true class label.

[0138] Coordinate regression Loss bbox : Each predicted bounding rectangle is represented by four parameters: center coordinates (x, y) and width and height (w, h). Coordinate regression loss is used to calculate the difference between these parameters predicted by the model and the true values. , as shown in formula (10).

[0139] Loss=Loss obj ×α obj +Loss cls ×α cls +Loss bbox ×α box (10)

[0140] Among them, α obj , α cls , α box are preset parameters.

[0141] Both the target confidence loss and the category confidence loss are binary cross entropy losses. The target confidence loss is calculated by matching the sample pairs obtained from the positive sample. P0 is represented as the target confidence score in the prediction box, P IOU It is expressed as the IOU value of the target box corresponding to the predicted box. The binary cross entropy is calculated to obtain the final target confidence loss. As shown in formula (11). The category confidence loss is calculated by the category score c of the predicted box. p and target box category c gt To calculate the category loss, as shown in formula (12).

[0142]

[0143] in, and Expressed as binary cross entropy loss, w obj With w cls Represents the weight of the positive sample.

[0144] The coordinate regression loss uses CIoU Loss. CIoU Loss takes into account the overlapping area, center point distance and aspect ratio, and penalizes the result that the predicted shape is inconsistent with the ground truth, as shown in formula (13).

[0145]

[0146] Among them, α represents the weight coefficient, ν represents the consistency of the relative proportions of the two rectangular boxes, b and b gt represents the center points of the two rectangular boxes, ρ represents the Euclidean distance between the two rectangular boxes, and c represents the distance between the diagonals of the closure areas of the two rectangular boxes.

[0147] In order to verify the effectiveness and advancement of the technical solution of the embodiment of the present application, the embodiment of the present application is compared with the Achelous algorithm, and the mean average accuracy (mAP) and target recognition accuracy (AP) are used as evaluation indicators, and the larger the indicator, the better. The experimental results are shown in Table 1. When IOU=0.5 is selected as the standard for testing the performance of the target recognition algorithm, the average accuracy of the Achelous algorithm under this indicator is 76.2%, while the mean average accuracy (MAP) of the embodiment of the present application reaches 89.3%, and the accuracy is improved by 13.1%, which shows that compared with the existing research algorithm, the algorithm proposed in the embodiment of the present application has leading target detection performance.

[0148] Table 1 Comparison of algorithm type accuracy

[0149] algorithm sensor Average Average Precision (mAP) Achelous Millimeter wave radar + camera 76.2％ RIF-YOLO Millimeter wave radar + camera 89.3％

[0150] In addition, the present application embodiment also compares the detection accuracy (AP) of the two algorithms in different categories of the data set, and the results are shown in Table 2. It can be seen from the table that compared with the Achelous algorithm, the RIF-YOLO algorithm has significantly improved the detection accuracy of each category, among which the dock has increased by 11%, the fishermen have increased by 24%, and the buoy has increased by 13%. This effectively proves the effectiveness of the RIF-YOLO algorithm compared with similar algorithms.

[0151] Table 2 Algorithm recognition accuracy comparison

[0152]

[0153]

[0154] In order to demonstrate the importance of the normalized attention mechanism and radar feature extraction module used in the embodiment of the present application, an ablation experiment is conducted on the above two methods. The ablation experiment table is shown in Table 3 below.

[0155] Table 3 Ablation experiment results

[0156]

[0157] As can be seen from Table 3 above, after the radar feature extraction module is introduced alone, the average accuracy of the model is improved by 5% compared with YOLOv7. After adding the normalized attention mechanism, the average accuracy of the model in the embodiment of the present application is improved by another 1%. Adding the two together increases the accuracy by 6%. Experiments show that the radar feature extraction module and the normalized attention mechanism can improve the accuracy of target detection in a water environment, with an average accuracy of 89%, proving that the model with the above two modules is the optimal structure.

[0158] In order to further directly prove the effectiveness of the embodiment of the present application in common difficult water surface target detection scenarios, including target occlusion, motion blur, water fog, etc., the visible light target detection algorithm YOLOv7 algorithm, millimeter wave radar and visible light fusion Achelous detection algorithm and the (RIF-YOLO) algorithm of the embodiment of the present application are selected for comparison. The comparison results are shown in Figure 2. Figure 4 shown.

[0159] It can be found from scene (a) that when the target is obscured, the YOLOv7 algorithm has the phenomenon of missed detection. The Achelous algorithm with added radar branch and the algorithm of the embodiment of the present application (RIF-YOLO) can identify objects, and the embodiment of the present application has higher recognition accuracy. In scene (b), motion blur is caused by the rapid movement of the target. The YOLOv7 algorithm failed to detect the target ship. In scene (c), the influence of water mist interfered with target recognition. Both the YOLOv7 algorithm and the Achelous algorithm failed to successfully identify all fishermen. However, the technical solution of the embodiment of the present application can detect as many targets as possible, greatly improving the detection accuracy. Experiments show that the technical solution proposed in the embodiment of the present application is higher than the YOLOv7 algorithm and the Achelous algorithm in detection accuracy and recognition rate.

[0160] Figure 5 FIG. 1 shows a schematic diagram of the structure of a target detection device based on millimeter wave radar and visible light fusion according to an embodiment of the present disclosure. Figure 5 As shown, the target detection device based on millimeter wave radar and visible light fusion in the embodiment of the present disclosure includes:

[0161] A radar image processing unit 50 is used to collect a first image of a target area in a water surface environment based on a millimeter wave radar, convert three-dimensional data in a radar coordinate system in the first image into two-dimensional image data in a camera coordinate system, and make the distance, Doppler intensity, and energy intensity in the radar image data correspond to the RGB of an image channel respectively;

[0162] The acquisition unit 51 is used to acquire a visible light image of a target area and perform a dedistortion process on the visible light image to obtain an image to be processed;

[0163] A feature processing unit 52 is used to extract first feature data of the radar two-dimensional image data and second feature data of the image to be processed, suppress the extracted features using a normalization algorithm, and add a proportional factor measuring the importance of the features to the first feature data and the second feature data respectively;

[0164] A loss processing unit 53 is used to fuse the first feature data and the second feature data, and to process the fused feature data with target confidence loss, category confidence loss, and coordinate regression loss to obtain confident feature data;

[0165] The recognition unit 54 is used to detect and recognize the target object based on the confirmed feature data, and extract the target object in the water surface environment.

[0166] In one possible implementation manner, the target detection device based on millimeter wave radar and visible light fusion in the embodiment of the present disclosure further includes:

[0167] Convolutional unit ( Figure 4 (not shown) for performing variable convolution on the two-dimensional data in the radar coordinate system in the first image, that is, adding a displacement variable of the two-dimensional data to the convolution layer of the two-dimensional data, and changing the receptive field range of the two-dimensional data by the displacement variable so that the receptive field range is a polygon;

[0168] The displacement variables are obtained by online learning based on two-dimensional data; or the displacement variables are obtained based on preset values ​​corresponding to different application scenarios.

[0169] In one possible implementation, the feature processing unit 52 is further configured to:

[0170] Add a scaling factor to the data that measures the importance of the feature by:

[0171]

[0172] Among them, BN() is a batch normalization operation, Bin is the input three-dimensional data, and are the mean and standard deviation of the mini-batch β, γ and β are trainable affine transformation parameters, and ∈ is a preset constant.

[0173] In one embodiment, the loss processing unit 53 is further used to process the fused feature data with target confidence loss, category confidence loss and coordinate regression loss in the following manner:

[0174]

[0175] in, and It is expressed as the binary cross entropy loss, P0 is expressed as the target confidence score in the prediction box, and P IOU It is expressed as the IOU value of the target box corresponding to the predicted box, w obj With w cls is the weight of the positive sample, IoU represents the intersection-over-union ratio between two bounding rectangles, α represents the weight coefficient, ν represents the consistency of the relative proportions of the two rectangles, b and b gt represents the center points of the two bounding rectangular boxes, ρ represents the Euclidean distance between the two bounding rectangular boxes, and c represents the distance between the diagonals of the closure areas of the two bounding rectangular boxes.

[0176] In one embodiment, the acquisition unit 51 is further used for:

[0177] The radial distortion of an image is determined by:

[0178] x dr =x(1+k1r 2 +k2r 4 +k3r 6 )

[0179] y dr =y(1+k1r 2 +k2r 4 +k3r 6 )

[0180] r 2 =x 2 +y 2

[0181] Among them, (x, y) is the ideal undistorted image coordinate system coordinate, (x dr ,y dr ) is the coordinate of the pixel point of the distorted image, k1, k2, k3 are the distortion coefficients, which are obtained when calibrating the camera;

[0182] r 2 =x 2 +y 2

[0183] The tangential distortion of the image is determined by:

[0184] x dt =2p1xy+p2(r 2 +2x 2 )+1

[0185] y dt =2p1(r 2 +2y 2 )+2p2xy+1

[0186] Among them, (x dt ,y dt ) is the coordinate of the pixel point of the distorted image, p1 and p2 are the distortion coefficients, which are obtained when calibrating the camera;

[0187] The sum of radial distortion and tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined by the following method:

[0188] x d =x dr +x dt

[0189] y d =y dr +y dt

[0190] The visible light image is dedistorted using the determined distortion parameters to obtain a dedistorted visible light image.

[0191] In an exemplary embodiment, the aforementioned units and the like may be implemented by one or more central processing units (CPU), graphics processing units (GPU), application specific integrated circuits (ASIC), DSPs, programmable logic devices (PLD), complex programmable logic devices (CPLD), field programmable gate arrays (FPGA), general-purpose processors, controllers, microcontrollers (MCU), microprocessors, or other electronic components.

[0192] Regarding the device in the above embodiment, the specific manner in which each module and unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0193] Figure 6 8 is a schematic block diagram of an example network element 800 that can be used to implement an embodiment of the present disclosure. Figure 6 As shown, the network element 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the network element 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0194] Multiple components in the network element 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a data processing transceiver, etc. The communication unit 809 allows the network element 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0195] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a target detection method based on millimeter wave radar and visible light fusion. For example, in some embodiments, the target detection method based on millimeter wave radar and visible light fusion may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the network element 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the target detection method based on millimeter wave radar and visible light fusion described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute a target detection method based on millimeter-wave radar and visible light fusion in any other appropriate manner (for example, by means of firmware).

[0196] Various implementations of the systems and techniques described above in the embodiments of the present application can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: being implemented in one or more computer programs, which may be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general programmable processor, which may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0197] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0198] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0199] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0200] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0201] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0202] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the embodiments of the present application can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, and this document is not limited here.

[0203] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0204] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A target detection method based on millimeter wave radar and visible light fusion, characterized in that: The method comprises: A first image of a target area in a water surface environment is collected based on a millimeter-wave radar, and three-dimensional data in a radar coordinate system in the first image is converted into two-dimensional image data in a camera coordinate system, so that the distance, Doppler intensity, and energy intensity in the radar image data correspond to the RGB of an image channel, respectively; Collecting a visible light image of the target area, and performing dedistortion processing on the visible light image to obtain an image to be processed; Extracting first feature data of the radar two-dimensional image data and extracting second feature data of the image to be processed, suppressing the extracted features using a normalization algorithm, and adding a proportional factor for measuring the importance of the features to the first feature data and the second feature data respectively; The first feature data and the second feature data are fused, and the fused feature data are processed with target confidence loss, category confidence loss, and coordinate regression loss to obtain confident feature data; Detect and identify target objects based on confident feature data and extract target objects in water surface environments.

2. The method according to claim 1, characterized in that After converting the three-dimensional data in the radar coordinate system in the image into radar image data in the camera coordinate system, the method further includes: Performing variable convolution on the two-dimensional data in the radar coordinate system in the first image, that is, adding a displacement variable of the two-dimensional data to the convolution layer of the two-dimensional data, and changing the receptive field range of the two-dimensional data by the displacement variable so that the receptive field range is a polygon; The displacement variables are obtained by online learning based on two-dimensional data; or the displacement variables are obtained based on preset values ​​corresponding to different application scenarios.

3. The method according to claim 1, characterized in that The method also includes adding a scaling factor measuring the importance of the feature to the data by: Among them, BN() is the batch normalization operation, B in is the input three-dimensional data, and are the mean and standard deviation of the mini-batch β, γ and β are trainable affine transformation parameters, and ∈ is a preset constant.

4. The method according to claim 3, characterized in that The method further includes processing the target confidence loss, the category confidence loss and the coordinate regression loss on the fused feature data in the following manner: in, and It is expressed as the binary cross entropy loss, P0 is expressed as the target confidence score in the prediction box, and P IOU It is expressed as the IOU value of the target box corresponding to the predicted box, w obj With w cls is the weight of the positive sample, IoU represents the intersection-over-union ratio between two bounding rectangles, α represents the weight coefficient, ν represents the consistency of the relative proportions of the two rectangles, b and b gt represents the center points of the two bounding rectangular boxes, ρ represents the Euclidean distance between the two bounding rectangular boxes, and c represents the distance between the diagonals of the closure areas of the two bounding rectangular boxes.

5. The method according to claim 4, characterized in that The dedistortion processing of the visible light image includes: The radial distortion of an image is determined by: x dr =x(1+k1r 2 +k2r 4 +k3r 6 ) and dr =y(1+k1r 2 +k2r 4 +k3r 6 ) r 2 =x 2 +y 2 Among them, (x, y) is the ideal undistorted image coordinate system coordinate, (x dr ,y dr ) is the coordinate of the pixel point of the distorted image, k1, k2, k3 are the distortion coefficients, which are obtained when calibrating the camera; r 2 =x 2 +y 2 The tangential distortion of the image is determined by: x dt =2p1xy+p2(r 2 +2x 2 )+1 <h2 style=";text-align:left;direction:ltr">y<h2 style=";text-align:left;direction:ltr"> dt <h2 style=";text-align:left;direction:ltr"> =2p1(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2y<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> )+2p2xy+1 Among them, (x dt ,y dt ) is the coordinate of the pixel point of the distorted image, p1 and p2 are the distortion coefficients, which are obtained when calibrating the camera; The sum of radial distortion and tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined by the following method: x d =x dr +x dt and d =and dr +y dt The visible light image is dedistorted using the determined distortion parameters to obtain a dedistorted visible light image.

6. A target detection device based on millimeter wave radar and visible light fusion, characterized in that: The device comprises: A radar image processing unit, for collecting a first image of a target area in a water surface environment based on a millimeter wave radar, converting three-dimensional data in a radar coordinate system in the first image into two-dimensional image data in a camera coordinate system, so that the distance, Doppler intensity, and energy intensity in the radar image data correspond to the RGB of an image channel respectively; An acquisition unit is used to acquire a visible light image of a target area and perform a dedistortion process on the visible light image to obtain an image to be processed; A feature processing unit, used to extract first feature data of the radar two-dimensional image data and second feature data of the image to be processed, suppress the extracted features using a normalization algorithm, and add a proportional factor measuring the importance of the features to the first feature data and the second feature data respectively; A loss processing unit, used to fuse the first feature data and the second feature data, and process the fused feature data with target confidence loss, category confidence loss, and coordinate regression loss to obtain confident feature data; The recognition unit is used to detect and recognize the target object based on the confirmed feature data, and extract the target object in the water surface environment.

7. The device according to claim 6, characterized in that The device also includes: A convolution unit performs variable convolution on the two-dimensional data in the radar coordinate system in the first image, that is, adds a displacement variable of the two-dimensional data to the convolution layer of the two-dimensional data, and changes the receptive field range of the two-dimensional data through the displacement variable so that the receptive field range is a polygon; The displacement variables are obtained by online learning based on two-dimensional data; or the displacement variables are obtained based on preset values ​​corresponding to different application scenarios.

8. The device according to claim 6, characterized in that The feature processing unit is further used for: Add a scaling factor to the data that measures the importance of the feature by: Among them, BN() is the batch normalization operation, B in is the input three-dimensional data, and are the mean and standard deviation of the mini-batch β, γ and β are trainable affine transformation parameters, and ∈ is a preset constant.

9. The device according to claim 8, characterized in that The loss processing unit is further used to process the target confidence loss, the category confidence loss and the coordinate regression loss on the fused feature data in the following manner: in, and It is expressed as the binary cross entropy loss, P0 is expressed as the target confidence score in the prediction box, and P IOU It is expressed as the IOU value of the target box corresponding to the predicted box, w obj With w cls is the weight of the positive sample, IoU represents the intersection-over-union ratio between two bounding rectangles, α represents the weight coefficient, ν represents the consistency of the relative proportions of the two rectangles, b and b gt represents the center points of the two bounding rectangular boxes, ρ represents the Euclidean distance between the two bounding rectangular boxes, and c represents the distance between the diagonals of the closure areas of the two bounding rectangular boxes.

10. The device according to claim 9, characterized in that The acquisition unit is further used for: The radial distortion of an image is determined by: x dr =x(1+k1r 2 +k2r 4 +k3r 6 ) and dr =y(1+k1r 2 +k2r 4 +k3r 6 ) r 2 =x 2 +y 2 Among them, (x, y) is the ideal undistorted image coordinate system coordinate, (x dr ,y dr ) is the coordinate of the pixel point of the distorted image, k1, k2, k3 are the distortion coefficients, which are obtained when calibrating the camera; r 2 =x 2 +y 2 The tangential distortion of the image is determined by: x dt =2p1xy+p2(r 2 +2x 2 )+1 <h2 style=";text-align:left;direction:ltr">y<h2 style=";text-align:left;direction:ltr"> dt <h2 style=";text-align:left;direction:ltr"> =2p1(r<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> +2y<h2 style=";text-align:left;direction:ltr"> 2 <h2 style=";text-align:left;direction:ltr"> )+2p2xy+1 Among them, (x dt ,y dt ) is the coordinate of the pixel point of the distorted image, p1 and p2 are the distortion coefficients, which are obtained when calibrating the camera; The sum of radial distortion and tangential distortion is taken as the overall distortion, and the overall distortion of the visible light image is determined by the following method: x d =x dr +x dt and d =and dr +y dt The visible light image is dedistorted using the determined distortion parameters to obtain a dedistorted visible light image.

Images