An adaptive weighted data fusion method based on vision and multi-source radar

Abstract

The present invention provides an adaptive weighted data fusion method based on vision and multi-source radar. The method obtains data of a target ship through a target detection model and expresses the data with image coordinates to obtain image information; obtains AIS information and radar data, and converts the radar data into the same coordinate system under AIS to obtain data information; performs data fusion on the image information and the data information according to a preset rule to obtain fused data, obtains the position of the target ship in the image information by a visual sensor and obtains the position of the target ship in the data information by a radar sensor, and fuses the two, thereby improving the accuracy of the fusion of visual information with radar and AIS position information.

Description

Technical Field

[0001] The present invention relates to the field of radar technology, and in particular to an adaptive weighted data fusion method based on vision and multi-source radar. Background Art

[0002] During maritime navigation, single-source sensors are used for environmental perception. Although each sensor has its own advantages, the perception effect is slightly insufficient. Visual information is used for target detection. Since the accuracy of visual information in measuring physical distance information is relatively low, and visual-based target detection becomes unstable in extreme weather, the introduction of radar and AIS position information for fusion can improve the stability of target detection and maintain the accuracy of distance information. Usually, when fusing visual information with position information, the IOU matching algorithm or azimuth matching algorithm is used for information fusion, but these methods will produce information fusion errors.

[0003] Video target detection cannot detect distant targets, and the accuracy of existing ship target detection technology is still insufficient; radar can detect distant ships and monitor target movements in real time, but it cannot obtain detailed data of the target; AIS can conduct detailed detection of cooperative targets in a certain area and identify detailed data such as the identity and type of the other party, but it cannot provide data when the target is not an AIS cooperative ship.

[0004] Single-source sensors may fail suddenly or be interfered by the external environment and collect false information. If multi-source information is directly fused without data verification, the accuracy of the fusion will be greatly affected and the fusion accuracy will be reduced.

[0005] Therefore, an adaptive weighted data fusion method based on vision and multi-source radar is needed to improve the accuracy of the fusion of visual information with radar and AIS position information. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In order to solve the above problems in the prior art, the present invention provides an adaptive weighted data fusion method based on vision and multi-source radar, which can improve the accuracy of the fusion of visual information with radar and AIS position information.

[0008] (II) Technical solution

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] An adaptive weighted data fusion method based on vision and multi-source radar includes the following steps:

[0011] S1. Obtain the data of the target ship through the target detection model and express it with image coordinates to obtain image information;

[0012] S2, obtaining AIS information and radar data, converting the radar data into the same coordinate system under AIS, and obtaining data information;

[0013] S3. Perform data fusion on the image information and the data information according to a preset rule to obtain fused data.

[0014] (III) Beneficial effects

[0015] The beneficial effects of the present invention are as follows: the data of the target ship is obtained through a target detection model, and is represented by image coordinates to obtain image information; AIS information and radar data are obtained, and the radar data is converted into the same coordinate system under AIS to obtain data information; the image information and the data information are fused according to preset rules to obtain fused data, and the position of the target ship in the image information is obtained by a visual sensor and the position of the target ship in the data information is obtained by a radar sensor, and the two are fused, thereby improving the accuracy of the fusion of visual information with radar and AIS position information. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The present invention is a flowchart of an adaptive weighted data fusion method based on vision and multi-source radar according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] In order to better explain the present invention and facilitate understanding, the present invention is described in detail below through specific implementation modes in conjunction with the accompanying drawings.

[0018] Please refer to Figure 1 , an adaptive weighted data fusion method based on vision and multi-source radar, comprising the steps of:

[0019] S1. Obtain the data of the target ship through the target detection model and express it with image coordinates to obtain image information;

[0020] Step S1 is specifically as follows:

[0021] S11, obtaining target ship data through a target detection model, wherein the target ship data includes the type and location information of the target ship;

[0022] S12. Represent the position information of the target ship with image coordinates to obtain image information.

[0023] Specifically, the width of the target ship and the projected length on the camera can be derived according to the position information, and the ship distance can be obtained according to the focal length of the camera, the projected length of the target on the camera, and the actual length of the ship;

[0024] The step S11 is specifically as follows:

[0025] S111, collecting target ship images on the network and target ship images taken by the actual ship camera through sensors, and marking all collected images as yolo format files using labelImg;

[0026] S112, training a target detection model of a required ship through the YOLOv5 target detection algorithm according to the YOLO format file;

[0027] S113. Obtaining data of the target ship according to the target detection model.

[0028] S2, obtaining AIS information and radar data, converting the radar data into the same coordinate system under AIS, and obtaining data information;

[0029] Step S2 is specifically as follows:

[0030] S21, calibrating the camera using Zhang Zhengyou's camera calibration method to obtain the intrinsic parameters, extrinsic parameters and distortion coefficient of the camera;

[0031] S22, after obtaining the AIS information and the radar data through the sensor and performing alignment processing, the radar data is converted into the same coordinate system as the AIS information to obtain a 3D coordinate data set;

[0032] S23, solving the projected 2D coordinates according to the intrinsic parameters, extrinsic parameters, distortion coefficients and 3D coordinate data set of the camera, and converting them into an image coordinate system to obtain data information.

[0033] S3. Perform data fusion on the image information and the data information according to a preset rule to obtain fused data.

[0034] Step S3 is specifically as follows:

[0035] S31, calculating the optimal weighting factor of each sensor;

[0036] S32. Fusing the image information and data information according to the optimal weighting factors of all sensors to obtain fused data.

[0037] Step S31 is specifically as follows:

[0038] Calculate the estimated threshold for each sensor;

[0039]

[0040] In the formula, X i(k) represents the measurement value of sensor i at the kth measurement, Rii(k) represents the time estimation threshold of the measurement value when sensor i measures k times;

[0041] Calculate the time estimation threshold R between different sensors ij (k);

[0042]

[0043] In the formula, X i (k) is the k-th measurement value of sensor i, X j (k) is the k-th measurement value of sensor j, Rij(k) represents the time estimation threshold of the k-th measurement values ​​of sensor i and sensor j;

[0044] All R ij The mean of (k) is taken as the measured mean R at time k ij ;

[0045]

[0046] Calculate the variance of each sensor;

[0047]

[0048] In the formula, σ i 2 is the variance of sensor i;

[0049] calculating an optimal weighting factor for each sensor based on the variance;

[0050]

[0051] Where W i * is the optimal weighting factor of sensor i.

[0052] The calculation process of the fused data is as follows:

[0053]

[0054] Where n is the total number of sensors.

[0055] Embodiment 2

[0056] The difference between this embodiment and the first embodiment is that this embodiment will further illustrate how the above-mentioned adaptive weighted data fusion method based on vision and multi-source radar of the present invention is implemented in combination with specific application scenarios:

[0057] Step 1: Collect target ship images on the network and those taken by the actual ship camera through sensors, and use labelImg to mark all collected images as yolo format files;

[0058] Step 2: calibrate the camera using Zhang Zhengyou's camera calibration method to obtain the camera's intrinsic parameters, extrinsic parameters and distortion coefficients;

[0059] Step 3: According to the YOLO format file, a target detection model of the required ship is obtained by training the YOLOv5 target detection algorithm, and the data of the target ship is obtained according to the target detection model, wherein the data of the target ship includes the type and location information of the target ship;

[0060] Step 4: After obtaining AIS information and radar data through sensors and performing alignment processing;

[0061] Specifically, the alignment processing is implemented according to the spatial alignment and velocity interpolation algorithm, converting the AIS data information into the coordinate system of the own ship, predicting its motion trajectory, and estimating the position coordinates of the AIS ship at any time;

[0062] Step 5: Convert the radar data into the same coordinate system as the AIS information to obtain a 3D coordinate data set;

[0063] Step 6: Solve the projected 2D coordinates according to the intrinsic parameters, extrinsic parameters, distortion coefficients and 3D coordinate data set of the camera, and convert them to the image coordinate system to obtain data information;

[0064] Step 7, the position information of the target ship obtained in step 3 is expressed by image coordinates to obtain image information;

[0065] Step 8: According to the preset rules, the data information obtained in step 6 and the image information obtained in step 7 are fused to obtain fused data. The specific process is as follows:

[0066] 8.1. Calculate the optimal weighting factor for each sensor;

[0067] 8.2. The image information and data information are fused according to the optimal weighting factors of all sensors to obtain fused data.

[0068] Step 8.1 is as follows:

[0069] Calculate the estimated threshold for each sensor;

[0070]

[0071] In the formula, X i(k) represents the measurement value of sensor i at the kth measurement, Rii(k) represents the time estimation threshold of the measurement value when sensor i measures k times;

[0072] Calculate the time estimation threshold R between different sensors ij (k);

[0073]

[0074] In the formula, X i (k) is the k-th measurement value of sensor i, X j (k) is the k-th measurement value of sensor j, Rij(k) represents the time estimation threshold of the k-th measurement values ​​of sensor i and sensor j;

[0075] All R ij The mean of (k) is taken as the measured mean R at time k ij ;

[0076]

[0077] Calculate the variance of each sensor;

[0078]

[0079] In the formula, σ i 2 is the variance of sensor i;

[0080] calculating an optimal weighting factor for each sensor based on the variance;

[0081]

[0082] Where W i * is the optimal weighting factor of sensor i.

[0083] The calculation process of the fused data is as follows:

[0084]

[0085] Where n is the total number of sensors.

[0086] Specifically, step 8 also includes verifying the data information and image information. In multi-source fusion, a single-source sensor may fail suddenly or be interfered by the external environment to collect false information. If the multi-source information is directly fused without data verification, the fusion accuracy will be greatly affected and the fusion accuracy will be reduced.

[0087] The key to data fusion is to ensure that the variance is minimized and the optimal weighted value is obtained through adaptive methods based on the data, so as to ensure the accurate fusion of visual information and position information.

[0088] IOU matching algorithm, where IOU means intersection-over-union ratio, that is, calculating the ratio of the intersection and union areas of two boxes. When the ratio is higher than the set threshold, it means that the video frame and the radar AIS fusion position frame are detected as the same object, thus associating the position information with the visual information. When using the IOU matching algorithm for fusion, there will be a problem of mismatching when matching adjacent ships.

[0089] The azimuth matching algorithm matches the position information in the 90° field of view of the ship with the visual information according to the overlap of the azimuth range, based on the target azimuth information of the optical camera. The azimuth comes from the angle between the centroid of the visual information frame, the radar AIS fusion position point and the origin of the world coordinates. When the angle is lower than the set threshold, it means that the video frame and the radar AIS fusion position frame are detected as the same object, thereby associating the position information with the visual information. In the process of fusion using the azimuth matching algorithm, mismatching may occur when matching ships at the same angle but different distances.

[0090] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An adaptive weighted data fusion method based on vision and multi-source radar, characterized in that: Includes steps: S1. Obtain the data of the target ship through the target detection model and express it with image coordinates to obtain image information; S2, obtaining AIS information and radar data, converting the radar data into the same coordinate system under AIS, and obtaining data information; S3, fusing the image information and the data information according to a preset rule to obtain fused data; Step S3 is specifically as follows: S31, calculating the optimal weighting factor of each sensor; S32, fusing the image information and the data information according to the optimal weighting factors of all sensors to obtain fused data; Step S31 is specifically as follows: Calculate the estimated threshold for each sensor; Where, X i (k) represents the measurement value of sensor i at the kth measurement, Rii(k) represents the time estimation threshold of the measurement value when sensor i measures k times; Calculate the time estimation threshold R between different sensors ij (k); Where, X i (k) is the k-th measurement value of sensor i, X j (k) is the k-th measurement value of sensor j, Rij(k) represents the time estimation threshold of the k-th measurement values ​​of sensor i and sensor j; All R ij The mean of (k) is taken as the measured mean R at time k ij ; Calculate the variance of each sensor; In the formula, σ i 2 is the variance of sensor i; calculating an optimal weighting factor for each sensor based on the variance; Where W i * is the optimal weighting factor of sensor i; The calculation process of the fused data is as follows: Where n is the total number of sensors.

2. The adaptive weighted data fusion method based on vision and multi-source radar according to claim 1 is characterized in that: Step S1 is specifically as follows: S11, obtaining target ship data through a target detection model, wherein the target ship data includes the type and location information of the target ship; S12. Represent the position information of the target ship with image coordinates to obtain image information.

3. The adaptive weighted data fusion method based on vision and multi-source radar according to claim 2 is characterized in that: The step S11 is specifically as follows: S111, collecting target ship images on the network and target ship images taken by the actual ship camera through sensors, and marking all collected images as yolo format files using labelImg; S112, training a target detection model of a required ship through the YOLOv5 target detection algorithm according to the YOLO format file; S113. Obtaining data of the target ship according to the target detection model.

4. The adaptive weighted data fusion method based on vision and multi-source radar according to claim 1 is characterized in that: Step S2 is specifically as follows: S21, calibrating the camera using Zhang Zhengyou's camera calibration method to obtain the intrinsic parameters, extrinsic parameters and distortion coefficient of the camera; S22, after obtaining the AIS information and the radar data through the sensor and performing alignment processing, the radar data is converted into the same coordinate system as the AIS information to obtain a 3D coordinate data set; S23, solving the projected 2D coordinates according to the intrinsic parameters, extrinsic parameters, distortion coefficients and 3D coordinate data set of the camera, and converting them into an image coordinate system to obtain data information.

Images