Method and apparatus for ship positioning using a radar-assisted camera based on multi-source data fusion

Abstract

The present invention provides a method and device for ship positioning of a radar-assisted camera based on multi-source data fusion, which relates to the technical field of target monitoring and positioning. The method includes: obtaining multi-source data of a ship; wherein, the multi-source data includes radar data, camera data and auxiliary data; constructing a unified spatial coordinate system for the multi-source data, and determining the camera observation range based on the unified spatial coordinate system; performing multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; extracting radar features and camera features for the candidate ships, and determining fusion features; and determining a target ship matching the radar data from the candidate ships according to the fusion features. This solution improves the accuracy and reliability of ship target matching and positioning in border and coastal defense.

Description

Technical Field

[0001] The present invention relates to the technical field of target monitoring and positioning, in particular to the technical field of ship track tracking, and particularly to a ship positioning method and device based on a radar-assisted camera with multi-source data fusion. Background Art

[0002] In fields such as border and coastal defense security guarantee and marine resource management, accurately mastering the position of ships in the camera's field of view is crucial. As common monitoring devices, radar and cameras each have their advantages and disadvantages. Radar can detect ships at a long distance and obtain key information such as distance, azimuth angle, and speed, but it is insufficient in the details of target recognition; cameras can visually present the appearance features of ships. However, limited by the detection distance and environmental factors, the positioning accuracy is limited. Relying solely on their own data for ship positioning is easily interfered by factors such as sea clutter, weather changes, and target occlusion, resulting in inaccurate positioning and large matching errors, and it is difficult to meet the actual application requirements. Therefore, there is an urgent need to provide a ship positioning method and device based on a radar-assisted camera with multi-source data fusion. Summary of the Invention

[0003] The present invention provides a ship positioning method and device based on a radar-assisted camera with multi-source data fusion, which improves the accuracy and reliability of ship target matching and positioning in border and coastal defense scenarios.

[0004] In a first aspect, the present invention provides a ship positioning method based on a radar-assisted camera with multi-source data fusion, including:

[0005] Obtaining multi-source data of a ship; wherein, the multi-source data includes radar data, camera data, and auxiliary data;

[0006] Constructing a unified spatial coordinate system for the multi-source data, and determining the camera observation range based on the unified spatial coordinate system;

[0007] Performing multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships;

[0008] Performing radar feature extraction and camera feature extraction on the candidate ships, and determining the fusion features;

[0009] Determining the target ship from the candidate ships according to the fusion features.

[0010] Optionally, the radar for collecting the radar data and the camera for collecting the camera data are both located on an optoelectronic turntable;

[0011] The constructing a unified spatial coordinate system for the multi-source data, and determining the camera observation range based on the unified spatial coordinate system includes:

[0012] Take the installation position of the optoelectronic turntable as the origin of the coordinate system, and establish a unified space coordinate system;

[0013] Convert the original polar coordinates of the radar data into rectangular coordinates in the unified space coordinate system;

[0014] According to the position of the camera in the unified space coordinate system, the pitch angle, azimuth angle of the optoelectronic turntable when collecting the camera data, the field of view angle and the maximum detection distance of the camera, determine the spatial observation range of the camera;

[0015] Calculate the coordinate interval of the spatial observation range of the camera in the unified space coordinate system, and determine the coordinate interval as the camera observation range.

[0016] Optionally, the multi-level target screening is performed within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships, including:

[0017] Determine the ship targets included in the radar data located within the camera observation range as the first ship targets;

[0018] Screen the first ships according to the auxiliary data to obtain the second ships;

[0019] Calculate the similarity between the current navigation trajectory of the second ship and the historical ship operation trajectory;

[0020] Screen the second ships according to the similarity to obtain the candidate ships.

[0021] Optionally, the screening of the first ships according to the auxiliary data to obtain the second ships includes:

[0022] For each of the first ships, perform:

[0023] According to the ship navigation plan data and meteorological data of the first ship, judge whether the included angle between the target course in the ship navigation plan data of the first ship and the wind direction in the meteorological data is greater than a preset included angle, and whether the navigation speed is less than a preset navigation speed;

[0024] If the judgment result is yes, then the first ship is a drifting target, and the drifting target is eliminated;

[0025] If the judgment result is no, then obtain the deviation distance between the position of the first ship and the nearest waterway included in the electronic nautical chart, and judge whether the deviation distance is greater than a preset deviation threshold;

[0026] When the judgment result is yes, then the first ship is an abnormal target, and the abnormal target is eliminated;

[0027] When the judgment result is negative, the first ship is determined as the second ship.

[0028] Optionally, the radar feature extraction and camera feature extraction are performed on the candidate ship, and the fusion feature is determined, including:

[0029] Performing radar feature extraction according to the radar data of the candidate ship to obtain a radar feature vector;

[0030] Comparing the radar feature vector of the candidate ship with the radar feature vectors of the ships identical to the candidate ship included in the historical database to obtain a radar feature distance;

[0031] Performing camera feature extraction according to the camera data of the candidate ship to obtain a camera feature vector;

[0032] Comparing the camera feature vector of the candidate ship with the camera feature vectors of the ships identical to the candidate ship included in the historical database to obtain a camera feature distance;

[0033] Obtaining the environmental parameters of the scene where the candidate ship is located; wherein, the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity, and atmospheric turbulence intensity;

[0034] Fusing the radar feature distance and the camera feature distance according to the environmental parameters to obtain the fusion feature.

[0035] Optionally, the fusion feature is determined by the following formula:

[0036]

[0037] Wherein, d h is the fusion feature distance used to characterize the fusion feature; w i is the weight value of the i th environmental parameter; f i ( X i ) is the influence function of the i th environmental parameter; d r is the radar feature distance; d c is the camera feature distance.

[0038] Optionally, determining the target ship from the candidate ships according to the fusion feature includes:

[0039] Performing radar feature extraction and camera feature extraction on the target ship based on the radar data and known camera data of the target ship to obtain target fusion features;

[0040] Comparing the target fusion features with the fusion features of the candidate ships to obtain the fusion features that match the target fusion features;

[0041] Determining the candidate ship corresponding to the fusion features as the target ship.

[0042] In a second aspect, the present invention also provides a ship positioning device for a radar-assisted camera based on multi-source data fusion, including:

[0043] A data acquisition module for acquiring multi-source data of a ship; wherein, the multi-source data includes radar data, camera data, and auxiliary data;

[0044] A coordinate processing module for constructing a unified space coordinate system for the multi-source data and determining a camera observation range based on the unified space coordinate system;

[0045] A target screening module for performing multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships;

[0046] A feature matching module for performing radar feature extraction and camera feature extraction on the candidate ships and determining fusion features; and determining a target ship from the candidate ships according to the fusion features.

[0047] In a third aspect, the present invention also provides a computing device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the ship positioning method for a radar-assisted camera based on multi-source data fusion described in any one of the above is implemented.

[0048] In a fourth aspect, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, the computer is made to execute the ship positioning method for a radar-assisted camera based on multi-source data fusion described in any one of the above.

[0049] In a fifth aspect, an embodiment of the present invention also provides a computer program product, including computer instructions, and when the computer instructions are executed by a processor, the steps of the method described in any one of the first aspects of this specification are implemented.

[0050] The present invention provides a method and device for ship positioning of a radar-assisted camera based on multi-source data fusion. By fusing multi-source data and making full use of the advantages of each data, it assists in the matching and positioning of radar targets and ship targets in the camera field of view from multiple angles, effectively overcoming the problems such as inaccuracy and large errors that are prone to occur when relying solely on the data of the radar and the camera itself, and significantly improving the accuracy and reliability of ship target matching and positioning. Considering the complex and changeable border and coastal defense environment, the present invention also designs a real-time tracking and dynamic matching adjustment mechanism, enabling the system to have good adaptability and robustness, and being able to stably and continuously complete the ship target matching and positioning task under various complex working conditions, providing strong support for border and coastal defense security guarantee work. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0052] Figure 1 is a flowchart of a method for ship positioning of a radar-assisted camera based on multi-source data fusion provided by an embodiment of the present invention;

[0053] Figure 2 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention;

[0054] Figure 3 is a structural diagram of a device for ship positioning of a radar-assisted camera based on multi-source data fusion provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0056] The following describes the concept of the present invention. Please refer to Figure 1 , an embodiment of the present invention provides a method for ship positioning of a radar-assisted camera based on multi-source data fusion, including:

[0057] Step 100, obtaining multi-source data of the ship; wherein, the multi-source data includes radar data, camera data, and auxiliary data;

[0058] Step 102: Construct a unified spatial coordinate system for multi-source data and determine the camera observation range based on the unified spatial coordinate system;

[0059] Step 104: According to the radar data and auxiliary data, perform multi-level target screening within the camera observation range to obtain candidate vessels;

[0060] Step 106: Extract radar features and camera features from the candidate vessels and determine the fusion features;

[0061] Step 108: Determine the target vessel from the candidate vessels according to the fusion features.

[0062] In the present invention, first, the multi-source data such as the radar data, camera data, and auxiliary data of the vessel are unified in coordinate system to construct a unified spatial coordinate system, and then the camera observation range is determined based on the unified spatial coordinate system. Further, based on the camera observation range and the multi-source data, multi-level screening is performed on the current vessel to obtain candidate vessels. Then, the radar features and camera features are extracted from the candidate vessels to determine the fusion features, so as to determine the target vessel through the fusion features that integrate various features. In this way, by fusing multi-source data, the present invention makes full use of the advantages of each data, assists in the matching and positioning of the radar target and the vessel target in the camera field of view from multiple angles, effectively overcomes the problems such as inaccuracy and large error that are prone to occur when only relying on the data of the radar and the camera itself for matching, and significantly improves the accuracy and reliability of the matching and positioning of the vessel target.

[0063] The following describes Figure 1 the execution manner of each step shown.

[0064] First, in step 100, real-time radar data, camera data, and other multi-source auxiliary data are collected, and the multi-source data is cleaned and processed. The auxiliary data includes data such as AIS, meteorology, electronic nautical charts, satellite remote sensing, ship navigation plans, and historical voyages. The radar can collect information such as the distance, azimuth angle, elevation angle, speed, and radar echo intensity of ship targets within the target area in real time. For the radar data, the preprocessing includes: using the ordered-statistic constant false alarm rate (OS-CFAR) algorithm for clutter suppression, and at the same time using the DBSCAN algorithm for target clustering on the radar point cloud data. In the present invention, the neighborhood radius ϵ = 0.2 nautical miles and the minimum number of points minPts = 3 are set based on the ship distribution characteristics in the border and coastal defense scenarios. For the camera data, the preprocessing includes: using the CLAHE algorithm for image enhancement, with a block size of 8×8 and a contrast limit threshold of 2.0; extracting based on the YOLOv5s model, with an input resolution of 640×640 and a confidence threshold of 0.7. For the auxiliary data, the preprocessing of AIS data includes: performing consistency verification on the AIS speed and the radar speed. If the deviation exceeds 15%, triggering the data retransmission mechanism; the preprocessing of meteorological data includes: using the Kriging interpolation method to convert the discrete meteorological station data into continuous distribution in the monitoring area.

[0065] It should be noted that AIS is the Automatic Identification System for Ships, which refers to a new type of navigation aid system applied to maritime safety and communication between ships and the shore, and between ships. AIS exchanges information with surrounding ships through the VHF band, and is used to provide detailed information about targets such as ship names, positions, headings, etc.

[0066] In step 102, both the radar for collecting radar data and the camera for collecting camera data are located on the optoelectronic turret.

[0067] A unified space coordinate system is constructed for the multi-source data, and the camera observation range is determined based on the unified space coordinate system, including:

[0068] Taking the installation position of the optoelectronic turret as the origin of the coordinate system, and establishing a unified space coordinate system;

[0069] Converting the original polar coordinates of the radar data into rectangular coordinates in the unified space coordinate system;

[0070] According to the position of the camera in the unified space coordinate system, the elevation angle and azimuth angle of the optoelectronic turret when collecting camera data, the field of view angle and the maximum detection distance of the camera, determine the spatial observation range of the camera;

[0071] Calculate the coordinate interval of the spatial observation range of the camera in the unified space coordinate system, and determine the coordinate interval as the camera observation range.

[0072] Specifically, taking the installation position O of the optoelectronic turret ( x 0,y 0, z 0) is the origin of the coordinate system. The optoelectronic turntable is the physical carrier platform for the radar and the camera. Its installation point can be directly measured and has high stability, making it suitable as a spatial reference. The axis direction is defined as follows: the X-axis points to the true north direction (calibrated by a high-precision magnetometer or GNSS azimuth, with an error < 0.1°); the Y-axis is perpendicular to the X-axis and points to the due east direction; the Z-axis is perpendicular to the X-Y plane and points upward, aligned with the local plumb line (corrected by an inclinometer, with an error < 0.05°);

[0073] The original data of the radar detecting ship targets is in polar coordinate form ( r , α , β ), which needs to be converted to diameter coordinates ( X , Y , Z ) in the unified space coordinate system; the coordinate conversion formula is:

[0074]

[0075]

[0076]

[0077] Among them, r is the straight-line distance between the radar and the ship target, α is the pitch angle between the radar and the ship target, β is the azimuth angle (the clockwise angle of the ship target relative to the true north direction), Δ hcurvature is the earth curvature correction amount. It should be noted that when the detection distance r > 10 nautical miles (about 18.52 km), the influence of the earth curvature on the height needs to be corrected; since the earth's surface is a curved surface, the actual height of a long-distance target needs to subtract the virtual height caused by the curvature. At this time, the true height of the ship target is , , Re is the average radius of the earth;

[0078] For the extrinsic calibration of the radar-camera, it includes: checkerboard calibration: placing a high-reflectivity checkerboard calibration plate at a known position, which is detected by both the radar and the camera simultaneously;

[0079] Feature point matching: extracting the coordinates of the calibration plate corner points in the radar point cloud (radar data) and the camera image (camera data);

[0080] Least squares optimization: calculating the extrinsic parameter matrix Text ∈ R 4×4 (including the rotation matrix R and the translation vectort to minimize the reprojection error;

[0081] Finally, it is achieved that: angular error: < 0.05° (guaranteed by a high-precision optoelectronic turntable), translational error: < 5 cm (calibrated by a laser rangefinder);

[0082] Finally, by deploying reflection targets at known coordinate positions of static targets and measuring their positions through radar and cameras respectively to verify the unified spatial coordinate system; for dynamic targets, an unmanned aerial vehicle is used to fly along a preset trajectory, and the consistency of radar, camera, and GNSS positioning data is compared. The acceptance criteria are that the static target positioning error is < 0.5 m; the dynamic target tracking error is < 1.0 m (speed ≤ 30 knots).

[0083] Specifically, determining the camera observation range includes: based on the azimuth angle and elevation angle of the optoelectronic turntable, as well as the field of view angle and maximum detection distance of the camera, establishing a quadrangular pyramid frustum model of the camera observation space, then calculating the coordinate interval of the frustum in the unified spatial coordinate system through trigonometric functions, and performing corrections for the earth's curvature and atmospheric refraction, and then using the projection matrix method to determine whether the radar target is within the frustum, and ensuring the accuracy of screening through static and dynamic verification.

[0084] In the present invention, in order to ensure the geometric consistency of multi-source data, the original coordinate systems of the multi-source data are unified into a unified three-dimensional space framework.

[0085] For step 104, according to the radar data and auxiliary data, multi-level target screening is performed within the camera observation range to obtain candidate ships, including:

[0086] Determining the ship targets included in the radar data located within the camera observation range as the first ship targets;

[0087] Screening the first ships according to the auxiliary data to obtain the second ships;

[0088] Calculating the similarity between the current navigation trajectory of the second ships and the historical ship operation trajectories;

[0089] Screening the second ships according to the similarity to obtain candidate ships.

[0090] In a specific embodiment, screening the first ships according to the auxiliary data to obtain the second ships includes:

[0091] For each first ship, the following operations are performed:

[0092] Based on the auxiliary data including the vessel navigation plan data and meteorological data of the first vessel, determine whether the angle between the target course in the vessel navigation plan data of the first vessel and the wind direction in the meteorological data is greater than a preset angle (for example, 90°), and whether the navigation speed is less than a preset navigation speed (for example, 5 knots (i.e., 5 nautical miles per hour));

[0093] If the judgment result is yes, then the first vessel is a drifting target, and the drifting target is excluded;

[0094] If the judgment result is no, then obtain the deviation distance between the position of the first vessel and the nearest waterway included in the electronic nautical chart in the auxiliary data, and determine whether the deviation distance is greater than a preset deviation threshold (for example, 1.5 nautical miles);

[0095] When the judgment result is yes, then the first vessel is an abnormal target, and the abnormal target is excluded;

[0096] When the judgment result is no, determine the first vessel as the second vessel.

[0097] In a specific embodiment, calculate the similarity between the current navigation track of the second vessel and the historical vessel operation track; screen the second vessels according to the similarity to obtain candidate vessels, including:

[0098] For each second vessel, perform the following:

[0099] Calculate the similarity between the current navigation track of the second vessel and the historical vessel operation track, and the similarity is determined by the following formula:

[0100]

[0101] where, D is the similarity; P c is the current navigation track, P c i are the first i position points in the current navigation track; P h is the historical vessel operation track, P h j are the first j position points in the historical vessel operation track;

[0102] When the minimum similarity is greater than a preset threshold (for example, 200), determine that the current navigation track of the second vessel is an abnormal track, and exclude the second vessel;

[0103] ​​When the minimum similarity is not greater than a preset threshold (for example, 200), it is determined that the second ship is a candidate ship. It should be noted that the minimum similarity is the minimum value among the similarities between the current navigation trajectory of the second ship and the historical ship operation trajectories of all historical ships.

[0104] In the present invention, first, based on the spatial position relationship, the range of potential matching targets is quickly narrowed down to determine the first ship target. Then, secondary screening is carried out based on the course data and meteorological data in the auxiliary data to further exclude ships that do not conform to the conventional navigation logic and geographical logic, narrowing the matching range. Finally, based on the current trajectories of the remaining ships and the historical ship operation trajectories in the historical database, further screening is carried out to further exclude ships with abnormal trajectories, obtaining candidate ships and improving the accuracy of matching.

[0105] In the present invention, according to the coordinate interval between the coordinates of the ship target detected by the radar and the camera observation range, preliminary potential matching targets are screened out, that is, those first ship targets that fall within the camera observation range, forming a preliminary set. Then, further screening of the targets is carried out through other multi-source data to narrow the matching range and accurately locate the target ship.

[0106] In step 106, radar feature extraction and camera feature extraction are performed on the candidate ship, and the fusion feature is determined, including:

[0107] Radar feature extraction is performed according to the radar data of the candidate ship to obtain a radar feature vector;

[0108] The radar feature vector of the candidate ship is compared with the radar feature vectors of the ships in the historical database that are the same as the candidate ship to obtain a radar feature distance;

[0109] Camera feature extraction is performed according to the camera data of the candidate ship to obtain a camera feature vector;

[0110] The camera feature vector of the candidate ship is compared with the camera feature vectors of the ships in the historical database that are the same as the candidate ship to obtain a camera feature distance;

[0111] The environmental parameters of the scene where the candidate ship is located are obtained; among them, the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity, and atmospheric turbulence intensity;

[0112] The radar feature distance and the camera feature distance are fused according to the environmental parameters to obtain a fusion feature.

[0113] It should be noted that the historical database includes multi-source data of any ship stored historically. The radar feature distance can be the Euclidean distance, and the camera feature distance can be the Bhattacharyya distance. Preferably, the camera feature extraction adopts Hu moment calculation, and 7 invariant moments are used for rotation-invariant matching; the color histogram is based on the RGB three channels, and after normalization, it is spliced into a 48-dimensional feature vector, and then the Bhattacharyya distance between the two is calculated to measure the similarity of color distribution.

[0114] Specifically, the radar feature vector is ; where are the speed, acceleration, heading angle, and radar cross-section area respectively.

[0115] In the present invention, since the radar data and the camera data are heterogeneous data, it is difficult to directly fuse the radar feature vector and the camera feature vector. Therefore, the present invention first determines a standard ship identical or similar to the candidate ship from the historical database, and then compares the extracted radar feature vector with the radar feature vector of the standard ship to obtain the radar feature distance, and then compares the extracted camera feature vector with the camera feature vector of the standard ship to obtain the camera feature distance, and then fuses based on the radar feature distance and the camera feature distance. In this way, by comparing the feature information of the current ship and the historical identical or similar ships, the fusion of heterogeneous data is converted into the fusion of similarity under different feature information, thereby effectively integrating multi-source data information and further improving the accurate positioning of ship targets.

[0116] In a preferred embodiment, the fusion feature is determined by the following formula;

[0117]

[0118] where d h is the fusion feature distance used to characterize the fusion feature; w i is the weight value of the i th environmental parameter; f i ( X i ) is the influence function of the i th environmental parameter; d r is the radar feature distance; d c is the camera feature distance. Where , i = 1, 2, 3, 4, 5.

[0119] Specifically, i when = 1, f 1( X 1) =f snr ( SNR ) represents the influence function of the signal-to-noise ratio;

[0120]

[0121] Among them, SNR is the signal-to-noise ratio;

[0122] i When = 2, f 2( X 2) = f vis ( VIS ) represents the influence function of visibility;

[0123]

[0124] Among them, VIS is visibility;

[0125] i When = 3, f 3( X 3) = f pre ( PRE ) represents the influence function of precipitation intensity;

[0126]

[0127] Among them, PRE is precipitation intensity;

[0128] i When = 4, f 4( X 4) = f lig ( LIG ) represents the influence function of light intensity;

[0129]

[0130] Among them, LIG is light intensity;

[0131] i When = 5, f 5( X 5) = f tur ( TUR ) represents the influence function of atmospheric turbulence intensity;

[0132]

[0133] Among them, TURis the atmospheric turbulence intensity, which is a normalized value measured by relevant instruments and has no unit of dimension.

[0134] In the present invention, since environmental parameters such as signal-to-noise ratio, visibility, precipitation intensity, light intensity, and atmospheric turbulence intensity in the scene change in real time, and their dynamic changes will have varying degrees of impact on the performance of the radar and the camera, it is necessary to consider the impact of these environmental parameters in the fusion process and further improve the precise positioning of the target ship based on the current scene information.

[0135] In the present invention, when the signal-to-noise ratio ≥ 30, both the radar and the camera can work stably and accurately. The images obtained by the camera are clear, and the radar detection data is accurate. As the signal-to-noise ratio gradually decreases, the decline trend of the camera performance intensifies. To ensure the accuracy of ship positioning, it is necessary to gradually reduce the proportion of the camera feature distance in the fusion feature. When the signal-to-noise ratio ≤ 15, the environmental interference is severe, the imaging quality of the camera is severely damaged, the obtained images are blurred, and it is difficult to accurately extract ship features. The reliability of the camera features is extremely low, so it is necessary to mainly rely on the radar for ship positioning to ensure the effective operation of the system in harsh environments.

[0136] In the present invention, the lower the visibility, the greater the impact on the camera performance. However, when the visibility is between 0 and 2000 meters, the impact value of visibility is inversely proportional to the visibility, and the camera performance gradually decreases. For the precipitation intensity, when the precipitation intensity ≤ 5 mm / h, the impact on the radar and camera performance is small, so the function value is 0. However, as the precipitation intensity gradually increases, this impact value also increases linearly, and the impact on the camera performance is greater. For the light intensity, both too low and too high light intensities will seriously affect camera imaging. However, between 500 and 5250 lux, the impact value will decrease as the light intensity increases. However, between 5250 and 10000 lux, the impact value increases as the light intensity increases. For the atmospheric turbulence intensity, the impact value increases as the atmospheric turbulence intensity increases, and the reliability of the camera decreases. At this time, it is necessary to increase the radar feature weight to ensure the accuracy of ship positioning.

[0137] In a specific embodiment, for the candidate ship - Ship 1, step 106 includes:

[0138] S1. Extract radar features based on the radar data of Ship 1 to obtain a radar feature vector X = [9.26, 0.3, 45, -19.57];

[0139] S2. Find the feature vector of a ship similar to Ship 1 from the historical database Y = [9.0, 0.25, 40, -20], calculate the Euclidean distance between the two to obtain the radar feature distance d r ,

[0140]

[0141] S3. Extract camera features based on the camera data of vessel 1, calculate the Hu moments of the image of vessel 1, and obtain the values of 7 invariant moments, which are respectively ; then calculate the color histogram, divide each of the RGB three channels into 16 bins, and splice them into a 48-dimensional feature vector after normalization P ;

[0142] S4. Find the color histogram feature vector of the vessel corresponding to vessel 1 from the historical database Q , calculate the Bhattacharyya distance between the two to obtain the camera feature distance d c ,

[0143]

[0144] S5. Assume that at a certain moment, the environmental parameters of the scene where vessel 1 is located are obtained as follows: signal-to-noise ratio is 18 dB, visibility is 1200 m, precipitation intensity is 8 mm / h, light intensity is 6000 lux, and atmospheric turbulence intensity is 0.4; according to expert experience, set w 1 = 0.3, w 2 = 0.25, w 3 = 0.2, w 4 = 0.15, w 5 = 0.1; then f snr (18) = 0.8, f vis (1200) = 0.4, f pre (8) = 0.1, f lig (6000) ≈ 0.16, f tur (0.4) ≈ 0.33;

[0145] Fusion feature .

[0146] It should be noted that if the target signal loss time of the candidate vessel exceeds the preset time threshold, or the heading mutation angle of the candidate vessel is greater than the preset angle threshold, then re-matching is triggered, that is, return to step 100. In this way, accurate tracking and matching of the target vessel can be ensured.

[0147] In step 108, determining the target vessel from the candidate vessels according to the fusion feature includes:

[0148] Extract the radar features and camera features of the target ship based on the radar data and known camera data of the target ship to obtain the target fusion features;

[0149] Compare the target fusion features with the fusion features of the candidate ships to obtain the fusion features that match the target fusion features;

[0150] Determine the candidate ship corresponding to the fusion feature as the target ship.

[0151] It should be noted that the known camera data can be collected in advance.

[0152] In the present invention, the method and system for assisting the radar target to locate the ship in the camera field of view based on multi-source data can effectively achieve the accurate matching and positioning of the ship target, providing strong support for border and coastal defense monitoring. In a complex border and coastal defense environment, the system overcomes the limitations of a single sensor through multi-source data fusion and intelligent algorithms, improving the accuracy and reliability of ship positioning, and has important practical application value.

[0153] As Figure 2 、 Figure 3 shown, an embodiment of the present invention provides a ship positioning device based on a radar-assisted camera with multi-source data fusion. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From a hardware perspective, as Figure 2 shown, it is a hardware architecture diagram of a computing device where the ship positioning device based on a radar-assisted camera with multi-source data fusion provided by an embodiment of the present invention is located. In addition to Figure 2 the processor, memory, network interface, and non-volatile memory shown, the computing device where the device is located in the embodiment usually may also include other hardware, such as a forwarding chip responsible for processing packets, and so on. Taking software implementation as an example, as Figure 3 shown, as a logically meaningful device, it is formed by the CPU of its computing device reading the corresponding computer program in the non-volatile memory into the memory and running. An embodiment of the present invention provides a ship positioning device based on a radar-assisted camera with multi-source data fusion, including:

[0154] A data acquisition module 300, configured to acquire multi-source data of a ship; wherein, the multi-source data includes radar data, camera data, and auxiliary data;

[0155] A coordinate processing module 302, configured to construct a unified space coordinate system for the multi-source data, and determine the camera observation range based on the unified space coordinate system;

[0156] A target screening module 304, configured to perform multi-level target screening within the camera observation range according to the radar data and auxiliary data to obtain candidate ships;

[0157] A feature matching module 306, configured to perform radar feature extraction and camera feature extraction on candidate vessels, and determine fused features; and determine a target vessel from the candidate vessels according to the fused features.

[0158] In some specific embodiments, the data acquisition module 300 may be configured to perform the above-mentioned step 100, the coordinate processing module 302 may be configured to perform the above-mentioned step 102, the target screening module 304 may be configured to perform the above-mentioned step 104, and the feature matching module 306 may be configured to perform the above-mentioned steps 106 and 108.

[0159] In some specific embodiments, both the radar for acquiring radar data and the camera for acquiring camera data are located on an optoelectronic turntable;

[0160] The coordinate processing module 302 is further configured to perform the following operations:

[0161] Taking the installation position of the optoelectronic turntable as the origin of the coordinate system, and establishing a unified space coordinate system;

[0162] Converting the original polar coordinates of the radar data into rectangular coordinates in the unified space coordinate system;

[0163] Determining the spatial observation range of the camera according to the position of the camera in the unified space coordinate system, the pitch angle, azimuth angle of the optoelectronic turntable when acquiring camera data, the field of view angle of the camera, and the maximum detection distance;

[0164] Calculating the coordinate interval of the spatial observation range of the camera in the unified space coordinate system, and determining the coordinate interval as the camera observation range.

[0165] In some specific embodiments, the target screening module 304 is further configured to perform the following operations:

[0166] Determining the vessel targets included in the radar data located within the camera observation range as first vessel targets;

[0167] Screening the first vessels according to the auxiliary data to obtain second vessels;

[0168] Calculating the similarity between the current navigation trajectory of the second vessel and the historical vessel operation trajectories;

[0169] Screening the second vessels according to the similarity to obtain candidate vessels.

[0170] In some specific embodiments, the feature matching module 306 is further configured to perform the following operations:

[0171] Performing radar feature extraction according to the radar data of the candidate vessels to obtain radar feature vectors;

[0172] Compare the radar feature vector of the candidate ship with the radar feature vectors of the same ships included in the historical database to obtain the radar feature distance;

[0173] Extract camera features from the camera data of the candidate ship to obtain a camera feature vector;

[0174] Compare the camera feature vector of the candidate ship with the camera feature vectors of the same ships included in the historical database to obtain the camera feature distance;

[0175] Obtain the environmental parameters of the scene where the candidate ship is located; among them, the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity, and atmospheric turbulence intensity;

[0176] Fuse the radar feature distance and the camera feature distance according to the environmental parameters to obtain a fusion feature; the fusion feature is determined by the following formula:

[0177]

[0178] Where d h Is the fusion feature distance used to characterize the fusion feature; w i Is the weight value of the i th environmental parameter; f i ( X i ) Is the influence function of the i th environmental parameter; d r Is the radar feature distance; d c Is the camera feature distance.

[0179] In some specific embodiments, the feature matching module 306 is further configured to perform the following operations:

[0180] Perform radar feature extraction and camera feature extraction on the target ship based on the radar data and known camera data of the target ship to obtain a target fusion feature;

[0181] Compare the target fusion feature with the fusion feature of the candidate ship to obtain a fusion feature that matches the target fusion feature;

[0182] Determine the candidate ship corresponding to the fusion feature as the target ship.

[0183] It can be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on a ship positioning device of a radar-assisted camera based on multi-source data fusion. In some other embodiments of the present invention, a ship positioning device of a radar-assisted camera based on multi-source data fusion may include more or fewer components than those illustrated, or combine certain components, or split certain components, or have different component arrangements. The illustrated components can be implemented in hardware, software, or a combination of software and hardware.

[0184] Regarding the information interaction, execution process, etc. between the various modules within the above-mentioned device, since it is based on the same concept as the method embodiments of the present invention, the specific content can be referred to the description in the method embodiments of the present invention and will not be elaborated here.

[0185] The embodiments of the present invention further provide a computing device, including a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, it implements a ship positioning method of a radar-assisted camera based on multi-source data fusion in any one of the embodiments of the present invention.

[0186] The embodiments of the present invention further provide a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it causes the processor to execute a ship positioning method of a radar-assisted camera based on multi-source data fusion in any one of the embodiments of the present invention.

[0187] The embodiments of the present application further provide a computer program product. The computer program product includes a computer program. The processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes a ship positioning method of a radar-assisted camera based on multi-source data fusion described in any one of the above embodiments.

[0188] Specifically, a system or device equipped with a storage medium can be provided. A software program code for implementing the functions in any one of the above embodiments is stored on the storage medium, and the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0189] In this case, the program code read from the storage medium itself can implement the functions of any one of the above embodiments. Therefore, the program code and the storage medium storing the program code constitute a part of the present invention.

[0190] Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Optionally, the program code can be downloaded from a server computer via a communication network.

[0191] In addition, it should be clear that not only can the functions of any one of the above embodiments be realized by executing the program code read by a computer, but also by causing an operating system or the like operating on the computer to perform part or all of the actual operations based on the instructions of the program code.

[0192] Furthermore, it can be understood that the program code read from the storage medium is written into the memory provided in an expansion board inserted into the computer or into the memory provided in an expansion module connected to the computer, and then the CPU or the like installed on the expansion board or the expansion module is caused to perform part and all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0193] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "comprising a..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0194] Those of ordinary skill in the art can understand that all or part of the steps for implementing the above method embodiments can be completed by hardware related to program instructions. The foregoing program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the foregoing storage medium includes various media such as ROM, RAM, magnetic disks, or optical disks that can store program code.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for ship positioning of a radar-assisted camera based on multi-source data fusion, characterized in that Including: Obtain multi-source data of a ship; wherein, the multi-source data includes radar data, camera data, and auxiliary data including ship navigation plan data, meteorological data, electronic nautical charts, and historical ship operation trajectories; Construct a unified spatial coordinate system for the multi-source data, and determine the camera observation range based on the unified spatial coordinate system; According to the radar data and the auxiliary data, perform multi-level target screening within the camera observation range to obtain candidate ships; Extract radar features from the radar data of the candidate ships to obtain radar feature vectors; Compare the radar feature vectors of the candidate ships with the radar feature vectors of the ships in the historical database that are the same as the candidate ships to obtain radar feature distances; Extract camera features from the camera data of the candidate ships to obtain camera feature vectors; Compare the camera feature vectors of the candidate ships with the camera feature vectors of the ships in the historical database that are the same as the candidate ships to obtain camera feature distances; Obtain the environmental parameters of the scene where the candidate ships are located; the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity, and atmospheric turbulence intensity; Fuse the radar feature distance and the camera feature distance according to the environmental parameters to obtain a fused feature; Among them, d h is the fusion feature distance for characterizing the fusion feature; w i is the weight value of the i th environmental parameter; f i ( X i ) is the influence function of the i th environmental parameter; d r is the radar feature distance; d c is the camera feature distance; Determine the target ship from the candidate ships according to the fused feature; Wherein, the candidate ships are obtained through the following method: Determine the ship targets included in the radar data located within the camera observation range as the first ship targets; Screen the first ships according to the auxiliary data to obtain the second ships; Calculate the similarity between the current navigation trajectory of the second ship and the historical ship operation trajectory; Screen the second ships according to the similarity to obtain the candidate ships.

2. The method according to claim 1, wherein The radar for collecting the radar data and the camera for collecting the camera data are both located on the optoelectronic turret; The constructing a unified spatial coordinate system for the multi-source data and determining the camera observation range based on the unified spatial coordinate system includes: Take the installation position of the optoelectronic turret as the origin of the coordinate system, and establish a unified spatial coordinate system; Convert the original polar coordinates of the radar data into rectangular coordinates in the unified spatial coordinate system; According to the position of the camera in the unified spatial coordinate system, the pitch angle, azimuth angle of the optoelectronic turret when collecting the camera data, the field of view angle of the camera, and the maximum detection distance, determine the spatial observation range of the camera; Calculate the coordinate interval of the spatial observation range of the camera in the unified spatial coordinate system, and determine the coordinate interval as the camera observation range.

3. The method according to claim 1, characterized in that The screening the first ships according to the auxiliary data to obtain the second ships includes: For each of the first ships, perform: According to the ship navigation plan data and meteorological data of the first ship, judge whether the included angle between the target course in the ship navigation plan data and the wind direction in the meteorological data is greater than a preset included angle, and whether the navigation speed is less than a preset navigation speed; If the judgment result is yes, then the first ship is a drifting target, and the drifting target is eliminated; If the judgment result is negative, obtain the deviation distance between the position of the first ship and the nearest waterway included in the electronic nautical chart, and determine whether the deviation distance is greater than a preset deviation threshold; When the judgment result is positive, the first ship is an abnormal target, and the abnormal target is removed; When the judgment result is negative, the first ship is determined as the second ship.

4. The method according to claim 1, wherein The determining the target ship from the candidate ships according to the fusion feature includes: Performing radar feature extraction and camera feature extraction on the target ship based on the radar data and the known camera data of the target ship to obtain a target fusion feature; Comparing the target fusion feature with the fusion features of the candidate ships to obtain a fusion feature that matches the target fusion feature; Determining the candidate ship corresponding to the fusion feature as the target ship.

5. A ship positioning device for a radar-assisted camera based on multi-source data fusion, characterized in that, For implementing the method according to any one of claims 1 to 4, includes: A data acquisition module, configured to acquire multi-source data of a ship; wherein, the multi-source data includes radar data, camera data, and auxiliary data; A coordinate processing module, configured to construct a unified spatial coordinate system for the multi-source data, and determine a camera observation range based on the unified spatial coordinate system; A target screening module, configured to perform multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; A feature matching module, configured to perform radar feature extraction and camera feature extraction on the candidate ships, and determine a fusion feature; and determine a target ship from the candidate ships according to the fusion feature.

6. A computing device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1-4 is implemented.

7. A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed in a computer, the computer is made to execute the method according to any one of claims 1-4.

8. A computer program product, characterized in that, Including computer instructions, when the computer instructions are executed by a processor, the steps of the method according to any one of claims 1-4 are implemented.

Images