Multi-source data fusion-based radar-assisted camera ship positioning method and device

Abstract

The invention provides a ship positioning method and device of a radar-assisted camera based on multi-source data fusion, and relates to the technical field of target monitoring and positioning. The method comprises the following steps: acquiring multi-source data of a ship; wherein the multi-source data comprises radar data, camera data and auxiliary data; constructing a unified space coordinate system for the multi-source data, and determining a camera observation range based on the unified space coordinate system; according to the radar data and the auxiliary data, performing multi-stage target screening in a camera observation range to obtain candidate ships; radar feature extraction and camera feature extraction are carried out on the candidate ships, and fusion features are determined; and determining a target ship matched with the radar data from the candidate ships according to the fusion features. According to the scheme, the accuracy and the reliability of matching and positioning the ship target at the marginal defense and the coast defense are improved.

Description

Technical Field

[0001] The present invention relates to the field of target monitoring and positioning technology, in particular to the field of ship track tracking technology, and more particularly to a ship positioning method and device based on a radar-assisted camera with multi-source data fusion. Background Art

[0002] In the fields of border and coastal defense security and marine resource management, it is crucial to accurately grasp the position of the ship in the camera's field of view. Radar and camera, as commonly used monitoring equipment, each has its own advantages and disadvantages. Radar can detect ships at a long distance and obtain key information such as distance, azimuth, speed, etc., but it is insufficient in the details of target recognition; the camera can intuitively present the appearance characteristics of the ship, but is limited by the detection distance and environmental factors, and the positioning accuracy is limited. Relying solely on the data of the two to locate the ship is easily interfered by factors such as sea clutter, weather changes, and target occlusion, resulting in inaccurate positioning and large matching errors, which is difficult to meet the needs of actual applications. Therefore, it is urgent to provide a ship positioning method and device based on multi-source data fusion with radar-assisted cameras. Summary of the invention

[0003] The present invention provides a ship positioning method and device based on a radar-assisted camera with multi-source data fusion, which improves the accuracy and reliability of ship target matching and positioning in border and coastal defense scenarios.

[0004] In a first aspect, the present invention provides a ship positioning method of a radar-assisted camera based on multi-source data fusion, comprising: Acquire multi-source data of the ship; wherein the multi-source data includes radar data, camera data and auxiliary data; Constructing a unified spatial coordinate system for the multi-source data, and determining a camera observation range based on the unified spatial coordinate system; Perform multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; Extracting radar features and camera features from the candidate ship, and determining fusion features; A target ship is determined from the candidate ships according to the fusion features.

[0005] Optionally, the radar for collecting the radar data and the camera for collecting the camera data are both located on a photoelectric turntable; The step of constructing a unified spatial coordinate system for the multi-source data and determining a camera observation range based on the unified spatial coordinate system includes: Taking the installation position of the photoelectric turntable as the origin of the coordinate system, and establishing a unified spatial coordinate system; Converting the original polar coordinates of the radar data into rectangular coordinates in the unified space coordinate system; Determine the spatial observation range of the camera according to the position of the camera in the unified spatial coordinate system, the pitch angle, azimuth angle of the optoelectronic turntable when collecting the camera data, the field of view angle of the camera, and the maximum detection distance; A coordinate interval of the spatial observation range of the camera in the unified spatial coordinate system is calculated, and the coordinate interval is determined as the camera observation range.

[0006] Optionally, performing multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships includes: Determining a ship target included in the radar data located within the observation range of the camera as a first ship target; Screening the first ship according to the auxiliary data to obtain a second ship; Calculating the similarity between the current navigation track of the second ship and the historical ship operation track; The second ship is screened according to the similarity to obtain the candidate ship.

[0007] Optionally, screening the first ship according to the auxiliary data to obtain the second ship includes: For each of the first ships, the following steps are performed: According to the ship navigation plan data and the meteorological data of the first ship, determining whether an angle between a target heading in the ship navigation plan data and a wind direction in the meteorological data of the first ship is greater than a preset angle, and whether a navigation speed is less than a preset navigation speed; If the judgment result is yes, the first ship is a drifting target, and the drifting target is eliminated; If the judgment result is no, obtaining the deviation distance between the position of the first ship and the nearest channel included in the electronic nautical chart, and judging whether the deviation distance is greater than a preset deviation threshold; When the judgment result is yes, the first ship is an abnormal target, and the abnormal target is eliminated; When the determination result is negative, the first ship is determined as the second ship.

[0008] Optionally, the extracting radar features and camera features from the candidate ship and determining fusion features includes: Extract radar features based on the radar data of the candidate ship to obtain a radar feature vector; Comparing the radar characteristic vector of the candidate ship with the radar characteristic vector of the same ship as the candidate ship included in the history library to obtain the radar characteristic distance; Performing camera feature extraction based on the camera data of the candidate ship to obtain a camera feature vector; Comparing the camera feature vector of the candidate ship with the camera feature vector of the same ship as the candidate ship in the history library to obtain a camera feature distance; Acquire environmental parameters of the scene where the candidate ship is located; wherein the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity and atmospheric turbulence intensity; The radar characteristic distance and the camera characteristic distance are fused according to the environmental parameters to obtain the fused feature.

[0009] Optionally, the fusion feature is determined by the following formula: in, d h is a fusion feature distance used to characterize the fusion feature; w i For the i The weight value of each environmental parameter; f i ( X i ) is the i The influence function of environmental parameters; d r is the radar characteristic distance; d c is the camera characteristic distance.

[0010] Optionally, determining a target ship from the candidate ships according to the fusion feature includes: Performing radar feature extraction and camera feature extraction on the target ship based on the radar data of the target ship and known camera data to obtain target fusion features; Comparing the target fusion feature with the fusion feature of the candidate ship to obtain a fusion feature that matches the target fusion feature; The candidate ship corresponding to the fusion feature is determined as the target ship.

[0011] In a second aspect, the present invention further provides a ship positioning device based on a radar-assisted camera with multi-source data fusion, comprising: A data acquisition module, used to acquire multi-source data of the ship; wherein the multi-source data includes radar data, camera data and auxiliary data; A coordinate processing module, used to construct a unified spatial coordinate system for the multi-source data, and determine the camera observation range based on the unified spatial coordinate system; A target screening module, used to perform multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; The feature matching module is used to extract radar features and camera features from the candidate ships and determine fusion features; and to determine a target ship from the candidate ships according to the fusion features.

[0012] In a third aspect, the present invention further provides a computing device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the ship positioning method of the radar-assisted camera based on multi-source data fusion as described above is implemented.

[0013] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, enables the computer to execute any of the above-mentioned ship positioning methods using a radar-assisted camera based on multi-source data fusion.

[0014] In a fifth aspect, an embodiment of the present invention further provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in any first aspect of this specification.

[0015] The present invention provides a ship positioning method and device based on a radar-assisted camera with multi-source data fusion. The method fuses multi-source data, fully utilizes the advantages of each data, and assists in matching and positioning radar targets with ship targets in the camera field of view from multiple angles, effectively overcoming the problems of inaccuracy and large errors that are prone to occur when matching only by relying on radar and camera data, and significantly improving the accuracy and reliability of ship target matching and positioning. Taking into account the complex and changeable border and coastal defense environment, the present invention also designs a real-time tracking and dynamic matching adjustment mechanism, so that the system has good adaptability and robustness, and can stably and continuously complete the ship target matching and positioning tasks under various complex working conditions, providing strong support for border and coastal defense security work. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flow chart of a ship positioning method of a radar-assisted camera based on multi-source data fusion provided by an embodiment of the present invention; Figure 2 is a hardware architecture diagram of a computing device provided by an embodiment of the present invention; Figure 3It is a structural diagram of a ship positioning device based on a radar-assisted camera with multi-source data fusion provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0019] The concept of the present invention is described below. Figure 1 The embodiment of the present invention provides a ship positioning method of a radar-assisted camera based on multi-source data fusion, comprising: Step 100, acquiring multi-source data of the ship; wherein the multi-source data includes radar data, camera data and auxiliary data; Step 102, constructing a unified spatial coordinate system for multi-source data, and determining the camera observation range based on the unified spatial coordinate system; Step 104, performing multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; Step 106, performing radar feature extraction and camera feature extraction on the candidate ship, and determining fusion features; Step 108, determining a target ship from the candidate ships according to the fusion features.

[0020] In the present invention, firstly, the multi-source data such as the ship's radar data, camera data and auxiliary data are unified into a coordinate system to construct a unified spatial coordinate system, and then the camera observation range is determined based on the unified spatial coordinate system, so as to further perform multi-level screening on the current ship based on the camera observation range and multi-source data to obtain candidate ships, and then determine the fusion features by performing radar feature extraction and camera feature extraction on the candidate ships, so as to determine the target ship through the fusion features that fuse various features. In this way, the present invention, by fusing multi-source data and making full use of the advantages of each data, assists in matching and locating the radar target with the ship target in the camera field of view from multiple angles, effectively overcoming the problems of inaccuracy and large errors that are prone to occur when matching only by relying on the radar and camera's own data, and significantly improves the accuracy and reliability of ship target matching and positioning.

[0021] Described below Figure 1 How the various steps are performed.

[0022] First, in step 100, real-time radar data, camera data and other multi-source auxiliary data are collected, and the multi-source data are cleaned and processed. Auxiliary data include AIS, meteorological, electronic charts, satellite remote sensing, ship navigation plans and historical navigation data. The radar can collect information such as the distance, azimuth, pitch angle, speed and radar echo intensity of ship targets in the target area in real time. For radar data, preprocessing includes: using the ordered statistical constant false alarm rate (OS-CFAR) algorithm for clutter suppression, and using the DBSCAN algorithm for target clustering on radar point cloud data. In the present invention, the neighborhood radius ϵ=0.2 nautical miles and the minimum number of points minPts=3 are set based on the distribution characteristics of ships in the border and coastal defense scene. For camera data, preprocessing includes: using the CLAHE algorithm for image enhancement, the block size is 8×8, and the contrast limit threshold is 2.0; extraction is based on the YOLOv5s model, the input resolution is 640×640, and the confidence threshold is 0.7. For auxiliary data, the preprocessing of AIS data includes: checking the consistency between AIS speed and radar speed. If the deviation exceeds 15%, the data retransmission mechanism is triggered. The preprocessing of meteorological data includes: using Kriging interpolation method to convert discrete meteorological station data into continuous distribution in the monitoring area.

[0023] It should be noted that AIS is the Automatic Identification System, which refers to a new navigation aid system used for maritime safety and communication between ships and shores, and between ships. AIS exchanges information with surrounding ships through the VHF band to provide detailed information of the target such as ship name, location, heading, etc.

[0024] In step 102, the radar for collecting radar data and the camera for collecting camera data are both located on an optoelectronic turntable. Construct a unified spatial coordinate system for multi-source data and determine the camera observation range based on the unified spatial coordinate system, including: Take the installation position of the photoelectric turntable as the origin of the coordinate system and establish a unified spatial coordinate system; Convert the original polar coordinates of radar data into rectangular coordinates in a unified space coordinate system; Determine the spatial observation range of the camera based on the position of the camera in the unified spatial coordinate system, the pitch angle and azimuth angle of the optoelectronic turntable when collecting camera data, the field angle of the camera, and the maximum detection distance; The coordinate interval of the spatial observation range of the camera in the unified spatial coordinate system is calculated, and the coordinate interval is determined as the camera observation range.

[0025] Specifically, the installation position O of the photoelectric turntable ( x 0, y 0, z0) is the origin of the coordinate system. The optoelectronic turntable is the physical carrier platform for the radar and camera. Its installation point can be directly measured and has high stability, making it suitable as a spatial reference. The axis system direction is defined as the X-axis pointing to the geographic north direction (calibrated by a high-precision magnetometer or GNSS azimuth, with an error of <0.1°); the Y-axis is perpendicular to the X-axis and points to the east direction; the Z-axis is perpendicular to the XY plane and upward, aligned with the local plumb line (corrected by a tilt sensor, with an error of <0.05°); The original data of the radar detecting the ship target is in polar coordinate form ( r , α , β ), needs to be converted into diameter coordinates in a unified space coordinate system ( X , Y , Z ); the coordinate transformation formula is: in, r is the straight-line distance between the radar and the ship target, α is the elevation angle between the radar and the ship target, β is the azimuth (clockwise angle of the ship target relative to the true north direction), Δ hcurvature is the correction value for the curvature of the earth. It should be noted that when the detection distance r When the distance is >10 nautical miles (about 18.52 kilometers), the effect of the earth's curvature on the height needs to be corrected; since the earth's surface is curved, the actual height of the distant target needs to be subtracted from the virtual height caused by the curvature. At this time, the true height of the ship target is , , Re is the average radius of the Earth; External parameter calibration for radar-camera, including: Chessboard calibration: Place a high-reflectivity chessboard calibration plate at a known position and detect it by the radar and camera at the same time; Feature point matching: extract the coordinates of the corner points of the calibration plate in the radar point cloud (radar data) and the camera image (camera data); Least squares optimization: Calculate the external parameter matrix Text∈ R 4×4 (including rotation matrix R and translation vectors t ), so that the reprojection error is minimized; The final result is: angle error: <0.05° (guaranteed by high-precision photoelectric turntable), translation error: <5cm (calibrated by laser rangefinder); Finally, reflective targets are deployed at known coordinates of static targets, and their positions are measured by radar and camera respectively to verify the unified spatial coordinate system; dynamic targets are flown by drones according to preset trajectories to compare the consistency of radar, camera and GNSS positioning data. The acceptance criteria are static target positioning error <0.5 meters; dynamic target tracking error <1.0 meters (speed ≤30 knots).

[0026] Specifically, determining the camera observation range includes: establishing a four-sided pyramidal viewing cone model of the camera observation space according to the azimuth and pitch angles of the optoelectronic turntable, the field of view angle of the camera, and the maximum detection distance; then calculating the coordinate interval of the viewing cone in the unified space coordinate system through trigonometric functions, and making corrections for the earth's curvature and atmospheric refraction; then using the projection matrix method to determine whether the radar target is within the viewing cone, and ensuring the accuracy of screening through static and dynamic verification.

[0027] In the present invention, in order to ensure the geometric consistency of multi-source data, the original coordinate systems of the multi-source data are unified into a unified three-dimensional space framework.

[0028] With respect to step 104, multi-level target screening is performed within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships, including: A ship target included in the radar data located within the observation range of the camera is determined as a first ship target; Screening the first ship according to the auxiliary data to obtain a second ship; Calculating the similarity between the current navigation track of the second ship and the historical ship operation track; The second ship is screened according to the similarity to obtain a candidate ship.

[0029] In a specific embodiment, screening the first ship according to the auxiliary data to obtain the second ship includes: For each first ship, execute: According to the auxiliary data including the ship navigation plan data and the meteorological data of the first ship, it is determined whether the angle between the target heading in the ship navigation plan data of the first ship and the wind direction in the meteorological data is greater than a preset angle (for example, 90°), and whether the navigation speed is less than a preset navigation speed (for example, 5 knots (i.e., 5 nautical miles per hour)); If the judgment result is yes, the first ship is a drifting target, and the drifting target is eliminated; If the judgment result is no, obtaining the deviation distance between the position of the first ship and the nearest channel included in the electronic nautical chart in the auxiliary data, and judging whether the deviation distance is greater than a preset deviation threshold (for example, 1.5 nautical miles); When the judgment result is yes, the first ship is an abnormal target, and the abnormal target is eliminated; When the determination result is negative, the first ship is determined as the second ship.

[0030] In a specific implementation, calculating the similarity between the current navigation track of the second ship and the historical ship operation track; screening the second ship according to the similarity to obtain the candidate ship includes: For each second vessel, perform: The similarity between the current navigation track of the second ship and the historical ship operation track is calculated. The similarity is determined by the following formula: in, D for similarity; P c is the current navigation track, P c [ i ] is the current navigation track i location points; P h For historical ship operation tracks, P h [ j ] is the previous one in the historical ship operation track j location points; When the minimum similarity is greater than a preset threshold (for example, 200), the current navigation track of the second ship is determined to be an abnormal track, and the second ship is removed; When the minimum similarity is not greater than a preset threshold (eg, 200), the second ship is determined to be a candidate ship. It should be noted that the minimum similarity is the minimum value of the similarities between the current navigation track of the second ship and all historical ship operation tracks.

[0031] In the present invention, firstly, based on the spatial position relationship, the scope of potential matching targets is quickly narrowed down, and the first ship target is determined. Then, a secondary screening is performed based on the heading data and meteorological data in the auxiliary data, and ships that do not conform to conventional navigation logic and geographical logic are further excluded, thereby narrowing the matching scope. Finally, based on the current trajectories of the remaining ships and the historical ship operation trajectories in the history library, another screening is performed to further exclude ships with abnormal trajectories, obtain candidate ships, and improve the accuracy of matching.

[0032] In the present invention, preliminary potential matching targets are screened out according to the coordinates of the ship targets detected by the radar and the coordinate interval of the camera observation range, that is, the first ship targets that fall within the camera observation range, to form a preliminary set; and then the targets are further screened through other multi-source data to narrow the matching range and accurately locate the target ship.

[0033] In step 106, radar feature extraction and camera feature extraction are performed on the candidate ship, and fusion features are determined, including: Extract radar features according to radar data of candidate ships to obtain radar feature vectors; The radar characteristic vector of the candidate ship is compared with the radar characteristic vector of the same ship as the candidate ship in the history library to obtain the radar characteristic distance; Perform camera feature extraction based on the camera data of the candidate ship to obtain a camera feature vector; Comparing the camera feature vector of the candidate ship with the camera feature vector of the same ship as the candidate ship in the history library to obtain the camera feature distance; Obtaining environmental parameters of the scene where the candidate ship is located; wherein the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity and atmospheric turbulence intensity; The radar feature distance and the camera feature distance are fused according to the environmental parameters to obtain the fused feature.

[0034] It should be noted that the history library includes multi-source data of any ship stored in history. The radar feature distance can be the Euclidean distance, and the camera feature distance can be the Bhattacharyya distance. Preferably, the camera feature extraction adopts Hu moment calculation, and 7 invariant moments are used for rotation invariance matching; the color histogram is spliced ​​into a 48-dimensional feature vector after normalization through the RGB three channels, and then the Bhattacharyya distance between the two is calculated to measure the similarity of color distribution.

[0035] Specifically, the radar feature vector is ;in, They are velocity, acceleration, heading angle, and radar cross-section.

[0036] In the present invention, since radar data and camera data are heterogeneous data, it is difficult to directly fuse the radar feature vector and the camera feature vector. Therefore, the present invention first determines the standard ship that is the same or similar to the candidate ship from the historical library, and then compares the extracted radar feature vector with the radar feature vector of the standard ship to obtain the radar feature distance, and then compares the extracted camera feature vector with the camera feature vector of the standard ship to obtain the camera feature distance, and then fuses based on the radar feature distance and the camera feature distance. In this way, by comparing the feature information of the current ship and the same or similar ships in history, the fusion of heterogeneous data is converted into the fusion of similarity under different feature information, thereby effectively integrating multi-source data information and further improving the precise positioning of ship targets.

[0037] In a preferred embodiment, the fusion feature is determined by the following formula: in,d h is the fusion feature distance used to characterize the fusion feature; w i For the i The weight value of each environmental parameter; f i ( X i ) is the i The influence function of environmental parameters; d r is the radar characteristic distance; d c is the camera characteristic distance. , i =1,2,3,4,5.

[0038] Specifically, i =1, f 1( X 1)= f snr ( SNR ), represents the influence function of the signal-to-noise ratio; in, SNR is the signal-to-noise ratio; i =2, f 2( X 2)= f vis ( VIS ), represents the influence function of visibility; in, VIS For visibility; i =3, f 3( X 3)= f pre ( PRE ), represents the influence function of precipitation intensity; in, PRE is the precipitation intensity; i =4, f 4( X 4)= f lig ( LIG ), represents the influence function of light intensity; in, LIG is the light intensity; i =5, f 5( X 5)= f tur ( TUR ), represents the influence function of atmospheric turbulence intensity; in, TUR is the atmospheric turbulence intensity, which is a normalized value measured by relevant instruments and a dimensionless unit.

[0039] In the present invention, since environmental parameters such as the signal-to-noise ratio, visibility, precipitation intensity, light intensity and atmospheric turbulence intensity in the scene will change in real time, and their dynamic changes will have different degrees of impact on the performance of the radar and camera, it is necessary to take the impact of the environmental parameters into account in the fusion process, and further improve the precise positioning of the target ship based on the current scene information.

[0040] In the present invention, when the signal-to-noise ratio is ≥30, both the radar and the camera can work stably and accurately, the image obtained by the camera is clear, and the radar detection data is accurate. As the signal-to-noise ratio gradually decreases, the performance degradation trend of the camera intensifies. In order to ensure the accuracy of ship positioning, it is necessary to gradually reduce the proportion of the camera feature distance in the fusion feature. When the signal-to-noise ratio is ≤15, the environmental interference is serious, the camera imaging quality is seriously damaged, the acquired image is blurred, the ship features are difficult to accurately extract, and the reliability of the camera features is extremely low. It is necessary to rely mainly on radar for ship positioning, so as to ensure that the system can operate effectively in harsh environments.

[0041] In the present invention, the lower the visibility, the greater the impact on camera performance; but when visibility is between 0 and 2000 meters, the impact value of visibility is inversely proportional to visibility, and camera performance gradually decreases. For precipitation intensity, when precipitation intensity ≤5mm / h, the impact on radar and camera performance is small, so the function value is 0; but as precipitation intensity gradually increases, the impact value also increases linearly, and has a greater impact on camera performance. For light intensity, too low or too high light intensity will seriously affect camera imaging, but between 500 and 5250 lux, the impact value will decrease with increasing light intensity; but between 5250 and 10000 lux, the impact value increases with increasing light intensity. For atmospheric turbulence intensity, the impact value increases with increasing atmospheric turbulence intensity, and the reliability of the camera decreases. At this time, the radar feature weight needs to be increased to ensure the accuracy of ship positioning.

[0042] In a specific embodiment, for the candidate ship-vessel 1, step 106 includes: S1, extract radar features based on the radar data of ship 1 to obtain the radar feature vector X=[9.26,0.3,45,-19.57]; S2, find the feature vector of ships similar to ship 1 from the historical database Y =[9.0,0.25,40,-20], calculate the Euclidean distance between the two and get the radar characteristic distance d r , S3, extract camera features based on the camera data of ship 1, calculate the Hu moment of the image of ship 1, and obtain the values ​​of 7 invariant moments, which are ; Then calculate the color histogram, divide the RGB channels into 16 bins each, and splice them into a 48-dimensional feature vector after normalization P ; S4, find the color histogram feature vector of the ship corresponding to ship 1 from the historical database Q , calculate the Bhattacharyya distance between the two and get the camera feature distance d c , S5, assuming that at a certain moment, the environmental parameters of the scene where ship 1 is located are: signal-to-noise ratio is 18dB, visibility is 1200m, precipitation intensity is 8mm / h, light intensity is 6000 lux, and atmospheric turbulence intensity is 0.4; according to expert experience, set w 1=0.3, w 2=0.25, w 3=0.2, w 4=0.15, w 5=0.1; then f snr (18)=0.8, f vis (1200)=0.4, f pre (8)=0.1, f lig (6000)≈0.16, f tur (0.4)≈0.33; Fusion Features .

[0043] It should be noted that if the target signal loss time of the candidate ship exceeds the preset time threshold, or the heading sudden change angle of the candidate ship is greater than the preset angle threshold, rematching is triggered, that is, returning to step 100, so as to ensure accurate tracking and matching of the target ship.

[0044] In step 108, a target ship is determined from candidate ships according to the fusion features, including: Based on the radar data of the target ship and the known camera data, radar features and camera features of the target ship are extracted to obtain target fusion features; Compare the target fusion feature with the fusion feature of the candidate ship to obtain the fusion feature that matches the target fusion feature; The candidate ship corresponding to the fusion feature is determined as the target ship.

[0045] It should be noted that the known camera data can be collected in advance.

[0046] In the present invention, the method and system for locating ships in the camera field of view with radar targets assisted by multi-source data can effectively achieve accurate matching and positioning of ship targets, providing strong support for border and coastal defense monitoring. In a complex border and coastal defense environment, the system overcomes the limitations of a single sensor through multi-source data fusion and intelligent algorithms, improves the accuracy and reliability of ship positioning, and has important practical application value.

[0047] like Figure 2 , Figure 3 As shown, an embodiment of the present invention provides a ship positioning device based on a radar-assisted camera with multi-source data fusion. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. From the hardware level, Figure 2 As shown in FIG. 1 , a hardware architecture diagram of a computing device where a ship positioning device based on a radar-assisted camera with multi-source data fusion is provided in an embodiment of the present invention is located. Figure 2 In addition to the processor, memory, network interface, and non-volatile memory shown in the figure, the computing device in which the device is located in the embodiment may also generally include other hardware, such as a forwarding chip responsible for processing messages, etc. Taking software implementation as an example, Figure 3 As shown, as a device in a logical sense, the CPU of the computing device in which it is located reads the corresponding computer program in the non-volatile memory into the memory and runs it. This embodiment provides a ship positioning device based on a radar-assisted camera with multi-source data fusion, including: The data acquisition module 300 is used to obtain multi-source data of the ship; wherein the multi-source data includes radar data, camera data and auxiliary data; A coordinate processing module 302 is used to construct a unified spatial coordinate system for multi-source data and determine the camera observation range based on the unified spatial coordinate system; The target screening module 304 is used to perform multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; The feature matching module 306 is used to extract radar features and camera features from the candidate ships and determine fusion features; and determine the target ship from the candidate ships according to the fusion features.

[0048] In some specific embodiments, the data acquisition module 300 can be used to execute the above step 100, the coordinate processing module 302 can be used to execute the above step 102, the target screening module 304 can be used to execute the above step 104, and the feature matching module 306 can be used to execute the above steps 106 and 108.

[0049] In some specific embodiments, the radar that collects radar data and the camera that collects camera data are both located on an optoelectronic turntable; The coordinate processing module 302 is also used to perform the following operations: Take the installation position of the photoelectric turntable as the origin of the coordinate system and establish a unified spatial coordinate system; Convert the original polar coordinates of radar data into rectangular coordinates in a unified space coordinate system; Determine the spatial observation range of the camera based on the position of the camera in the unified spatial coordinate system, the pitch angle and azimuth angle of the optoelectronic turntable when collecting camera data, the field angle of the camera, and the maximum detection distance; The coordinate interval of the spatial observation range of the camera in the unified spatial coordinate system is calculated, and the coordinate interval is determined as the camera observation range.

[0050] In some specific implementations, the target screening module 304 is further configured to perform the following operations: A ship target included in the radar data located within the observation range of the camera is determined as a first ship target; Screening the first ship according to the auxiliary data to obtain a second ship; Calculating the similarity between the current navigation track of the second ship and the historical ship operation track; The second ship is screened according to the similarity to obtain a candidate ship.

[0051] In some specific implementations, the feature matching module 306 is further configured to perform the following operations: Extract radar features according to radar data of candidate ships to obtain radar feature vectors; The radar characteristic vector of the candidate ship is compared with the radar characteristic vector of the same ship as the candidate ship in the history library to obtain the radar characteristic distance; Perform camera feature extraction based on the camera data of the candidate ship to obtain a camera feature vector; Comparing the camera feature vector of the candidate ship with the camera feature vector of the same ship as the candidate ship in the history library to obtain the camera feature distance; Obtaining environmental parameters of the scene where the candidate ship is located; wherein the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity and atmospheric turbulence intensity; The radar characteristic distance and the camera characteristic distance are fused according to the environmental parameters to obtain the fusion feature; the fusion feature is determined by the following formula: in, d h is the fusion feature distance used to characterize the fusion feature; w i For the i The weight value of each environmental parameter; f i ( X i ) is the i The influence function of environmental parameters; d r is the radar characteristic distance; d c is the camera feature distance.

[0052] In some specific implementations, the feature matching module 306 is further configured to perform the following operations: Based on the radar data of the target ship and the known camera data, radar features and camera features of the target ship are extracted to obtain target fusion features; Compare the target fusion feature with the fusion feature of the candidate ship to obtain the fusion feature that matches the target fusion feature; The candidate ship corresponding to the fusion feature is determined as the target ship.

[0053] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on a ship positioning device based on a radar-assisted camera with multi-source data fusion. In other embodiments of the present invention, a ship positioning device based on a radar-assisted camera with multi-source data fusion may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0054] The information interaction, execution process and other contents between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention. For the specific contents, please refer to the description in the embodiment of the method of the present invention, and no further description is given here.

[0055] An embodiment of the present invention further provides a computing device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, a ship positioning method of a radar-assisted camera based on multi-source data fusion in any embodiment of the present invention is implemented.

[0056] An embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes a ship positioning method based on a radar-assisted camera with multi-source data fusion in any embodiment of the present invention.

[0057] An embodiment of the present application also provides a computer program product, which includes a computer program. A processor of a computer device reads the computer program from a computer-readable storage medium, and the processor executes the computer program, so that the computer device executes a ship positioning method based on a radar-assisted camera with multi-source data fusion as described in any of the above embodiments.

[0058] Specifically, a system or device equipped with a storage medium can be provided, on which software program code that implements the functions of any of the above-mentioned embodiments is stored, and a computer (or CPU or MPU) of the system or device can be enabled to read and execute the program code stored in the storage medium.

[0059] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0060] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0061] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0062] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion module connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or expansion module is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0063] It should be noted that, in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprise a ..." do not exclude the presence of other identical factors in the process, method, article or device including the elements.

[0064] A person of ordinary skill in the art can understand that all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0065] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ship positioning method based on radar-assisted camera with multi-source data fusion, characterized in that: include: Acquire multi-source data of the ship; wherein the multi-source data includes radar data, camera data, and auxiliary data including ship navigation plan data, meteorological data, electronic nautical charts, and historical ship operation tracks; Constructing a unified spatial coordinate system for the multi-source data, and determining a camera observation range based on the unified spatial coordinate system; Perform multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; Extracting radar features and camera features from the candidate ship, and determining fusion features; determining a target ship from the candidate ships according to the fusion feature; The candidate ship is obtained by the following method: Determining a ship target included in the radar data located within the observation range of the camera as a first ship target; Screening the first ship according to the auxiliary data to obtain a second ship; Calculating the similarity between the current navigation track of the second ship and the historical ship operation track; The second ship is screened according to the similarity to obtain the candidate ship.

2. The method according to claim 1, characterized in that The radar for collecting the radar data and the camera for collecting the camera data are both located on the photoelectric turntable; The step of constructing a unified spatial coordinate system for the multi-source data and determining a camera observation range based on the unified spatial coordinate system includes: Taking the installation position of the photoelectric turntable as the origin of the coordinate system, and establishing a unified spatial coordinate system; Converting the original polar coordinates of the radar data into rectangular coordinates in the unified space coordinate system; Determine the spatial observation range of the camera according to the position of the camera in the unified spatial coordinate system, the pitch angle, azimuth angle of the optoelectronic turntable when collecting the camera data, the field of view angle of the camera, and the maximum detection distance; A coordinate interval of the spatial observation range of the camera in the unified spatial coordinate system is calculated, and the coordinate interval is determined as the camera observation range.

3. The method according to claim 1, characterized in that The step of screening the first ship according to the auxiliary data to obtain the second ship includes: For each of the first ships, execute: According to the ship navigation plan data and the meteorological data of the first ship, determining whether an angle between a target heading in the ship navigation plan data and a wind direction in the meteorological data of the first ship is greater than a preset angle, and whether a navigation speed is less than a preset navigation speed; If the judgment result is yes, the first ship is a drifting target, and the drifting target is eliminated; If the judgment result is no, obtaining the deviation distance between the position of the first ship and the nearest channel included in the electronic nautical chart, and judging whether the deviation distance is greater than a preset deviation threshold; When the judgment result is yes, the first ship is an abnormal target, and the abnormal target is eliminated; When the judgment result is negative, the first ship is determined as the second ship.

4. The method according to any one of claims 1 to 3, characterized in that: The extracting radar features and camera features of the candidate ship and determining fusion features includes: Extract radar features based on the radar data of the candidate ship to obtain a radar feature vector; Comparing the radar characteristic vector of the candidate ship with the radar characteristic vector of the same ship as the candidate ship included in the history library to obtain the radar characteristic distance; Performing camera feature extraction based on the camera data of the candidate ship to obtain a camera feature vector; Comparing the camera feature vector of the candidate ship with the camera feature vector of the same ship as the candidate ship included in the history library to obtain a camera feature distance; Acquire environmental parameters of the scene where the candidate ship is located; wherein the environmental parameters include signal-to-noise ratio, visibility, precipitation intensity, light intensity and atmospheric turbulence intensity; The radar characteristic distance and the camera characteristic distance are fused according to the environmental parameters to obtain the fused feature.

5. The method according to claim 4, characterized in that The fusion feature is determined by the following formula: in, d h is a fusion feature distance used to characterize the fusion feature; w i For the i The weight value of each environmental parameter; f i ( X i ) is the i The influence function of environmental parameters; d r is the radar characteristic distance; d c is the camera characteristic distance.

6. The method according to claim 4, characterized in that The determining a target ship from the candidate ships according to the fusion feature comprises: Performing radar feature extraction and camera feature extraction on the target ship based on the radar data of the target ship and known camera data to obtain target fusion features; Comparing the target fusion feature with the fusion feature of the candidate ship to obtain a fusion feature that matches the target fusion feature; The candidate ship corresponding to the fusion feature is determined as the target ship.

7. A ship positioning device based on radar-assisted camera with multi-source data fusion, characterized in that: include: A data acquisition module, used to acquire multi-source data of the ship; wherein the multi-source data includes radar data, camera data and auxiliary data; A coordinate processing module, used to construct a unified spatial coordinate system for the multi-source data, and determine the camera observation range based on the unified spatial coordinate system; A target screening module, used to perform multi-level target screening within the camera observation range according to the radar data and the auxiliary data to obtain candidate ships; The feature matching module is used to extract radar features and camera features from the candidate ships and determine fusion features; and to determine a target ship from the candidate ships according to the fusion features.

8. A computing device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.

Images