Overwater and underwater target recognition system based on radar and sonar multi-source information fusion
Through the radar sonar multi-source information fusion system, combining photoelectric and underwater images, a multi-modal classification algorithm is used to distinguish overwater and underwater targets, solving the identification error and positioning error problems of the existing system in multi-target scenarios, achieving high Accurate target recognition.
Patent Information
- Application Number
- CN202411071585.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-11-19
AI Technical Summary
Existing water and underwater target recognition systems are prone to errors in multi-target scenarios, making it difficult to distinguish between overwater and underwater targets. In addition, traditional signal processing methods have positioning errors and false alarm problems, and lack effective fusion of multi-sensor information.
The system based on radar sonar multi-source information fusion is adopted to obtain target tracks through detection and tracking algorithms, sort them with track fusion algorithms, and use multimodal classification algorithms to combine photoelectric images or underwater images for identification to realize water on water. distinguishing and identifying targets.
It improves the accuracy of target recognition, reduces the positioning error of radar and sonar targets, reduces false alarms, and realizes the distinction between targets and submersible targets and detailed type recognition.
Smart Images

Figure CN118981012A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the intersection of multi-sensor target recognition technology and computer vision, and in particular to an above-water and underwater target recognition system based on radar and sonar multi-source information fusion. Background Art
[0002] Radar and sonar are commonly used sensors for detecting targets above and below the water. They are widely used in civil and military monitoring systems. The two sensors have a mature technical system in their respective fields. Shore-based or marine radars are usually combined with PTZ optoelectronic equipment to monitor ships on the sea surface and low-altitude targets. Although the traditional radar target detection method has a high recall rate, it is sensitive to sea clutter and is prone to false alarms, thus affecting the accuracy and continuity of the track.
[0003] Although many monitoring systems on the market use radar optoelectronic linkage, they all use visible light alone to identify targets, which is prone to errors in multi-target scenarios. Passive listening sonar and active imaging sonar have different detection ranges. The former must be actively generated by the target, and it is difficult to distinguish between above-water and underwater target signals, and the positioning error is large. The latter can only detect underwater targets, and most of them use traditional signal processing methods to identify targets, which can only process known types of signals. Due to visibility issues, underwater cameras have limited monitoring ranges. Currently, there are few systems that combine passive listening sonar, active imaging sonar or underwater cameras. Summary of the invention
[0004] In view of the technical problems existing in the prior art, the purpose of the present invention is to provide a water and underwater target recognition system based on radar and sonar multi-source information fusion. The present invention first obtains the radar and sonar target tracks through a detection and tracking algorithm, and then uses a track fusion algorithm to sort the targets, thereby distinguishing between water and underwater targets. Finally, in combination with optoelectronic images or underwater images, a multimodal classification algorithm is used to identify the water and underwater targets respectively.
[0005] The technical solution of the present invention is:
[0006] A system for identifying underwater targets on the water based on radar and sonar multi-source information fusion, characterized in that it comprises a radar information acquisition device, a pan-tilt optoelectronic device, a sonar information acquisition device, an underwater image acquisition device, a radar detection processor, a sonar detection processor, a fusion sorting processor, a radar identification processor, and a sonar identification processor; the radar information acquisition device and the pan-tilt optoelectronic device are connected to the radar detection processor, the sonar information acquisition device and the underwater image acquisition device are connected to the sonar detection processor, the radar detection processor is connected to the fusion sorting processor and the radar identification processor, the sonar detection processor is connected to the fusion sorting processor and the sonar identification processor, and the fusion sorting processor is connected to the radar identification processor and the sonar identification processor;
[0007] The radar information acquisition device is used to collect radar signal data from shore-based or marine radars, and convert it into network port or serial port data to input into the radar detection processor for processing to generate continuous frame radar echo images;
[0008] The pan / tilt optoelectronic device is used to aim at the radar target, obtain the radar optoelectronic image of the radar target and input it into the radar detection processor for processing to obtain the target position and perform real-time tracking, and output the optoelectronic target trajectory and the radar target track;
[0009] The sonar information acquisition device is used to collect sonar signal data and convert it into network port or serial port data, and input it into the sonar detection processor for processing to generate continuous frame sonar waveform images;
[0010] The underwater image acquisition device is used to collect underwater photoelectric or sonar images and input them into the sonar detection processor for processing to obtain the target position and track it in real time, and output the underwater target trajectory and sonar target track;
[0011] The fusion sorting processor is used to receive the radar target track and sonar target track in real time, and obtain the AIS target track through the AIS receiving device; then the received three track data are time-space aligned, and the least square method is used for interpolation to obtain data with the same time interval; then the historical position in each sonar target track is calculated by the nearest neighbor method with the correlation rate of the corresponding position in the radar target track and the AIS target track to form a track correlation matrix, find the optimal allocation for the track correlation matrix, and output the track correlation result; then the track correlation result is screened and distributed, and the water target is sent to the radar recognition processor, and the underwater target is sent to the sonar recognition processor;
[0012] The radar identification processor is used to control the PTZ optoelectronic device to aim at the radar target according to the optoelectronic target trajectory and the longitude and latitude of the radar target;
[0013] The sonar recognition processor is used to call the underwater image acquisition device at the corresponding position to collect underwater visible light or sonar images according to the underwater target trajectory and the sonar target longitude and latitude.
[0014] Furthermore, the radar detection processor includes a radar target detection module and an optoelectronic target detection module. The radar target detection module is connected to the radar information acquisition device, and the optoelectronic target detection module is connected to the gimbal optoelectronic device; the radar target detection module adopts a YOLO-based radar time-series image target detection algorithm to obtain the target frame position in the radar image, converts the radar position into longitude and latitude coordinates, and then performs real-time tracking to obtain the radar target track, which is input into the fusion sorting processor; the optoelectronic target detection module adopts an optoelectronic target detection method based on deep learning to obtain the target position from the radar optoelectronic image of the radar target and performs real-time tracking, and outputs the optoelectronic target trajectory.
[0015] Furthermore, the sonar detection processor includes a sonar target detection module and an underwater target detection module. The sonar target detection module is connected to the sonar information acquisition device, and the underwater target detection module is connected to the underwater image acquisition device. The sonar target detection module adopts a sonar target ranging lateral method to obtain the sonar target direction and distance information, converts it into longitude and latitude coordinates in combination with the sonar position, and then performs real-time tracking to obtain the sonar target track, which is input into the fusion sorting processor. The underwater target detection module adopts an underwater target detection method based on deep learning to obtain the target position from the underwater optoelectronic or sonar image and tracks it in real time, and outputs the underwater target trajectory.
[0016] Furthermore, the fusion and sorting processor includes a multi-source information fusion processing module and an above-water and underwater target sorting module. The radar target detection module and the sonar target detection module are connected to the multi-source information fusion processing module, and the multi-source information fusion processing module is connected to the above-water and underwater target sorting module; the multi-source information fusion processing module is used to generate track association results according to the radar target track, sonar target track, and AIS target track; the above-water and underwater target sorting module is used to screen and distribute the track association results, sort the track data in combination with the track history information and the association rate, send the above-water targets to the radar recognition processor, and send the underwater targets to the sonar recognition processor.
[0017] Furthermore, the above-water and underwater target sorting module associates the radar target tracks, sonar target tracks, and AIS target tracks in pairs according to steps ① to ④: ① Each sonar target track is judged with the radar target track and the AIS target track. If the correlation rate is greater than the set high correlation threshold, the corresponding target is judged to be an above-water target; ② Each sonar target track is judged with the radar target track and the AIS target track. If the correlation rate is less than the set low correlation threshold or is not associated, the corresponding target is an underwater target; ③ Each sonar target track is judged with the radar target track and the AIS target track. If the correlation rate is between the high correlation threshold and the low correlation threshold, the corresponding target is sent to the radar recognition processor and the sonar recognition processor at the same time; ④ The remaining radar and AIS target tracks that are not associated with the sonar target track are judged, and the corresponding target is judged to be an above-water target.
[0018] Furthermore, the radar identification processor includes a radar optoelectronic linkage module and an above-water target identification module. The optoelectronic target detection module is connected to the radar optoelectronic linkage module, and the radar optoelectronic linkage module and the above-water and underwater target sorting module are connected to the above-water target identification module; the radar optoelectronic linkage module is used to calculate the horizontal azimuth angle Pan, the vertical pitch angle Tilt, and the lens zoom multiple Zoom of the gimbal according to the longitude and latitude of the radar target, control the gimbal optoelectronic device to aim at the radar target, and obtain the radar optoelectronic multimodal image of the same target; the above-water target identification module is used to adopt a radar optoelectronic multimodal classification method based on deep learning, and jointly identify the above-water targets according to the input radar optoelectronic multimodal image, and output the type of the above-water target; the radar optoelectronic multimodal image of the same target includes the radar echo image and the radar optoelectronic image of the same target.
[0019] Furthermore, the sonar recognition processor includes a sonar target matching module and an underwater target recognition module. The underwater target detection module is connected to the sonar target matching module, and the sonar target matching module and the above-water and underwater target sorting module are connected to the underwater target recognition module; the sonar target matching module is used to call the underwater image acquisition device at the corresponding position according to the longitude and latitude of the sonar target, associate and match the target in the underwater image with the sonar target, and obtain the sonar multimodal image of the same target; the underwater target recognition module is used to adopt a sonar multimodal classification method based on deep learning, jointly identify the underwater target according to the input sonar multimodal image, and output the underwater target type; the sonar multimodal image of the same target includes the sonar image of the same target and the image of the underwater image acquisition device.
[0020] Furthermore, the underwater target detection module adopts the YOLO-based underwater time-series image target detection algorithm to obtain the target frame position in the underwater target image, converts the sonar target longitude and latitude into image coordinates according to the position and parameters of the underwater image acquisition device, adopts the Hungarian algorithm to associate and match the corresponding targets in the sonar underwater image, and respectively captures two local images of the same target from the sonar underwater image, namely, the sonar waveform local image and the underwater imaging local image of the target.
[0021] Furthermore, the optimal allocation of the track association matrix is found through the Hungarian algorithm, and the track association result is output.
[0022] The present invention can be deployed on a shore-based or ship-borne platform to achieve near-shore, offshore, underwater and underwater three-dimensional monitoring. Combining the advantages of multiple sensors such as radar, sonar, optoelectronics, and underwater imaging equipment, the shortcomings are mutually compensated, and the target is quickly and accurately located through a target detection method based on deep learning (including various image intelligent detection methods, the most common model is the yolo model). By associating and fusing sonar, radar, and AIS target tracks, the radar and sonar target positioning errors are reduced, and false alarms are reduced. At the same time, after the targets are distinguished into above-water and underwater targets, more targeted classification processing can be used respectively. For above-water or underwater targets, it is difficult to identify the target by relying on single-mode radar or sonar data. The use of multimodal classification methods can greatly improve the accuracy of target recognition and achieve target verification and fine type recognition.
[0023] An embodiment of the present invention provides an underwater target recognition system based on radar sonar multi-source information fusion, including a radar information acquisition device, a pan-tilt optoelectronic device, a sonar information acquisition device, an underwater image acquisition device, a radar detection processor, a sonar detection processor, a fusion sorting processor, a radar recognition processor, and a sonar recognition processor; the radar information acquisition device and the pan-tilt optoelectronic device are connected to the radar detection processor, the sonar information acquisition device and the underwater image acquisition device are connected to the sonar detection processor, the radar detection processor is connected to the fusion sorting processor and the radar recognition processor, the sonar detection processor is connected to the fusion sorting processor and the sonar recognition processor, and the fusion sorting processor is connected to the radar recognition processor and the sonar recognition processor;
[0024] The radar detection processor includes a radar target detection module and an optoelectronic target detection module, wherein the radar target detection module is connected to the radar information acquisition device, and the optoelectronic target detection module is connected to the PTZ optoelectronic device;
[0025] The sonar detection processor includes a sonar target detection module and an underwater target detection module, wherein the sonar target detection module is connected to the sonar information acquisition device, and the underwater target detection module is connected to the underwater image acquisition device;
[0026] The fusion sorting processor includes a multi-source information fusion processing module and an above-water and underwater target sorting module. The radar target detection module and the sonar target detection module are connected to the multi-source information fusion processing module, and the multi-source information fusion processing module is connected to the above-water and underwater target sorting module.
[0027] The radar recognition processor includes a radar optoelectronic linkage module and an above-water target recognition module. The optoelectronic target detection module is connected to the radar optoelectronic linkage module. The radar optoelectronic linkage module and the above-water and underwater target sorting module are connected to the above-water target recognition module.
[0028] The sonar recognition processor comprises a sonar target matching module and an underwater target recognition module. The underwater target detection module is connected to the sonar target matching module. The sonar target matching module and the above-water and underwater target sorting module are connected to the underwater target recognition module.
[0029] Preferably, the radar information acquisition device is a radar signal acquisition card that outputs a radar echo image.
[0030] Preferably, the pan-tilt optoelectronic device is a network camera or an infrared thermal imager, which outputs visible light or infrared images.
[0031] Preferably, the sonar information acquisition device is a sonar signal acquisition card that outputs sonar signal data.
[0032] Preferably, the underwater image acquisition device is an underwater camera or an imaging sonar, which outputs underwater photoelectric or sonar images.
[0033] Preferably, the radar detection processor is a computer or an embedded motherboard, which is equipped with a radar target detection module and a photoelectric target detection module.
[0034] Preferably, the sonar detection processor is a computer or an embedded mainboard, and is equipped with a sonar target detection module and an underwater target detection module.
[0035] Preferably, the fusion sorting processor is a computer or an embedded mainboard, which deploys a multi-source information fusion processing module and an above-water and underwater target sorting module.
[0036] Preferably, the radar identification processor is a computer or an embedded mainboard, which is equipped with a radar optoelectronic linkage module and an aquatic target identification module.
[0037] Preferably, the sonar recognition processor is a computer or an embedded mainboard, and is equipped with a sonar target matching module and an underwater target recognition module.
[0038] Preferably, the radar target detection module adopts a radar target detection method based on deep learning, inputs a radar echo time-series image, obtains the target position in the radar image, converts it into longitude and latitude for real-time tracking, and outputs the radar target track.
[0039] Preferably, the photoelectric target detection module adopts a photoelectric target detection method based on deep learning, inputs a visible light or infrared image, obtains the target position in the photoelectric image, tracks the target position in the image in real time, and outputs the photoelectric target trajectory.
[0040] Preferably, the sonar target detection module adopts a traditional sonar target ranging lateral method, inputs sonar signal data, obtains sonar target direction and distance information, converts it into longitude and latitude for real-time tracking, and outputs the sonar target track.
[0041] Preferably, the underwater target detection module adopts an underwater target detection method based on deep learning, inputs an underwater photoelectric or sonar image, obtains the target position in the underwater image, tracks the target position in the image in real time, and outputs the underwater target trajectory.
[0042] Preferably, the multi-source information fusion processing module performs spatiotemporal registration on the radar track, sonar track, and AIS track data, uses the nearest neighbor method to perform track fusion, and outputs track association results, namely, three association matrices of radar and sonar, radar and AIS, and sonar and AIS, including the association rate between every two tracks.
[0043] Preferably, the above-water and underwater target sorting module screens and distributes the track association results, determines the above-water and underwater targets based on the association rate in the track association matrix, sends the track data with a high probability of being an above-water target to the above-water target identification module for classification, and sends the track data with a high probability of being an underwater target to the underwater target identification module for classification.
[0044] Preferably, the radar optoelectronic linkage module calculates the pan / tilt horizontal azimuth angle Pan, vertical pitch angle Tilt, and lens zoom multiple Zoom, i.e., PTZ parameters, according to the longitude and latitude of the radar target, controls the pan / tilt optoelectronic device to aim at the radar target, and obtains radar and optoelectronic images of the same target, i.e., radar optoelectronic multimodal images.
[0045] Preferably, the water target recognition module adopts a radar optoelectronic multimodal classification method based on deep learning, inputs a radar optoelectronic multimodal image, performs joint recognition on water targets, and outputs the type of water targets.
[0046] Preferably, the sonar target matching module calls the underwater image acquisition device at the corresponding position according to the longitude and latitude of the sonar target, associates and matches the target in the underwater image with the sonar target, and obtains both sonar and underwater images of the same target, namely, a sonar multimodal image.
[0047] Preferably, the underwater target recognition module adopts a sonar multimodal classification method based on deep learning, inputs a sonar multimodal image, performs joint recognition on underwater targets, and outputs the type of underwater targets.
[0048] The beneficial effects of the above technical solution of the present invention are as follows:
[0049] In the above scheme, radar, optoelectronics, and underwater target detection modules all use target detection methods based on deep learning, which can quickly and accurately locate targets. Compared with traditional target detection methods, they have higher accuracy and high robustness in different scenarios. By associating and fusing sonar target tracks with radar and AIS target tracks, radar and sonar target positioning errors and false alarms can be reduced. At the same time, after distinguishing the targets into above-water and underwater targets, more targeted classification processing can be used respectively. For above-water or underwater targets, it is difficult to identify targets based on single-mode radar or sonar data. The use of multimodal classification methods can greatly improve the accuracy of target recognition and achieve target verification and fine-type recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 This is a structural diagram of an underwater target recognition system based on radar sonar multi-source information fusion according to an embodiment of the present invention.
[0051] Figure numerals: 1-radar information acquisition equipment; 2-pan-tilt optoelectronic equipment; 3-sonar information acquisition equipment; 4-underwater image acquisition equipment; 5-radar detection processor; 6-sonar detection processor; 7-fusion sorting processor; 8-radar recognition processor; 9-sonar recognition processor; 10-radar target detection module; 11-optoelectronic target detection module; 12-sonar target detection module; 13-underwater target detection module; 14-multi-source information fusion processing module; 15-above-water and underwater target sorting module; 16-radar optoelectronic linkage module; 17-above-water target recognition module; 18-sonar target matching module; 19-underwater target recognition module. DETAILED DESCRIPTION
[0052] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.
[0053] like Figure 1As shown, an embodiment of the present invention provides an above-water and underwater target recognition system based on radar sonar multi-source information fusion, the above-water and underwater target recognition system based on radar sonar multi-source information fusion includes a radar information acquisition device 1, a pan-tilt optoelectronic device 2, a sonar information acquisition device 3, an underwater image acquisition device 4, a radar detection processor 5, a sonar detection processor 6, a fusion sorting processor 7, a radar recognition processor 8, and a sonar recognition processor 9; the radar information acquisition device 1 and the pan-tilt optoelectronic device 2 are connected to the radar detection processor 5, the sonar information acquisition device 3 and the underwater image acquisition device 4 are connected to the sonar detection processor 6, the radar detection processor 5 is connected to the fusion sorting processor 7 and the radar recognition processor 8, the sonar detection processor 6 is connected to the fusion sorting processor 7 and the sonar recognition processor 9, and the fusion sorting processor 7 is connected to the radar recognition processor 8 and the sonar recognition processor 9;
[0054] The radar detection processor 5 includes a radar target detection module 10 and an optoelectronic target detection module 11. The radar target detection module 10 is connected to the radar information acquisition device 1, and the optoelectronic target detection module 11 is connected to the pan-tilt optoelectronic device 2;
[0055] The sonar detection processor 6 includes a sonar target detection module 12 and an underwater target detection module 13. The sonar target detection module 12 is connected to the sonar information acquisition device 3, and the underwater target detection module 13 is connected to the underwater image acquisition device 4.
[0056] The fusion sorting processor 7 includes a multi-source information fusion processing module 14 and an above-water and underwater target sorting module 15. The radar target detection module 10 and the sonar target detection module 12 are connected to the multi-source information fusion processing module 14, and the multi-source information fusion processing module 14 is connected to the above-water and underwater target sorting module 15.
[0057] The radar recognition processor 8 includes a radar optoelectronic linkage module 16 and an above-water target recognition module 17. The optoelectronic target detection module 11 is connected to the radar optoelectronic linkage module 16. The radar optoelectronic linkage module 16 and the above-water and underwater target sorting module 15 are connected to the above-water target recognition module 17.
[0058] The sonar recognition processor 9 includes a sonar target matching module 18 and an underwater target recognition module 19 . The underwater target detection module 13 is connected to the sonar target matching module 18 . The sonar target matching module 18 and the above-water and underwater target sorting module 15 are connected to the underwater target recognition module 19 .
[0059] Preferably, the radar information acquisition device 1 is a radar signal acquisition card that outputs a radar echo image.
[0060] Preferably, the pan / tilt optoelectronic device 2 is a network camera or an infrared thermal imager, which outputs visible light or infrared images.
[0061] Preferably, the sonar information acquisition device 3 is a sonar signal acquisition card that outputs sonar signal data.
[0062] Preferably, the underwater image acquisition device 4 is an underwater camera or an imaging sonar, which outputs underwater photoelectric or sonar images.
[0063] Preferably, the radar detection processor 5 is a computer or an embedded motherboard, which is equipped with a radar target detection module and a photoelectric target detection module.
[0064] Preferably, the sonar detection processor 6 is a computer or an embedded motherboard, and is equipped with a sonar target detection module and an underwater target detection module.
[0065] Preferably, the fusion sorting processor 7 is a computer or an embedded mainboard, which deploys a multi-source information fusion processing module and an above-water and underwater target sorting module.
[0066] Preferably, the radar identification processor 8 is a computer or an embedded mainboard, which is equipped with a radar optoelectronic linkage module and an aquatic target identification module.
[0067] Preferably, the sonar recognition processor 9 is a computer or an embedded mainboard, which is equipped with a sonar target matching module and an underwater target recognition module.
[0068] Preferably, the radar target detection module 10 adopts a radar target detection method based on deep learning, inputs a radar echo time-series image, obtains the target position in the radar image, converts it into longitude and latitude for real-time tracking, and outputs the radar target track.
[0069] Preferably, the photoelectric target detection module 11 adopts a photoelectric target detection method based on deep learning, inputs a visible light or infrared image, obtains the target position in the photoelectric image, tracks the target position in the image in real time, and outputs the photoelectric target trajectory.
[0070] Preferably, the sonar target detection module 12 adopts a traditional sonar target ranging lateral method, inputs sonar signal data, obtains sonar target direction and distance information, converts it into longitude and latitude for real-time tracking, and outputs the sonar target track.
[0071] Preferably, the underwater target detection module 13 adopts an underwater target detection method based on deep learning, inputs an underwater photoelectric or sonar image, obtains the target position in the underwater image, tracks the target position in the image in real time, and outputs the underwater target trajectory.
[0072] Preferably, the multi-source information fusion processing module 14 performs spatiotemporal registration on the radar track, sonar track, and AIS track data, uses the nearest neighbor method to perform track fusion, and outputs track association results, namely, three association matrices of radar and sonar, radar and AIS, and sonar and AIS, including the association rate between every two tracks.
[0073] Preferably, the above-water and underwater target sorting module 15 screens and distributes the track association results, determines the above-water and underwater targets based on the association rate in the track association matrix, sends the track data with a high probability of being an above-water target to the above-water target identification module for classification, and sends the track data with a high probability of being an underwater target to the underwater target identification module for classification.
[0074] Preferably, the radar optoelectronic linkage module 16 calculates the horizontal azimuth angle Pan, the vertical pitch angle Tilt, and the lens zoom factor Zoom, i.e., PTZ parameters, according to the longitude and latitude of the radar target, controls the optoelectronic device of the pan / tilt to aim at the radar target, and obtains two images, i.e., radar echo image and radar optoelectronic multimodal image, of the same target, i.e., radar optoelectronic multimodal image.
[0075] Preferably, the water target recognition module 17 adopts a radar optoelectronic multimodal classification method based on deep learning, inputs a radar optoelectronic multimodal image, performs joint recognition on water targets, and outputs the type of water targets.
[0076] Preferably, the sonar target matching module 18 calls the underwater image acquisition device at the corresponding position according to the longitude and latitude of the sonar target, associates and matches the target in the underwater image with the sonar target, and obtains the sonar and underwater images of the same target, that is, the sonar multimodal image. The sonar and underwater images of the same target include the sonar signal waveform of the sonar signal device and the underwater imaging image of the underwater imaging device (underwater camera or multi-beam sonar).
[0077] Preferably, the underwater target recognition module 19 adopts a sonar multimodal classification method based on deep learning, inputs a sonar multimodal image, performs joint recognition on underwater targets, and outputs the type of underwater targets.
[0078] The specific working process of the embodiment of the present invention is as follows:
[0079] 1) The radar information acquisition equipment collects radar signal data from shore-based or marine radars, converts it into network port or serial port data through analog-to-electronic conversion, inputs it into the radar detection processor, and generates radar echo images of continuous frames through data preprocessing. The radar target detection module uses the YOLO-based radar time-series image target detection algorithm to obtain the target frame position in the radar image, converts it into longitude and latitude coordinates in combination with the radar position, and then performs real-time tracking to obtain the radar target track, which is input into the fusion sorting processor. The track contains information such as the target's historical position, rectangular frame, confidence, and feature vector.
[0080] 2) The sonar information acquisition device collects sonar signal data from the passive listening sonar, converts it into network port or serial port data through analog-to-electronic conversion, inputs it into the sonar detection processor, and generates continuous frame sonar waveform images through data preprocessing. The sonar target detection module uses the traditional sonar target ranging lateral method to obtain the sonar target direction and distance information, combines the sonar position to convert it into longitude and latitude coordinates, and then performs real-time tracking to obtain the sonar target track, which is input into the fusion sorting processor. The track contains information such as the target's historical position, rectangular frame, confidence, frequency amplitude, etc.
[0081] 3) The multi-source information fusion processing module in the fusion sorting processor receives radar and sonar target tracks in real time, and obtains AIS target tracks through the AIS receiving device. The above three types of track data are temporally and spatially aligned, and the least squares method is used for interpolation to obtain data with the same time interval. For each historical position in the sonar target track, the nearest neighbor method is used to calculate the correlation rate with the corresponding position in the radar and AIS target tracks to form a track correlation matrix. The track correlation matrix is found to have the optimal allocation through the Hungarian algorithm, and the track correlation result is output.
[0082] The surface and underwater target sorting module screens and distributes the track association results, and sorts the track data based on the track history information and the association rate. Follow the steps below to associate the radar, sonar, and AIS target track sets in pairs: ① Determine each sonar target track with the radar and AIS target track. If there is a high correlation rate (such as greater than 0.7), the target is likely to be a surface target, such as surface ship radiation noise and active sonar signals, and is sent to the surface target identification module for further processing; ② Determine each sonar target track with the radar and AIS target track. If there is a low correlation rate or no correlation (such as less than 0.3), the target is likely to be an underwater target, such as a submarine underwater acoustic communication signal or a marine life signal, and is sent to the surface target identification module for further processing. The target is further processed by the surface target identification module; ③ Each sonar target track is judged against the radar and AIS target tracks. If there is a medium correlation rate (such as 0.3-0.7), the target is a suspected target and is difficult to judge directly. It may be caused by missed detection, tracking interruption, etc. It needs to be sent to the surface target identification module and the underwater target identification module for further processing; ④ The remaining radar and AIS target tracks that are not associated with the sonar target tracks are judged. The target is highly likely to be a surface target, such as a sea surface buoy, an island reef, or a low-altitude aircraft, and is sent to the surface target identification module for further processing.
[0083] 4) The radar optoelectronic linkage module in the radar identification processor receives the track data of the water target, and calculates the horizontal azimuth angle Pan, vertical pitch angle Tilt, and lens zoom multiple Zoom, i.e. PTZ parameters, based on the longitude and latitude of the radar target, and controls the optoelectronic device of the pan / tilt to aim at the radar target. The optoelectronic device of the pan / tilt collects visible light or infrared images and inputs them into the radar detection processor. The specific calculation method is as follows:
[0084] ① The target position (x, y, w, h) in the radar B-scan image can be used to calculate the target's azimuth distance (A1, D1) relative to the radar, where the radar coordinate system takes north as 0 degrees, the scanning range is 360°, the radar range is R, the radar image size is X*Y, A1=(x+w / 2) / X*360, D1=(Yyh / w) / Y*R.
[0085] ② The target longitude and latitude P2 can be calculated through the target relative position (A1, D1) and the radar longitude and latitude P1, and then combined with the longitude and latitude P3 of the optoelectronic device, the azimuth distance (A2, D2) of the target relative to the optoelectronic device can be calculated.
[0086] ③ Calculate the PTZ parameters through the relative position of the target (A2, D2), taking the sea surface target in the two-coordinate radar as an example.
[0087] Horizontal azimuth angle P = A2 + B, where B is the angle between 0 degrees of the photoelectric coordinate system and the north direction;
[0088] Vertical pitch angle T = arctan (L / D2), where L is the height of the optoelectronic equipment, and the optoelectronic coordinate system takes the horizontal as 0 degrees;
[0089] The lens zoom factor Z = D2*l / (W*f*2), where the CCD length is l, the minimum focal length is f, the target imaging occupies 1 / 2 of the image, and the actual length of the target is W = arctan(w / 2 / X*360-b)*D2*2, where b is the radar beam width.
[0090] The optoelectronic target detection module uses the optoelectronic time-series image target detection algorithm based on YOLO to obtain the target frame position in the visible light or infrared image. According to the position and parameters of the gimbal optoelectronic equipment, the longitude and latitude of the radar target are converted into image coordinates. The Hungarian algorithm is used to associate and match the corresponding targets in the radar optoelectronic image. Two local images of the same target are captured from the radar optoelectronic image, namely the radar echo local image and the visible light / infrared local image of the target, and input into the water target recognition module.
[0091] The water target recognition module adopts a radar optoelectronic dual-modal classification algorithm based on the cross-correlation matrix, combines the target features in the radar echo local image and the visible light / infrared local image, and jointly identifies the water targets, further distinguishes the categories of water targets and outputs them, such as warships, cargo ships, fishing boats, helicopters, etc.
[0092] 5) The sonar target matching module in the sonar recognition processor receives underwater target track data, and calls the underwater image acquisition device at the corresponding position according to the longitude and latitude of the sonar target, such as an underwater camera or active imaging sonar, to collect underwater visible light or sonar images, and inputs them into the sonar detection processor. In the scenario of ship navigation at sea, the ship-borne passive listening sonar corresponds to a ship-borne multi-beam imaging sonar. Based on the longitude and latitude of the underwater target discovered by the passive listening sonar, the underwater sonar image collected by the multi-beam imaging sonar is directly called and recorded at the same time; in the offshore security monitoring scenario, the seabed sonar array system corresponds to multiple underwater cameras. Based on the longitude and latitude of the underwater target discovered by the sonar array, the underwater visible light image collected by the underwater camera at the corresponding position is called and recorded at the same time.
[0093] The underwater target detection module uses the YOLO-based underwater time-series image target detection algorithm to obtain the target frame position in the underwater target image. According to the position and parameters of the underwater image acquisition equipment, the sonar target longitude and latitude are converted into image coordinates. The Hungarian algorithm is used to associate and match the corresponding targets in the sonar underwater image. Two local images of the same target are captured from the sonar underwater image, namely the sonar waveform local image and the underwater imaging local image of the target, and input into the underwater target recognition module.
[0094] The underwater target recognition module adopts a sonar underwater dual-modal classification algorithm based on the cross-correlation matrix, combines the target features in the sonar waveform local image and the underwater imaging local image, and jointly identifies underwater targets, further distinguishes and outputs underwater target categories, such as submarines, fish, rocks, sunken or suspended objects, etc.
[0095] The present invention does not elaborate on some of the well-known technologies that belong to those skilled in the art.
[0096] The above embodiments are provided only for the purpose of describing the present invention, and are not intended to limit the scope of the present invention. The scope of the present invention is defined by the appended claims. Various equivalent substitutions and modifications made without departing from the spirit and principles of the present invention should all be included within the scope of the present invention.
Claims
1. A surface and underwater target recognition system based on radar and sonar multi-source information fusion, characterized in that: It includes radar information acquisition equipment, pan-tilt optoelectronic equipment, sonar information acquisition equipment, underwater image acquisition equipment, radar detection processor, sonar detection processor, fusion sorting processor, radar identification processor, and sonar identification processor; the radar information acquisition equipment and pan-tilt optoelectronic equipment are connected to the radar detection processor, the sonar information acquisition equipment and underwater image acquisition equipment are connected to the sonar detection processor, the radar detection processor is connected to the fusion sorting processor and radar identification processor, the sonar detection processor is connected to the fusion sorting processor and sonar identification processor, and the fusion sorting processor is connected to the radar identification processor and sonar identification processor; The radar information acquisition device is used to collect radar signal data from shore-based or marine radars, and convert it into network port or serial port data to input into the radar detection processor for processing to generate continuous frame radar echo images; The pan / tilt optoelectronic device is used to aim at the radar target, obtain the radar optoelectronic image of the radar target and input it into the radar detection processor for processing to obtain the target position and perform real-time tracking, and output the optoelectronic target trajectory and the radar target track; The sonar information acquisition device is used to collect sonar signal data and convert it into network port or serial port data, and input it into the sonar detection processor for processing to generate continuous frame sonar waveform images; The underwater image acquisition device is used to collect underwater photoelectric or sonar images and input them into the sonar detection processor for processing to obtain the target position and track it in real time, and output the underwater target trajectory and sonar target track; The fusion sorting processor is used to receive the radar target track and sonar target track in real time, and obtain the AIS target track through the AIS receiving device; then the received three track data are time-space aligned, and the least square method is used for interpolation to obtain data with the same time interval; then the historical position in each sonar target track is calculated by the nearest neighbor method with the correlation rate of the corresponding position in the radar target track and the AIS target track to form a track correlation matrix, find the optimal allocation for the track correlation matrix, and output the track correlation result; then the track correlation result is screened and distributed, and the water target is sent to the radar recognition processor, and the underwater target is sent to the sonar recognition processor; The radar identification processor is used to control the PTZ optoelectronic device to aim at the radar target according to the optoelectronic target trajectory and the longitude and latitude of the radar target; The sonar recognition processor is used to call the underwater image acquisition device at the corresponding position to collect underwater visible light or sonar images according to the underwater target trajectory and the longitude and latitude of the sonar target.
2. The above-water and underwater target recognition system according to claim 1, characterized in that: The radar detection processor includes a radar target detection module and an optoelectronic target detection module. The radar target detection module is connected to the radar information acquisition device, and the optoelectronic target detection module is connected to the pan-tilt optoelectronic device. The radar target detection module uses a YOLO-based radar time-series image target detection algorithm to obtain the target frame position in the radar image, converts it into longitude and latitude coordinates in combination with the radar position, and then performs real-time tracking to obtain the radar target track, which is input into the fusion sorting processor. The photoelectric target detection module adopts a photoelectric target detection method based on deep learning to obtain the target position from the radar photoelectric image of the radar target and performs real-time tracking to output the photoelectric target trajectory.
3. The above-water and underwater target recognition system according to claim 2, characterized in that: The sonar detection processor includes a sonar target detection module and an underwater target detection module. The sonar target detection module is connected to the sonar information acquisition device, and the underwater target detection module is connected to the underwater image acquisition device. The sonar target detection module uses a sonar target ranging lateral method to obtain the sonar target direction and distance information, converts the sonar position into longitude and latitude coordinates, and then performs real-time tracking to obtain the sonar target track, which is input into the fusion sorting processor. The underwater target detection module uses an underwater target detection method based on deep learning to obtain the target position from the underwater optoelectronic or sonar image and tracks it in real time, and outputs the underwater target trajectory.
4. The above-water and underwater target recognition system according to claim 3, characterized in that: The fusion and sorting processor includes a multi-source information fusion processing module and an above-water and underwater target sorting module. The radar target detection module and the sonar target detection module are connected to the multi-source information fusion processing module, and the multi-source information fusion processing module is connected to the above-water and underwater target sorting module; the multi-source information fusion processing module is used to generate track association results according to the radar target track, the sonar target track, and the AIS target track; the above-water and underwater target sorting module is used to screen and distribute the track association results, sort the track data in combination with the track history information and the association rate, send the above-water targets to the radar recognition processor, and send the underwater targets to the sonar recognition processor.
5. The above-water and underwater target recognition system according to claim 4, characterized in that: The above-water and underwater target sorting module associates the radar target tracks, sonar target tracks, and AIS target tracks in pairs according to steps ① to ④: ① Each sonar target track is judged with the radar target track and the AIS target track. If the correlation rate is greater than the set high correlation threshold, the corresponding target is judged to be an above-water target; ② Each sonar target track is judged with the radar target track and the AIS target track. If the correlation rate is less than the set low correlation threshold or there is no correlation, the corresponding target is an underwater target; ③ Each sonar target track is judged with the radar target track and the AIS target track. If the correlation rate is between the high correlation threshold and the low correlation threshold, the corresponding target is sent to the radar recognition processor and the sonar recognition processor at the same time; ④ The remaining radar and AIS target tracks that are not associated with the sonar target track are judged, and the corresponding target is judged to be an above-water target.
6. The above-water and underwater target recognition system according to claim 4, characterized in that: The radar recognition processor includes a radar optoelectronic linkage module and an above-water target recognition module. The optoelectronic target detection module is connected to the radar optoelectronic linkage module, and the radar optoelectronic linkage module and the above-water and underwater target sorting module are connected to the above-water target recognition module; the radar optoelectronic linkage module is used to calculate the horizontal azimuth angle Pan, the vertical pitch angle Tilt, and the lens zoom multiple Zoom of the gimbal according to the longitude and latitude of the radar target, control the gimbal optoelectronic device to aim at the radar target, and obtain the radar optoelectronic multimodal image of the same target; the above-water target recognition module is used to adopt a radar optoelectronic multimodal classification method based on deep learning, and jointly identify the above-water target according to the input radar optoelectronic multimodal image, and output the above-water target type; the radar optoelectronic multimodal image of the same target includes the radar echo image and the radar optoelectronic image of the same target.
7. The above-water and underwater target recognition system according to claim 4, characterized in that: The sonar recognition processor includes a sonar target matching module and an underwater target recognition module. The underwater target detection module is connected to the sonar target matching module, and the sonar target matching module and the above-water and underwater target sorting module are connected to the underwater target recognition module. The sonar target matching module is used to call the underwater image acquisition device at the corresponding position according to the longitude and latitude of the sonar target, associate and match the target in the underwater image with the sonar target, and obtain the sonar multimodal image of the same target. The underwater target recognition module is used to adopt a sonar multimodal classification method based on deep learning, jointly identify the underwater target according to the input sonar multimodal image, and output the underwater target type. The sonar multimodal image of the same target includes the sonar image of the same target and the image of the underwater image acquisition device.
8. The above-water and underwater target recognition system according to claim 7, characterized in that: The underwater target detection module adopts the underwater time-series image target detection algorithm based on YOLO to obtain the target frame position in the underwater target image, converts the longitude and latitude of the sonar target into image coordinates according to the position and parameters of the underwater image acquisition device, adopts the Hungarian algorithm to associate and match the corresponding targets in the sonar underwater image, and respectively captures two local images of the same target from the sonar underwater image, namely, the sonar waveform local image and the underwater imaging local image of the target.
9. The above-water and underwater target recognition system according to claim 1, characterized in that: The optimal allocation of the track association matrix is found through the Hungarian algorithm, and the track association result is output.
Citation Information
Cited By
Radar-AIS ship tracking method and device based on visual feature optimization
CN120294743A
