A ship collision avoidance integrated method and system based on radar and vision fusion

Abstract

The present application provides an integrated ship collision avoidance method and system based on radar-visual fusion, which relates to the technical field of shore water traffic management, including synchronous detection of the echo information and visual information of the ship based on radar and infrared camera; fusing the echo feature vector and the visual feature vector based on adaptive weight adjustment to obtain a fused feature vector, using the Yolo network to identify and classify ships and obstacles, and tracking them in combination with the Deep SORT tracker; using the Kalman filter algorithm to fuse the echo information and visual information to obtain the real-time position, speed and acceleration of the target; analyzing the ship's trajectory, setting a safety threshold based on the static information of the ship provided by the ship-borne AIS module, and evaluating the risk of collision. The present application utilizes the information fusion of radar and infrared cameras to achieve real-time monitoring and precise positioning of surrounding obstacles. Through rapid processing and optimization algorithms, the system generates accurate positioning in real time, reducing the risk of false alarms and missed alarms.

Description

Technical Field

[0001] The present application relates to the technical field of shore water traffic management, and more specifically, to an integrated ship collision avoidance method and system based on radar and vision fusion. Background Art

[0002] Ship collision avoidance technology plays a vital role in ensuring maritime navigation safety, especially in busy ports and waterways. In ports and waterways, there are a large number of ships and a high density of traffic, which increases direct or potential traffic conflicts between ships and makes collision avoidance management more complicated. With the increase in the number of ships and the complexity of the navigation environment, traditional collision avoidance measures can no longer meet the needs of efficiency and safety. In order to improve the ability to manage coastal water traffic, the technical solutions used in the existing technology are:

[0003] (1) The Chinese invention patent with publication number CN113744570B discloses a method and device for ship collision prevention in bridge area waters, including obtaining radar characteristic information and AIS characteristic information of the target ship, and determining whether the radar characteristic information and the AIS characteristic information are correlated. If so, determining the navigation characteristic parameters of the target ship; determining the dynamic ship field of the target ship based on the navigation characteristic parameters of the target ship and the preset dynamic influence parameters; determining the target ship invasion depth factor according to the dynamic ship field of the target ship and the position of the target collision object, and determining the target ship invasion speed factor according to the speed in the navigation characteristic parameters of the target ship; determining the collision risk of the target ship according to the target ship invasion depth factor and the target ship invasion speed factor, and triggering a collision warning if the collision risk reaches the preset risk threshold. However, although radar can detect long-distance targets under various meteorological conditions, it often has the problem of insufficient accuracy in complex backgrounds and close-range recognition.

[0004] (2) The Chinese invention patent with publication number CN113990108A discloses a ship optimization identification and real-time tracking method and anti-collision warning system. The shore-based monitoring system, ship monitoring system and water level monitoring system all fuse data at the monitoring center. The optimization identification system and the ship real-time tracking system analyze and process the fused data at the monitoring center to obtain the height and predicted track of the ship. The shore-based monitoring system includes shore-based radar and pan-tilt camera arranged along the coastline, the ship monitoring system includes ship radar and ship AIS system, and the water level monitoring system includes a water level sensor arranged on the pier. The pan-tilt camera collects video images of the ship and accurately identifies the target ship through the optimization identification system. The shore-based radar and ship AIS system monitor the ship's position and speed, and perform coordinate transformation and speed transformation through the ship real-time tracking system. After obtaining the relevant transformation values, the ship's position and speed at the next moment are predicted. Although the pan-tilt camera provides high-resolution images, it is greatly affected by factors such as lighting and weather changes. In addition, in port waters with a large number of ships, the ship AIS system may not be able to update data in a timely manner, resulting in inaccurate or outdated information.

[0005] In summary, there is still a need for a water traffic management method that can be applied to a variety of weather and environments and to coastal waters with busy ship traffic. Summary of the invention

[0006] To solve the above problems, the technical solution adopted in this application is an integrated ship collision avoidance method based on radar and vision fusion, including:

[0007] Information acquisition: Based on radar and infrared cameras, the echo information and visual information of the ship are synchronously detected, and the echo information and visual information are processed into data accepted by the edge computing module through the data processing module, and then transmitted to the edge computing module for processing;

[0008] Risk assessment: The edge computing module represents the echo information as an echo feature vector, extracts the visual information through a convolutional neural network and represents it as a visual feature vector, fuses the echo feature vector and the visual feature vector based on adaptive weight adjustment to obtain a fused feature vector, uses the Yolo network to identify and classify ships and obstacles, and tracks them in combination with the Deep SORT tracker; uses the Kalman filter algorithm to fuse the echo information and visual information to obtain the real-time position, speed and acceleration of the target; analyzes the ship trajectory, sets a safety threshold based on the static information of the ship provided by the ship-borne AIS module, and assesses the risk of collision;

[0009] Alarm and remote data feedback: When a potential collision risk or ship speeding is detected, the alarm module transmits and prompts alarm information between the edge computing module, the ship-borne AIS module and the remote data center. The remote data center monitors the navigation status of the ship in real time through the information sent back by the edge computing module. The ship name, route, speed and location information provided by the ship-borne AIS module assists in target tracking and monitoring of the ship.

[0010] Shipborne AIS module warning response: After receiving the warning information from the remote data center, the shipborne AIS module displays the real-time position and navigation status of the ship, and automatically triggers an alarm when the ship's speed is too fast or the course deviates.

[0011] Optionally, the static information of the ship includes the ship name, call sign, captain, ship width and current load of the ship; setting the safety threshold based on the static information of the ship provided by the shipborne AIS module includes obtaining the ship's motion performance including the ship's design speed and acceleration / deceleration capability according to the ship name and call sign, correcting the ship's motion performance based on the ship's current load, and determining the safety threshold in combination with the ship's width and length.

[0012] Optionally, obtaining the fused feature vector includes: the distance of the radar to obtain the target , azimuth and speed , and pass these physical parameters to the edge computing module. The radar echo signal of the target is represented as a feature vector :

[0013] ;

[0014] In the formula, are distance, speed, and acceleration, respectively. is the feature vector of echo information;

[0015] Then, the optical infrared camera synchronously acquires the visual image of the target, and extracts features from the image data through a convolutional neural network, representing the extracted visual features as vectors. :

[0016] ;

[0017] In the formula, is the image acquired by optical infrared imaging, and CNN is the feature extraction operation of convolutional neural network;

[0018] The echo feature vector and the visual feature vector are mapped to the same feature space through a neural network. Before fusion, a small fully connected layer or convolutional layer is introduced for the echo feature and the visual feature respectively to adjust and align the dimensions of the two feature vectors, and then combine them to form a fused feature vector. :

[0019] ;

[0020] In the formula, and are the weight parameters of radar and visual features, which are determined by adaptive weight adjustment.

[0021] Optionally, the adaptive weight adjustment includes:

[0022] Data collection: Collect annotated datasets containing radar data and visual data. The annotated datasets cover a variety of natural conditions and objective scenarios to ensure that the model can learn performance in different environments.

[0023] Preprocessing: preprocessing the collected data according to the data preprocessing method of the data processing module;

[0024] Feature extraction: feature extraction is performed in the same way as the risk assessment step, and the echo information and visual information of the target are represented as echo feature vectors and visual feature vectors respectively;

[0025] Weight initialization: Initialize the weight parameters of the echo feature vector and the visual feature vector;

[0026] Model training: Use the labeled dataset to train the model and adjust the weight parameters of radar and visual features through the stochastic gradient descent algorithm;

[0027] Scenario adaptability adjustment: according to the actual application scenario, different weight parameters of radar and visual features are applied;

[0028] Performance testing and iterative optimization: Evaluate model performance on an independent test set and iteratively optimize the model based on the test results.

[0029] Optionally, obtaining the real-time position, velocity and acceleration of the target includes establishing a three-dimensional rectangular coordinate system {oxyz} with the millimeter-wave radar of the distributed radar vision module as the origin, wherein the X-axis points to the front of the radar, the Y-axis points to the right of the radar, and the Z-axis points to the top of the radar, and the three-dimensional coordinates (x, y, z) of the target are calculated by the following transformation matrix:

[0030] ;

[0031] Where R is the polar diameter between the target and the radar, is the target azimuth measured by the radar, is the target elevation angle measured by the radar;

[0032] The infrared camera tracks and photographs the target based on the azimuth and elevation information provided by the radar, obtaining the target's position in time. and The coordinate change within , the instantaneous velocity vector v( t ) and the acceleration vector a( t ) is estimated by the following finite difference method:

[0033] ;

[0034] ;

[0035] In the formula, and are the estimated error vectors of velocity and acceleration, respectively, is the time increment, Indicates time t- The velocity vector at the moment;

[0036] The signal processing module uses the Kalman filter algorithm to fuse the detection information of the radar and the optical infrared camera. In the scenario of three targets, the state vector of the target is , where x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinate values ​​of the three targets respectively, and the state transfer matrix F and the observation matrix H are defined as:

[0037] ;

[0038] ;

[0039] Through the prediction and update steps of the Kalman filter, the state vector S of the target is iteratively estimated to obtain the real-time position, velocity and acceleration of the target.

[0040] Optionally, after the alarm module sends an alarm to the shipborne AIS module and the remote data center, it enters a collision warning state. In this state, the remote data center sends a signal to the shipborne AIS module and the edge computing module through the communication module to instruct the ship to slow down or adjust its course to avoid a collision accident; adjust the working status of the radar and optical infrared camera, re-measure the distance between the ship and the obstacle, and determine whether to lift the alarm or take further measures based on the new data.

[0041] Optionally, the fusion detection process of the millimeter-wave radar and the image sensor includes: the detection results of the millimeter-wave radar and the neural network respectively generate corresponding target areas on the frame image, and the target detection information of the current scene is fused and output by the intersection and union of the millimeter-wave radar detection and neural network detection results on the graph, and the intersection and union between the recognition boxes is used to determine whether the object detected by the millimeter-wave radar and the object detected by the image are from the same target, and the DeepSORT target tracking algorithm is used to assign and track the unique identity of the ship based on the target detection result, and the next position of the target is predicted by Kalman filtering, and the Hungarian algorithm is combined for data association to ensure accurate tracking of the ship's running trajectory; finally, the risk of collision of the ship is evaluated based on the tracked trajectory information, and a waterway safety warning signal is provided in real time.

[0042] The present application also provides an integrated ship collision avoidance system based on radar and vision fusion, comprising:

[0043] Distributed radar vision module: used to synchronously detect the echo information and visual information of the ship based on radar and infrared camera;

[0044] Remote data center: used to receive the detection information of the distributed radar vision module, process the echo information and visual information into data accepted by the edge computing module through the data processing module, and then transmit it to the edge computing module for processing; the information returned in real time by the edge computing module is displayed to monitor the navigation status of the ship in real time, and the ship name, route, speed, and location information provided by the ship-borne AIS module are used to assist the target tracking and monitoring of the ship;

[0045] Edge computing module: used to represent the echo information of the target as an echo feature vector, extract the features of the visual information through a convolutional neural network and represent it as a visual feature vector, fuse the echo feature vector and the visual feature vector based on adaptive weight adjustment to obtain a fused feature vector, use the Yolo network to identify and classify ships and obstacles, and track them in combination with the Deep SORT tracker; use the Kalman filter algorithm to fuse the echo information and visual information to obtain the real-time position, speed and acceleration of the target; analyze the ship trajectory, set the safety threshold based on the static information of the ship provided by the ship-borne AIS module, and evaluate the collision risk;

[0046] Alarm module: used to transmit and prompt alarm information between the edge computing module, the ship-borne AIS module and the remote data center when the edge computing module detects a potential collision risk or a ship overspeeding;

[0047] Shipborne AIS module: used to provide static information of the ship, ship name, route, speed and position information. After receiving the warning information from the remote data center, it displays the real-time position and navigation status of the ship, and automatically triggers an alarm when the ship speed is too fast or deviates from the course.

[0048] Optionally, the distributed radar vision module includes a 77GHz long-range phased array radar and an optical infrared camera, the optical infrared camera is fixed above the 77GHz long-range phased array radar, the edge computing module is fixed to the rear of the distributed radar vision module through a rigid fixed bracket, the distributed radar vision module is installed on both sides of the channel at a distance of 10m from the water surface, and is suspended by a rigid fixed bracket, with a range of 1000m. The distributed radar vision module adopts an aluminum alloy shell material and is powered by a power module including a power cord and a battery;

[0049] The shipborne AIS module, which is composed of a shore-based data center and shipborne equipment, can display the routes, speeds and directions of all ships, ship names and ship category information on electronic charts, and track and monitor ships;

[0050] The remote data center includes a screen display module, a wireless communication module, and a data processing module, which are used to display current radar data and infrared data. When receiving the collision risk provided by the edge computing module, an alarm message is sent to the ship through the alarm module and displayed on the screen display module;

[0051] After receiving the alarm information, the alarm module installed in the shipborne AIS module controls the alarm device to send out an audible and visual alarm.

[0052] Optionally, the infrared camera has a built-in micro motor. When the target position information provided by the edge computing module is received, the micro motor starts to work, so that the optical infrared imaging module can quickly adjust to the direction of the target to collect visual information; the remote data center is used to control the working time, working cycle, measurement information and transmission cycle of the radar, as well as control the working time, rotation frequency and angle of the infrared turntable.

[0053] The beneficial effects of the integrated ship collision avoidance method and system based on radar and vision fusion provided by the present application are:

[0054] The integrated ship collision avoidance method based on radar and vision fusion has significantly improved the safety and navigation efficiency of ships in complex marine environments. The method uses the information fusion of radar and infrared cameras to realize real-time monitoring and precise positioning of surrounding obstacles. It can operate stably under various meteorological and lighting conditions. It sets safety thresholds based on the static information of the ship provided by the ship-borne AIS module to evaluate the collision risk, avoiding the problem that the ship AIS system may not be able to update data in time in busy sea areas, resulting in inaccurate or outdated information. It solves the problem of slow response of traditional collision avoidance technology in dynamic environments, and ensures the safety and intelligent management of ships when sailing at high speeds; based on the adaptive weight adjustment, the fusion of the echo feature vector and the visual feature vector is obtained to adapt the target recognition to different environmental scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art.

[0056] Figure 1 is a schematic diagram of an integrated ship collision avoidance system based on radar and vision fusion provided in an embodiment of the present application;

[0057] Figure 2 It is a flowchart of the operation of the integrated ship collision avoidance system based on radar and vision fusion provided in the embodiment of the present application;

[0058] Figure 3 It is a flowchart of the Deep SORT target tracking algorithm provided in the embodiment of the present application.

[0059] Explanation of the accompanying drawings: 1-distributed radar module; 11-77GHz long-range phased array radar; 12-optical infrared camera; 2-fixed bracket; 3-shipborne AIS module; 4-remote data center; 41-screen display module; 42-wireless communication module; 43-data processing module; 5-alarm module; 6-power module; 7-edge computing module. DETAILED DESCRIPTION

[0060] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] like Figure 2 As shown, a ship collision avoidance integrated method based on radar and vision fusion includes:

[0062] Information acquisition: Based on radar and infrared cameras, the echo information and visual information of the ship are synchronously detected, and the echo information and visual information are processed into data accepted by the edge computing module through the data processing module, and then transmitted to the edge computing module for processing;

[0063] Processing the echo information and the visual information includes converting the radar echo information and the visual information of the infrared camera into a specific data format, which is a JSON format in this embodiment, and storing the data in a structured manner and maintaining the hierarchy and relationship of the data. The radar echo information can be encapsulated into a JSON object containing fields such as timestamp, signal strength, angle, etc., and the visual information of the infrared camera is encapsulated so that the JSON contains sufficient information, including the width, height, and color depth of the image, and the image data is stored as Base64-encoded pixel data.

[0064] Risk assessment: The edge computing module represents the echo information as an echo feature vector, extracts the visual information through a convolutional neural network and represents it as a visual feature vector, obtains a fused feature vector by fusing the echo feature vector and the visual feature vector based on adaptive weight adjustment, uses the Yolo network to identify and classify ships and obstacles, and combines it with the Deep SORT tracker for tracking, such as Figure 3 As shown in the figure; the Kalman filter algorithm is used to fuse the echo information and visual information to obtain the real-time position, velocity and acceleration of the target; the ship trajectory is analyzed, and the safety threshold is set based on the static information of the ship provided by the ship-borne AIS (Automatic Identification System) module to evaluate the collision risk.

[0065] When the edge computing module detects that the distance between the ship and the obstacle is close to the safety threshold, it triggers the interruption of the alarm module, sounds an alarm, and prompts the ship to adjust its course or speed. Specifically, the shortest distance between the ship and the obstacle is calculated based on the ship's speed and position detected by the radar and visual system, as well as the position of the obstacle. When the distance reaches the critical value set by the system, the communication module sends a signal to the remote data center and the alarm module, triggering the interruption of the alarm module, causing the alarm device to sound and light alarms, prompting the crew to take necessary risk avoidance operations. At the same time, the remote data center will send a signal to instruct the ship to slow down or adjust its course to avoid collision accidents.

[0066] The video information captured by the optical infrared camera will be divided into frames of image information. The yolov5 target detection algorithm will detect the ship target in the single frame image, and then the Deep SORT tracking algorithm will be used to predict the ship's trajectory to assess whether there is a collision risk.

[0067] Alarm and remote data feedback: When a potential collision risk or ship speeding is detected, the alarm module transmits and prompts alarm information between the edge computing module, the ship-borne AIS module and the remote data center. The remote data center monitors the navigation status of the ship in real time through the information sent back by the edge computing module. The ship-borne AIS module provides the ship name, route, speed, and location information to assist in target tracking and monitoring of the ship.

[0068] When the detection and tracking result is inconsistent with the route, speed and position information corresponding to the ship name provided by the ship-borne AIS module, a positioning deviation alarm is issued to the ship-borne AIS module.

[0069] Shipborne AIS module warning response: After receiving the warning information from the remote data center, the shipborne AIS module displays the real-time position and navigation status of the ship, and automatically triggers an alarm when the ship speed is too fast or the course deviates. When the ship detects that the speed and course deviate from the set speed, and the edge computing module detects that there is a risk of collision or the ship speed is too fast, the alarm module of the shipborne AIS module will sound an alarm.

[0070] In another embodiment of the present application, the static information of the ship includes the ship name, call sign, captain, ship width and current load of the ship. Setting the safety threshold based on the static information of the ship provided by the shipborne AIS module includes obtaining the ship's motion performance including the ship's design speed and acceleration / deceleration capability according to the ship name and call sign, correcting the ship's motion performance based on the ship's current load, and determining the safety threshold in combination with the ship's width and length.

[0071] Assume that through the ship name and call sign, it is found that the ship's design speed is 15 knots, the design acceleration is 0.2 m / s, the design deceleration is 0.3 m / s, the ship's length is 120 meters, the ship's width is 25 meters, and the empty ship weight is 2000 tons;

[0072] Calculate the effect of load on motion performance: Assume that for every 1000 tons of load added, acceleration decreases by 0.01 m / s and deceleration increases by 0.01 m / s.

[0073] Safety threshold calculation: The collision avoidance distance is 1.5 times the ship length, the deceleration distance is calculated based on the deceleration and the design speed, and the lateral safety distance is 1.5 times the ship width. The safety threshold calculation results are shown in Table 1.

[0074] Table 1 Ship load and safety threshold adjustment table

[0075]

[0076] The speed calculation in actual application scenarios includes factors such as water flow velocity, wind speed and wind direction, and the impact of load on motion performance is usually not linear. The resistance of the ship in the water may increase with the increase of speed. At high speeds, the increase in resistance may cause the rate of increase of deceleration to exceed the linear prediction. This embodiment simplifies the remaining environmental factors.

[0077] In another embodiment of the present application, obtaining the fused feature vector includes: obtaining the distance of the target by the radar , azimuth and speed , and pass these physical parameters to the edge computing module. The radar echo signal of the target is represented as a feature vector :

[0078] ;

[0079] In the formula, are distance, speed, and acceleration, respectively. is the feature vector of echo information;

[0080] Then, the optical infrared camera synchronously acquires the visual image of the target, and extracts features from the image data through a convolutional neural network, representing the extracted visual features as vectors. :

[0081] ;

[0082] In the formula, The image is obtained by optical infrared imaging, and CNN is the feature extraction operation of convolutional neural network.

[0083] The echo feature vector and the visual feature vector are mapped to the same feature space through a neural network. Before fusion, a small fully connected layer or convolutional layer is introduced for the echo feature and the visual feature respectively to adjust and align the dimensions of the two feature vectors, and then combine them to form a fused feature vector. :

[0084] ;

[0085] In the formula, and are the weight parameters of radar and visual features, which are determined by adaptive weight adjustment.

[0086] The fully connected layer is used to convert the feature map into a one-dimensional feature vector. In the fully connected layer, each input neuron is connected to each output neuron. If the dimension of the input feature map is H×W×D (where H is the height, W is the width, and D is the depth or number of channels), the fully connected layer flattens these features into a one-dimensional vector, and then performs a linear transformation through the weight matrix W and the bias b, that is, Z=Wx+b, where Z represents the output vector and x represents the input vector. Through this process, the dimension of the feature can be adjusted to match the next layer of the network.

[0087] The convolution layer slides the convolution kernel on the input feature map and performs weighted summation on the local area to extract features. The convolution layer can adjust the dimension of the output feature map by changing the number, size or stride of the convolution kernel.

[0088] The convolution layer processes data through local connections and parameter sharing, and is suitable for processing data with spatial correlation such as visual feature vectors and echo feature vectors. The fully connected layer connects each neuron to all neurons in the previous layer to achieve comprehensive utilization of global features. It has advantages in feature integration when processing visual feature vectors and echo feature vectors with multiple feature parameters. This embodiment can choose any of the above methods for dimensionality adjustment.

[0089] In another embodiment of the present application, the adaptive weight adjustment includes:

[0090] Data collection: Collect annotated datasets containing radar data and visual data. The annotated datasets cover a variety of natural conditions and objective scenarios to ensure that the model can learn performance in different environments.

[0091] Natural conditions include weather conditions, seasonal changes, day and night changes, and sea conditions. Scenes include the wakes generated by the movement of a large number of ships and the density of ships. Scenes are classified according to the order of magnitude of natural conditions and ship density.

[0092] Preprocessing: preprocess the collected data according to the data preprocessing method of the data processing module;

[0093] Feature extraction: feature extraction is performed in the same way as the risk assessment step, and the echo information and visual information of the target are represented as echo feature vectors and visual feature vectors respectively;

[0094] Weight initialization: Initialize the weight parameters of the echo feature vector and the visual feature vector;

[0095] Model training: Use the labeled dataset to train the model and adjust the weight parameters of radar and visual features through the stochastic gradient descent algorithm;

[0096] Scenario adaptability adjustment: according to the actual application scenario, different weight parameters of radar and visual features are applied;

[0097] Performance testing and iterative optimization: Evaluate model performance on an independent test set and iteratively optimize the model based on the test results.

[0098] In another embodiment of the present application, obtaining the real-time position, velocity and acceleration of the target includes establishing a three-dimensional rectangular coordinate system {oxyz} with the millimeter-wave radar of the distributed radar vision module as the origin, wherein the X-axis points to the front of the radar, the Y-axis points to the right of the radar, and the Z-axis points to the top of the radar. The three-dimensional coordinates (x, y, z) of the target are calculated by the following transformation matrix:

[0099] ;

[0100] Where R is the polar diameter between the target and the radar, is the target azimuth measured by the radar, is the target elevation angle measured by the radar;

[0101] The infrared camera tracks and photographs the target based on the azimuth and elevation information provided by the radar, obtaining the target's position in time. and The coordinate change within , the instantaneous velocity vector v and acceleration vector a of the target are estimated by the following finite difference method:

[0102] ;

[0103] ;

[0104] In the formula, and are the estimated error vectors of velocity and acceleration respectively;

[0105] The signal processing module uses the Kalman filter algorithm to fuse the detection information of the radar and the optical infrared camera. In the scenario of three targets, the state vector of the target is , where x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinate values ​​of the three targets respectively, and the state transfer matrix F and the observation matrix H are defined as:

[0106] ;

[0107] ;

[0108] Through the prediction and update steps of the Kalman filter, the state vector S of the target is iteratively estimated to obtain the real-time position, velocity and acceleration of the target. Finally, the signal processing module uses the fused target motion state information to update the navigation status of the ship and perform anti-collision warning based on it, ensuring the high accuracy and real-time performance of the warning.

[0109] The millimeter wave radar is used to obtain the current moving speed and distance of the target object. The process of obtaining the current distance information of the target object through the millimeter wave radar is as follows: In the millimeter wave radar system, the sinusoidal signal emitted by the transmitting antenna is assumed to be , the reflected signal received by the receiving antenna is , these two signals can be expressed as follows:

[0110] ;

[0111] Where w1 and w2 represent the angular frequencies of the transmitted and received signals, respectively. and Respectively represent the initial phases of the transmitted signal and the received signal;

[0112] The linear frequency modulation pulse slope of the transmitted signal is equal to the linear frequency modulation pulse slope of the received signal. The two signals are relatively unchanged. Therefore, the intermediate frequency signal output by the mixer is a sinusoidal signal with a constant frequency. The frequency of the intermediate frequency signal can be expressed by the following formula:

[0113] ;

[0114] In the formula, Sτ represents the slope of the linear frequency modulated continuous wave at time τ. According to the above formula, the distance between the radar and the detection target can be obtained: :

[0115] ;

[0116] in represents the frequency of the intermediate frequency signal, represents the slope of the linear frequency modulated continuous wave, Represents the speed of light.

[0117] The process of obtaining the current speed information of the target object through the millimeter wave radar is as follows: the millimeter wave radar can use two linear frequency modulation pulses to measure the speed of the target in the scene, and transmit the pulses with an interval of Linear frequency modulation pulse, the peak difference of the FFT signal obtained by the reflection of the linear frequency modulation pulse is given by Phase difference, get the speed of the measured target v for:

[0118] ;

[0119] Where λ represents the wavelength of the radar signal.

[0120] In another embodiment of the present application, after the alarm module sends an alarm to the shipborne AIS module and the remote data center, it enters a collision warning state. In this state, the remote data center sends a signal to the shipborne AIS module and the edge computing module through the communication module to instruct the ship to slow down or adjust its course to avoid a collision accident; adjust the working status of the radar and optical infrared camera, re-measure the distance between the ship and the obstacle, and determine whether to lift the alarm or take further measures based on the new data.

[0121] After receiving the alarm information from the communication module, the remote data center records the ship number, time, location and warning information, and dispatches the ship to take safety measures. Specifically, the remote data center receives the ship position and navigation status information transmitted by the communication module in real time during the ship's navigation. When receiving the collision warning signal, the remote data center will immediately record the current ship number, time, location and obstacle information, and notify the crew to operate. The remote data center will send a signal to the ship-borne AIS module and edge computing module through the communication module to adjust the working status of the radar and optical infrared camera, re-measure the distance between the ship and the obstacle, and determine whether to lift the alarm or take further measures based on the new data.

[0122] In another embodiment of the present application, the fusion detection process of the millimeter-wave radar and the image sensor includes: the detection results of the millimeter-wave radar and the neural network respectively generate corresponding target areas on the frame image, and the target detection information of the current scene is fused and output by the intersection and union of the millimeter-wave radar detection and the neural network detection results on the graph, and the intersection and union between the recognition boxes is used to determine whether the object detected by the millimeter-wave radar and the object detected by the image are from the same target. The Deep SORT target tracking algorithm is used to assign and track the unique identity of the ship based on the target detection results, and the next position of the target is predicted by Kalman filtering, and the Hungarian algorithm is used for data association to ensure accurate tracking of the ship's running trajectory; finally, the ship is evaluated for collision risk based on the tracked trajectory information, and a waterway safety warning signal is provided in real time.

[0123] Through this recognition method of combining radar and vision fusion with deep learning, the system can accurately identify and classify targets in complex environments, further improving the effectiveness and safety of the anti-collision system. Based on the improved yolov5+Deep SORT, the improved method proposed to solve the problem of mutual occlusion on the sea surface is mainly used. After the target detection frame is obtained, it is input to the tracking end to complete the subsequent prediction, matching and update tasks, and finally output the tracking result with higher detection accuracy.

[0124] like Figure 1As shown, the present application also provides an integrated ship collision avoidance system based on radar and vision fusion, comprising:

[0125] Distributed radar and vision module 1: used to synchronously detect the echo information and visual information of the ship based on radar and infrared camera;

[0126] Remote data center 4: used to receive the detection information of the distributed radar vision module 1, process the echo information and visual information into data accepted by the edge computing module 7 through the data processing module 43, and then transmit it to the edge computing module 7 for processing; the information transmitted back in real time by the edge computing module 7 is displayed to monitor the navigation status of the ship in real time, and the ship name, route, speed, and position information provided by the ship-borne AIS module 3 are used to assist the target tracking and monitoring of the ship.

[0127] Edge computing module 7: used to represent the echo information of the target as an echo feature vector, extract the visual information through a convolutional neural network and represent it as a visual feature vector, obtain a fused feature vector by fusing the echo feature vector and the visual feature vector based on adaptive weight adjustment, use the Yolo network to identify and classify ships and obstacles, and track them in combination with the Deep SORT tracker; use the Kalman filter algorithm to fuse the echo information and visual information to obtain the real-time position, speed and acceleration of the target; analyze the ship trajectory, set the safety threshold based on the static information of the ship provided by the shipborne AIS module 3, and evaluate the collision risk; the edge computing module 7 provides high-performance computing capabilities for the distributed radar vision module 1, and uses a deep learning algorithm to monitor whether the ship is in danger of collision.

[0128] Alarm module 5: used to transmit and prompt alarm information between edge computing module 7, ship-borne AIS module 3 and remote data center 4 when edge computing module 7 detects potential collision risk or ship speeding;

[0129] Shipborne AIS module 3: used to provide static information of the ship, ship name, route, speed and position information. After receiving the warning information from the remote data center 4, it displays the real-time position and navigation status of the ship, and automatically triggers an alarm when the ship speed is too fast or deviates from the course.

[0130] In another embodiment of the present application, the distributed radar vision module 1 includes a 77 GHz long-range phased array radar 11 and an optical infrared camera 12. The optical infrared camera 12 is fixed above the 77 GHz long-range phased array radar 11. The edge computing module 7 is fixed to the rear of the distributed radar vision module 1 through a rigid fixed bracket 2. The distributed radar vision module 1 is installed on both sides of the channel 10 m away from the water surface and is suspended by a rigid fixed bracket 2. The effective range is 1000 m. The distributed radar vision module 1 adopts an aluminum alloy shell material and is powered by a power module 6 including a power cord and a battery. It operates in a combination of a power cord and a battery. Even when the external power supply is cut off, the system can still continue to be powered by the battery to ensure the reliability and high availability of the system.

[0131] The 77GHz long-range phased array radar 11 has a beam width of ±45° and a detection range of 1,000 meters. It is mainly used to measure the position of ships, floating objects and reefs at sea. It can locate targets more accurately, improve measurement resolution, and reduce interference to the surroundings. The optical infrared camera 12 has a maximum irradiation distance of up to 1 kilometer. The rotation of the optical infrared imaging module can be controlled by a built-in micro motor. It can rotate 45° in each direction, up, down, left, and right. The resolution is 1920*1080, and it can image both during the day and at night.

[0132] Millimeter-wave radar has good penetration and high-precision measurement, and can stably detect targets in adverse weather conditions, but its data processing is relatively complex and may be affected by the environment; while infrared imaging performs well in nighttime monitoring and providing high-resolution images, but is not effective in conditions such as haze. Radar-vision fusion uses the complementary advantages of the two to achieve all-weather monitoring, improve the accuracy and safety of target identification, and optimize navigation paths through intelligent decision support, effectively reducing collision risks.

[0133] In this embodiment, 5 to 8 distributed radar vision modules 1 are arranged in the same detection range coverage area, and the areas prone to collision accidents are covered at this density. The shell material of the distributed radar vision module 1 should be able to resist sea breeze erosion, wind and rain to ensure that it can still operate stably under various severe weather and environmental conditions. At the same time, it has protective measures, such as waterproof, dustproof, and vibration-proof, and can work normally within a wide temperature range.

[0134] Shipborne AIS module 3, composed of a shore-based data center and shipborne equipment, can display the routes, speeds and directions of all ships, ship names and ship categories on electronic charts, and track and monitor ships;

[0135] The remote data center 4 includes a display module 41, a wireless communication module 42, and a data processing module 43, which are used to display the current radar data and infrared data. When receiving the collision risk provided by the edge computing module 7, the alarm module 5 sends an alarm message to the ship and displays it on the display module 41;

[0136] In this embodiment, the edge computing module 7 uses Rockchip's RK3588 edge computing board. RK3588 is a new generation flagship high-end processor launched by Rockchip. It adopts 8nm process design, equipped with quad-core A76 + quad-core A55 eight-core CPU and Arm high-performance GPU, built-in 6T computing power NPU, and an external waterproof metal protective shell. RK3588 has efficient visual processing capabilities, coupled with a deep learning network, can have a good recognition effect on ships and obstacles in infrared images. At the same time, combined with radar image data, it evaluates whether the current ship is in danger of collision.

[0137] After receiving the alarm information, the alarm module 5 installed in the shipboard AIS module 3 controls the alarm device to emit an audible and visual alarm to remind the crew to take corrective measures. This module can communicate with the remote data center 4 and the shipboard AIS module 3 in real time to ensure that the warning information can be quickly transmitted.

[0138] In another embodiment of the present application, the optical infrared camera 12 has a built-in micro motor. After receiving the target position information provided by the edge computing module 7, the micro motor works to enable the optical infrared imaging module to quickly adjust to the direction of the target to collect visual information; the remote data center 4 is used to control the working time, working cycle, measurement information and transmission cycle of the radar, as well as control the working time, rotation frequency and angle of the infrared turntable.

[0139] After receiving the target's location information, the micro motor works to enable the optical infrared imaging module to quickly adjust to the direction of the target for filming.

[0140] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A ship collision avoidance integrated method based on radar and vision fusion, characterized in that: include: Information acquisition: Based on radar and infrared cameras, the echo information and visual information of the ship are synchronously detected, and the echo information and visual information are processed into data accepted by the edge computing module through the data processing module, and then transmitted to the edge computing module for processing; Risk assessment: The edge computing module represents the echo information as an echo feature vector, extracts the visual information through a convolutional neural network and represents it as a visual feature vector, fuses the echo feature vector and the visual feature vector based on adaptive weight adjustment to obtain a fused feature vector, uses the Yolo network to identify and classify ships and obstacles, and tracks them in combination with the Deep SORT tracker; uses the Kalman filter algorithm to fuse the echo information and visual information to obtain the real-time position, speed and acceleration of the target; analyzes the ship trajectory, sets a safety threshold based on the static information of the ship provided by the ship-borne AIS module, and assesses the risk of collision; Alarm and remote data feedback: When a potential collision risk or ship speeding is detected, the alarm module transmits and prompts alarm information between the edge computing module, the ship-borne AIS module and the remote data center. The remote data center monitors the navigation status of the ship in real time through the information sent back by the edge computing module. The ship name, route, speed and location information provided by the ship-borne AIS module assists in target tracking and monitoring of the ship. Shipborne AIS module warning response: After receiving the warning information from the remote data center, the shipborne AIS module displays the real-time position and navigation status of the ship, and automatically triggers an alarm when the ship speed is too fast or deviates from the course; The real-time position, velocity and acceleration of the target are obtained by establishing a three-dimensional rectangular coordinate system {oxyz} with the millimeter-wave radar of the distributed radar vision module as the origin, wherein the X-axis points to the front of the radar, the Y-axis points to the right of the radar, and the Z-axis points to the top of the radar. The three-dimensional coordinates (x, y, z) of the target are calculated by the following transformation matrix: ； Where R is the polar diameter between the target and the radar, is the target azimuth measured by the radar, is the target elevation angle measured by the radar; The infrared camera tracks and photographs the target based on the azimuth and elevation information provided by the radar, obtaining the target's position in time. and The coordinate change within , the instantaneous velocity vector v( t ) and the acceleration vector a( t ) is estimated by the following finite difference method: ； ； In the formula, and are the estimated error vectors of velocity and acceleration, respectively, is the time increment, Indicates time t- The velocity vector at the moment; The signal processing module uses the Kalman filter algorithm to fuse the detection information of the radar and the optical infrared camera. In the scenario of three targets, the state vector of the target is , where x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinate values ​​of the three targets respectively, and the state transfer matrix F and the observation matrix H are defined as: ； ； Through the prediction and update steps of the Kalman filter, the state vector S of the target is iteratively estimated to obtain the real-time position, velocity and acceleration of the target.

2. The integrated ship collision avoidance method based on radar and visual fusion according to claim 1 is characterized in that: The ship static information includes the ship name, call sign, length, ship width and current load of the ship; the setting of the safety threshold based on the ship static information provided by the ship-borne AIS module includes obtaining the ship motion performance including the ship design speed and acceleration / deceleration capability according to the ship name and call sign, correcting the ship motion performance based on the ship current load, and determining the safety threshold in combination with the ship width and length.

3. The integrated ship collision avoidance method based on radar and visual fusion as claimed in claim 1, characterized in that: The fusion feature vector is obtained by: , azimuth and speed , and pass these physical parameters to the edge computing module. The radar echo signal of the target is represented as a feature vector : ； In the formula, are distance, speed, and acceleration, respectively. is the feature vector of echo information; Then, the optical infrared camera synchronously acquires the visual image of the target, and extracts features from the image data through a convolutional neural network, representing the extracted visual features as vectors. : ； In the formula, is the image acquired by optical infrared imaging, and CNN is the feature extraction operation of convolutional neural network; The echo feature vector and the visual feature vector are mapped to the same feature space through a neural network. Before fusion, a small fully connected layer or convolutional layer is introduced for the echo feature and the visual feature respectively to adjust and align the dimensions of the two feature vectors, and then combine them to form a fused feature vector. : ； In the formula, and are the weight parameters of radar and visual features, which are determined by adaptive weight adjustment.

4. The integrated ship collision avoidance method based on radar and visual fusion as claimed in claim 3 is characterized by: The adaptive weight adjustment includes: Data collection: Collect annotated datasets containing radar data and visual data. The annotated datasets cover a variety of natural conditions and objective scenarios to ensure that the model can learn performance in different environments. Preprocessing: preprocessing the collected data according to the data preprocessing method of the data processing module; Feature extraction: feature extraction is performed in the same way as the risk assessment step, and the echo information and visual information of the target are represented as echo feature vectors and visual feature vectors respectively; Weight initialization: Initialize the weight parameters of the echo feature vector and the visual feature vector; Model training: Use the labeled dataset to train the model and adjust the weight parameters of radar and visual features through the stochastic gradient descent algorithm; Scenario adaptability adjustment: according to the actual application scenario, different weight parameters of radar and visual features are applied; Performance testing and iterative optimization: Evaluate model performance on an independent test set and iteratively optimize the model based on the test results.

5. The integrated ship collision avoidance method based on radar and visual fusion as claimed in claim 1, characterized in that: After the alarm module sends an alarm to the shipborne AIS module and the remote data center, it enters the collision warning state. In this state, the remote data center sends a signal to the shipborne AIS module and the edge computing module through the communication module to instruct the ship to slow down or adjust its course to avoid a collision accident; adjust the working status of the radar and optical infrared camera, re-measure the distance between the ship and the obstacle, and determine whether to lift the alarm or take further measures based on the new data.

6. The integrated ship collision avoidance method based on radar and visual fusion as claimed in claim 5 is characterized by: The fusion detection process of millimeter-wave radar and image sensor includes: the detection results of millimeter-wave radar and neural network generate corresponding target areas on the frame image respectively; the target detection information of the current scene is fused and output by the intersection and union of millimeter-wave radar detection and neural network detection results on the graph; the intersection and union between recognition frames is used to determine whether the object detected by millimeter-wave radar and the object detected by image are from the same target; the Deep SORT target tracking algorithm is used to assign and track the unique identity of the ship based on the target detection results; the next position of the target is predicted by Kalman filtering, and the Hungarian algorithm is used for data association to ensure accurate tracking of the ship's trajectory; finally, the risk of collision of the ship is evaluated based on the tracked trajectory information, and a waterway safety warning signal is provided in real time.

7. An integrated ship collision avoidance system based on radar and vision fusion, characterized in that: include: Distributed radar vision module: used to synchronously detect the echo information and visual information of the ship based on radar and infrared camera; Remote data center: used to receive the detection information of the distributed radar vision module, process the echo information and visual information into data accepted by the edge computing module through the data processing module, and then transmit it to the edge computing module for processing; the information returned in real time by the edge computing module is displayed to monitor the navigation status of the ship in real time, and the ship name, route, speed, and location information provided by the ship-borne AIS module are used to assist the target tracking and monitoring of the ship; Edge computing module: used to represent the echo information of the target as an echo feature vector, extract the features of the visual information through a convolutional neural network and represent it as a visual feature vector, fuse the echo feature vector and the visual feature vector based on adaptive weight adjustment to obtain a fused feature vector, use the Yolo network to identify and classify ships and obstacles, and track them in combination with the Deep SORT tracker; use the Kalman filter algorithm to fuse the echo information and visual information to obtain the real-time position, speed and acceleration of the target; analyze the ship trajectory, set the safety threshold based on the static information of the ship provided by the ship-borne AIS module, and evaluate the collision risk; Alarm module: used to transmit and prompt alarm information between the edge computing module, the ship-borne AIS module and the remote data center when the edge computing module detects a potential collision risk or a ship overspeeding; Shipborne AIS module: used to provide static information, ship name, route, speed and location information of the ship. After receiving the warning information from the remote data center, it displays the real-time location and navigation status of the ship, and automatically triggers an alarm when the ship speed is too fast or deviates from the course; The real-time position, velocity and acceleration of the target are obtained by establishing a three-dimensional rectangular coordinate system {oxyz} with the millimeter-wave radar of the distributed radar vision module as the origin, wherein the X-axis points to the front of the radar, the Y-axis points to the right of the radar, and the Z-axis points to the top of the radar. The three-dimensional coordinates (x, y, z) of the target are calculated by the following transformation matrix: ； Where R is the polar diameter between the target and the radar, is the target azimuth measured by the radar, is the target elevation angle measured by the radar; The infrared camera tracks and photographs the target based on the azimuth and elevation information provided by the radar, obtaining the target's position in time. and The coordinate change within , the instantaneous velocity vector v( t ) and the acceleration vector a( t ) is estimated by the following finite difference method: ； ； In the formula, and are the estimated error vectors of velocity and acceleration, respectively, is the time increment, Indicates time t- The velocity vector at the moment; The signal processing module uses the Kalman filter algorithm to fuse the detection information of the radar and the optical infrared camera. In the scenario of three targets, the state vector of the target is , where x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinate values ​​of the three targets respectively, and the state transfer matrix F and the observation matrix H are defined as: ； ； Through the prediction and update steps of the Kalman filter, the state vector S of the target is iteratively estimated to obtain the real-time position, velocity and acceleration of the target.

8. The integrated ship collision avoidance system based on radar and visual fusion as claimed in claim 7, characterized in that: The distributed radar vision module includes a 77GHz long-range phased array radar and an optical infrared camera. The optical infrared camera is fixed above the 77GHz long-range phased array radar. The edge computing module is fixed to the rear of the distributed radar vision module through a rigid fixed bracket. The distributed radar vision module is installed on both sides of the channel 10m away from the water surface and suspended by a rigid fixed bracket. The range of action is 1000m. The distributed radar vision module adopts an aluminum alloy shell material and is powered by a power module including a power cord and a battery. The shipborne AIS module, which is composed of a shore-based data center and shipborne equipment, can display the routes, speeds and directions of all ships, ship names and ship category information on electronic charts, and track and monitor ships; The remote data center includes a screen display module, a wireless communication module, and a data processing module, which are used to display current radar data and infrared data. When receiving the collision risk provided by the edge computing module, an alarm message is sent to the ship through the alarm module and displayed on the screen display module; After receiving the alarm information, the alarm module installed in the shipborne AIS module controls the alarm device to send out an audible and visual alarm.

9. The integrated ship collision avoidance system based on radar and visual fusion as claimed in claim 7, characterized in that: The infrared camera has a built-in micro motor. When receiving the target position information provided by the edge computing module, the micro motor works to enable the optical infrared imaging module to quickly adjust to the direction of the target to collect visual information. The remote data center is used to control the working time, working cycle, measurement information and transmission cycle of the radar, and control the working time, rotation frequency and angle of the infrared turntable.

Images