A new energy ship navigation situation awareness system, method and medium

Abstract

This invention provides a navigation situational awareness system, method, and medium for new energy vessels, belonging to the field of intelligent ship technology. The system includes a multi-source heterogeneous sensor group, a data preprocessing and spatiotemporal synchronization module, a multi-source data deep fusion module, a static and dynamic integrated situational awareness construction module, and a risk warning and energy efficiency auxiliary decision-making module. The multi-source data deep fusion module adopts a three-level progressive fusion architecture: the first level generates a preliminary fusion target through complementary fusion of radar and AIS data; the second level introduces video data for heterogeneous data enhancement fusion; the third level performs situational correlation fusion of dynamic targets with electronic chart static data, sudden obstacle data, and the electric propulsion status and energy consumption data of new energy vessels to construct four-dimensional fusion data. This invention achieves full-scene, blind-spot-free perception in complex waters and low-visibility environments, with rapid early warning response and high compatibility with the characteristics of new energy vessels, effectively reducing energy consumption while ensuring navigation safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent ship technology, and in particular to a new energy ship navigation situation perception system, method and medium. Background Technology

[0002] Existing ship navigation situational awareness technologies are insufficient to meet the special safety and energy efficiency requirements of new energy ships, especially electric propulsion ships, mainly due to the following key limitations: At the perception level, existing systems mostly rely on single or simple combinations of sensors such as radar, automatic identification systems (AIS), and cameras, resulting in a fragmented perception problem. Radar is susceptible to weather interference such as rain, snow, and waves, and its accuracy in identifying close-range targets is insufficient, making it difficult to distinguish between static and dynamic targets. AIS has blind spots in monitoring small vessels and fishing boats whose equipment is not turned on. Video cameras have a sharp drop in recognition capability in low visibility or sudden changes in lighting conditions, resulting in incomplete perception coverage across the entire scene.

[0003] At the multi-source data processing level, data from different sensors suffer from heterogeneous formats (such as radar point clouds, AIS messages, and video streams) and spatiotemporal asynchrony. Existing fusion technologies mostly involve simple overlay or shallow correlation, and have not established a deep fusion mechanism specifically for the navigation scenarios of new energy vessels, resulting in low target recognition accuracy and high data redundancy.

[0004] In terms of target perception, existing systems fail to achieve integrated perception of static targets such as bridges, shoals, and fishing nets, as well as dynamic targets such as other vessels and floating objects. Static target information largely relies on pre-set electronic nautical charts and cannot be updated in real time to address sudden obstacles; dynamic target tracking is susceptible to environmental interference, leading to trajectory drift.

[0005] In terms of risk warning and energy efficiency adaptation, traditional risk warning is calculated solely based on the distance to nearest encounter (DCPA) and time to nearest encounter (TCPA), failing to consider the response delay and braking characteristics of the electric propulsion system of new energy ships. This results in a mismatch between the timing of warnings and the actual operational needs of the ship. Furthermore, the sensing system is not linked to energy consumption data, thus failing to provide decision support for energy efficiency optimization of new energy ships.

[0006] In summary, existing technologies suffer from problems such as fragmented perception, inefficient fusion, disconnect between static and dynamic target perception, and lack of risk warning and energy efficiency adaptation, making it difficult to meet the navigation needs of new energy ships under high safety and low energy consumption requirements. Summary of the Invention

[0007] The purpose of this invention is to provide a navigation situation awareness system, method and medium for new energy ships, so as to solve the technical problems of fragmented perception, inefficient multi-source data fusion, separation of static and dynamic target perception and incompatibility between risk warning and the characteristics of new energy ships in the prior art.

[0008] To achieve the above objectives, the present invention provides a new energy ship navigation situational awareness system, comprising: A multi-source heterogeneous sensor array is used to collect multi-source data on the ship's navigation environment and its own status. The data preprocessing and spatiotemporal synchronization module is used to perform format standardization and spatiotemporal synchronization processing on multi-source data collected by multi-source heterogeneous sensor groups. The multi-source data deep fusion module is used to perform three-level progressive fusion of spatiotemporally synchronized multi-source data. The three-level progressive fusion includes: complementary fusion of homogeneous data, enhanced fusion of heterogeneous data, and situational correlation fusion, generating four-dimensional fused data containing environmental, target, ship self, and energy consumption information. The static and dynamic integrated situation building module is used to build an integrated navigation situation based on four-dimensional fusion data, which includes dynamic target situation, static target situation and the ship's own situation. The risk warning and energy efficiency auxiliary decision-making module is used to conduct dynamic risk warning based on the integrated navigation situation and the characteristics of the electric propulsion system of new energy ships, and to provide energy efficiency optimization decisions based on the correlation between sensing data and energy consumption data.

[0009] This invention also provides a method for situational awareness of new energy vessels, applied to the aforementioned system, comprising the following steps: Multi-source data on the ship's navigation environment and its own status are collected through a multi-source heterogeneous sensor array. The collected multi-source data is standardized in format and synchronized in time and space. The spatiotemporally synchronized multi-source data is fused in a three-level progressive manner to generate four-dimensional fused data that includes information on the environment, target, ship itself, and energy consumption. An integrated navigation situation is constructed based on four-dimensional fusion data, which includes dynamic target situation, static target situation and the ship's own situation; Based on the integrated navigation situation, dynamic risk warnings are carried out in combination with the characteristics of the electric propulsion system of new energy ships, and energy efficiency optimization decisions are provided based on the correlation between sensing data and energy consumption data.

[0010] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0011] Therefore, the present invention employs the above-mentioned new energy ship navigation situational awareness system, method, and medium, and the beneficial technical effects are as follows: (1) In terms of perception accuracy and coverage, this invention effectively solves the problem of the limitations of single sensor perception through a three-level progressive heterogeneous data deep fusion mechanism. It can effectively improve the accuracy of dynamic target recognition and the recognition rate of ships without AIS enabled, significantly reduce the response time for sudden static obstacle recognition, and realize full-scene blind-spot-free perception in complex waters and low-visibility environments.

[0012] (2) Regarding the timeliness and adaptability of risk warning, this invention embeds the response delay and braking characteristics of the electric propulsion system of new energy ships into the risk assessment model and dynamically adjusts the warning threshold. This greatly shortens the warning response time; the warning timing is precisely matched with the actual operation requirements of the ship, which can effectively reduce the collision risk rate.

[0013] (3) Regarding navigation energy efficiency and passage efficiency, this invention establishes a correlation mechanism between sensing data and energy consumption data of new energy ships, and recommends the optimal navigation plan in real time through reinforcement learning algorithms. Under the premise of ensuring safety, it significantly reduces navigation energy consumption and accurately controls the prediction error of remaining range; the accurate marking of static targets and the recommendation of the optimal path improve the passage efficiency of ships and greatly reduce the number of ineffective turns and decelerations.

[0014] (4) In terms of system adaptability and practicality, this invention supports diverse navigation scenarios in inland waterways, coastal areas, and complex waters, and can be adapted to new energy vessels of different tonnages and types, including pure electric propulsion vessels and hybrid power vessels. The visual interactive interface adopts a multi-mode switching and color coding design, which reduces the difficulty of operation for crew members; the full-process data traceability function supports data storage of more than 180 days and multi-speed playback, providing complete data support for system optimization, fault diagnosis, and accident review. Attached Figure Description

[0015] Figure 1 This is a hardware framework diagram of a new energy ship navigation situational awareness system. Figure 2 A software framework diagram of a new energy ship navigation situational awareness system; Figure 3 A software data flow diagram for a new energy ship navigation situational awareness system; Figure 4 This is a schematic diagram of a three-level progressive multi-source data deep fusion. Detailed Implementation

[0016] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0017] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0018] Example 1 This embodiment provides a new energy ship navigation situation awareness system based on multi-source heterogeneous data fusion. The system includes a hardware platform and software functional modules running on it, and is applicable to electric propulsion new energy ships in complex waters such as inland rivers, coastal areas, bridge areas, and narrow waterways.

[0019] I. Hardware Components.

[0020] Reference Figure 1 The hardware components of this system mainly include core sensing devices and high-performance processing servers.

[0021] 1. Core sensing equipment.

[0022] (1) Solid-state navigation radar: The new generation of SIMRAD solid-state radar is adopted, with a working frequency of 9.3 to 9.5 GHz and a range of 50 m to 24 nautical miles (n mile). The radar has optimized the close-range target recognition algorithm, and the accuracy can reach ±0.5 m when the detection distance is ≤100 m.

[0023] When detecting at close range, the radar switches to short-range mode and increases the bandwidth of the transmitted signal. Utilizing pulse compression technology in solid-state radar, a matched filter compresses the received wide pulse into an extremely narrow pulse. According to the formula... ( The minimum distance between two adjacent targets that a radar can distinguish. At the speed of light, (for bandwidth), so the algorithm increases the bandwidth to over 150MHz.

[0024] After the radar echo undergoes a Fast Fourier Transform (FFT), the target energy is distributed across multiple adjacent range gates. The optimization algorithm employs the three-point Gaussian centroid method: selecting the target peak and its two adjacent sampling points, and using their power spectral density for weighted estimation. By calculating the geometric center of the echo energy envelope, the ranging accuracy is improved from discrete sampling intervals to continuous values ​​within 0.5m at the digital level.

[0025] The radar incorporates a weather interference suppression module to reduce the impact of rain, snow, and ocean waves on the echo signal. This module employs segmented adaptive threshold logic. For each detection unit, the average noise power of its surrounding reference units is calculated in real time. When the environment enters rainy or snowy weather, the probability distribution of background noise changes from a Rayleigh distribution to... Distribution. The meteorological interference suppression module automatically adjusts the weighting factor. The specific process is as follows: the system calculates the kurtosis coefficient of the echo power within the reference unit in real time. This coefficient is used to quantify the impulse characteristics of the noise distribution. When The background is determined to conform to a Rayleigh distribution, weighted by factors. Assign the value to the first preset value of 1.5; when Determine the background distribution direction Distribution transformation, Automatically switch to the second preset value 2.2; when The background was determined to exhibit a significant long-tail distribution, indicating extreme weather disturbances. The value is assigned to the third preset value of 3.0. Only when the echo signal-to-noise ratio exceeds this dynamic threshold (3dB higher than the background level) will the signal be identified as the target, thereby filtering out weak rain and snow echoes distributed over a large area.

[0026] The module stores radar images from 3-5 consecutive scans and performs temporal coherent accumulation. If an echo from a certain area appears only in the current scan and has no trajectory correlation in the previous two scans, the algorithm identifies it as sudden meteorological noise and performs pixel suppression; conversely, it performs energy enhancement on signals with smooth displacement trajectories.

[0027] To address the characteristics of strong sea clutter at close range and weak clutter at long range, the module incorporates a built-in adaptive gain slope adjuster. It automatically adjusts the receiver's gain increment based on the real-time detected sea state level. First, the adjuster calculates the average background power of the radar in non-target areas within the close-range (50m-500m) area in real time. This is used as an evaluation index for low, medium, and high sea state levels. The system matches the corresponding gain correction coefficient according to the sea state level. When in low sea states When in mid-sea state When in high sea states The regulator is based on Real-time calculation of the distance after radar electromagnetic waves are reflected from obstacles Changing gain increment function ,according to The compensation law restores the gain to counteract the physical property of wave clutter attenuating with distance. The formula is expressed as: ; in, This is the system's reference slope constant, typically taken as 40. In this embodiment, 50m is used as the starting point for calculating the near-range reference distance.

[0028] In practical applications, at extremely short ranges, the formula above is used to calculate... Approaching zero, the system suppresses strong wave crest reflections by rapidly reducing sensitivity; as the detection distance increases, the gain recovers rapidly and linearly according to the aforementioned logarithmic compensation law. The operator's adaptive amplification of the slope ensures that short-range clutter is effectively suppressed below the noise floor under adverse weather conditions, while the echo intensity of medium- and long-range targets is fully compensated, thereby maximizing the target echo contrast.

[0029] (2) Multispectral Network Camera: The camera is a Hikvision 8MP wide-angle full-color + infrared dual-mode camera with a horizontal field of view of 180° and a large F1.0 aperture. It supports automatic switching between visible light (low-light full-color) and infrared (night / foggy) modes to fill in visual perception blind spots in low-visibility environments.

[0030] (3) The enhanced AIS terminal uses SPAT-1000B Class B+ equipment, operating in the frequency band of 156.025~162.025MHz. This terminal adds a passive detection module, and its specific operation process is as follows: The system monitors the energy distribution within the aforementioned operating frequency band using real-time Fast Fourier Transform. When small vessels use their VHF radios for voice calls, or when their onboard electronic equipment, such as navigation systems or radar, generates unintentional radio frequency leakage, the carrier energy generated will be captured by the passive detection module even if the vessel does not actively transmit AIS coordinate data. The algorithm automatically determines whether the signal originates from a ship's radiation source based on the envelope characteristics of the captured FM signal: the envelope amplitude of the FM signal is basically constant, while the envelope of other interference signals fluctuates significantly, thus distinguishing ship communication signals from background noise.

[0031] Upon acquiring a weak radio signal, the system uses a Multiple Signal Classification (MUSIC) algorithm for direction finding. Specifically, the passive detection module is equipped with an array antenna consisting of multiple antenna elements, each receiving the same signal with a phase difference. The MUSIC algorithm constructs the covariance matrix of the received signal and performs eigenvalue decomposition to separate the signal subspace and noise subspace. Then, it uses the orthogonality between the signal subspace and the steering vector to scan the direction of arrival, thereby obtaining one or more radio azimuth lines, i.e., the direction lines of the signal source. The system then overlays these azimuth lines onto the radar point cloud map in real time.

[0032] If a radar echo is detected along a certain azimuth line, but there is no corresponding AIS message at that location, the system will mark the radar target as a suspected target with AIS disabled and assign it a temporary identifier generated from passive detection characteristics, namely the passive signal characteristic ID. This ID contains information such as signal frequency, modulation type, and first detection time.

[0033] As the target vessel moves, the system continuously performs multi-point direction finding on the same signal source, acquiring multiple azimuth lines at different times and observation locations. A recursive least squares method is used to perform intersection calculations on these multiple azimuth lines: for each new azimuth line obtained, the algorithm updates the estimate of the target's geographic coordinates, minimizing the sum of the squared weighted distances from all azimuth lines to the estimated point. The weights are dynamically adjusted based on signal strength and direction finding accuracy. Through continuous iteration, the system gradually calculates the target's dynamic geographic coordinates, thereby enabling the positioning of vessels without AIS equipment activated.

[0034] This module can not only receive weak signals from small vessels that have not enabled AIS and assist in positioning, but also expand the static information repository and add identification fields for special vessel types such as dangerous goods vessels and fishing vessels, which facilitates subsequent navigation situation awareness and risk warning.

[0035] (4) Marine satellite electronic compass: It adopts the NGC-3000 enhanced system with an orientation accuracy of 0.4°RMS. It integrates a ship attitude sensor with roll and pitch measurement accuracy of ±0.1° and can output heading, track and ship attitude data simultaneously.

[0036] (5) New energy-specific sensors: including electric propulsion system status sensors (real-time acquisition of motor speed, torque output, and battery energy consumption data) and water flow speed / direction sensors (measurement accuracy ±0.05m / s), integrated in the electric propulsion system control cabinet, providing key data support for risk warning and energy efficiency optimization.

[0037] (6) High-performance processing server: Equipped with an Intel i9-13900K CPU, dual RTX4090 graphics cards (parallel computing), and a built-in ship-specific AI acceleration chip, providing computing power support for deep learning model inference, real-time data processing, and situation building in subsequent software modules, ensuring that the overall system response latency is ≤300ms. All the above sensors are connected to the server through the shipboard Ethernet switch, and network time synchronization is performed using the NTP protocol. The data collected in real time by each sensor is transmitted to the server for processing via UDP or multicast.

[0038] II. Software Functional Modules.

[0039] Reference Figure 2 and Figure 3 The software portion of this system consists of the following modules: 1. Heterogeneous data preprocessing and precise spatiotemporal synchronization module.

[0040] (1) Format standardization: For different formats such as radar point cloud data, AIS message data, video stream data, compass attitude data, and energy consumption data, a unified data encapsulation protocol is designed, which includes target ID, timestamp, spatial coordinates, data type, and confidence field to achieve format normalization of heterogeneous data.

[0041] (2) Spatiotemporal synchronization optimization: Timestamp alignment of multi-sensor data is achieved based on satellite time synchronization (synchronization error ≤ 5ms); spatial coordinate transformation algorithms are used to map spatial data from different sensors to the same coordinate system, resolving the spatial offset problem between radar, AIS, and video. The specific process of spatial coordinate transformation is as follows: a) Radar coordinate transformation.

[0042] The polar coordinate data (distance) detected by the radar Azimuth Convert to local rectangular coordinates: ; in, , These are the x-coordinate and y-coordinate of the target in the local rectangular coordinate system, respectively.

[0043] b) AIS coordinate transformation.

[0044] Using WGS-84 ellipsoid parameters (major axis) First eccentricity Convert AIS latitude and longitude coordinates to Cartesian rectangular coordinates. : ; in, , , These are the altitude, longitude, and latitude of the target vessel, respectively. Let be the radius of curvature of the y-axis.

[0045] Let the ship's own reference position be ,in, This is the ship's latitude. This is the longitude of the ship. The altitude of this ship is given by the Cartesian coordinate system. The target ship's coordinates in the ship's coordinate system are: : ; Wherein, rotation matrix for: ; c) Video coordinate transformation.

[0046] According to the pinhole imaging model, the relationship between pixel coordinates and world coordinates is as follows: ; in, This is the camera intrinsic parameter matrix, including focal length and principal point parameters; This is the camera extrinsic parameter matrix, including parameters such as installation height, pitch angle, and yaw angle; As a scale factor, and These are the horizontal and vertical pixel coordinates, respectively.

[0047] If we define sea level =0, then the above relationship simplifies to a homography matrix. Inverse projection is the process of obtaining sea level coordinates from pixel coordinates. , ),in , , The target point has three-dimensional coordinates in the world coordinate system.

[0048] Assume the installation height is The camera's pitch angle relative to the sea level is Then the inverse projection formula can be expressed as: ; in, The coordinates of the principal point in the image; It represents the pixel focal length.

[0049] (3) Data cleaning and noise reduction: An adaptive threshold filtering algorithm is used to remove meteorological interference noise in the radar echo. Taking the current range cell to be measured as the center, reference cells on both sides are selected to estimate the local background noise power. Real-time calculation of current echo power The ratio to the local background power, i.e. The algorithm sets a detection threshold that dynamically fluctuates according to the ambient noise level. At that time, the system determined that the echo signal was masked by meteorological noise and could not effectively extract target features, so it was regarded as redundant noise and removed; static redundant information in the video stream was removed by the inter-frame difference method. First, the images of two adjacent frames (the first and second frames) were compared. Frame and the The first frame is converted to grayscale and histogram equalized to eliminate brightness deviations caused by uneven lighting. A pixel-by-pixel subtraction operation is then performed on the two frames to obtain the difference image. ,in These are the pixel coordinates. Set the pixel change threshold. (Generally taken as 30), if the difference value at a certain point If the pixel is a static background (such as sky, distant scenery, or camera noise), its grayscale value is forcibly set to 0 (black); if If the pixel change is preserved, the data quality will be improved.

[0050] 2. Three-level progressive multi-source data deep fusion module.

[0051] Reference Figure 4 This module adopts a three-level progressive fusion architecture: (1) First-level fusion unit: complementary fusion of data from the same source (radar + AIS).

[0052] An improved Kalman filter algorithm is employed to match and correlate dynamic targets detected by radar with ship information received by AIS. The improvement lies in introducing a target motion trend prediction factor. The algorithm monitors the radar observations of the filter in real time. Compared with model predictions The residuals between .

[0053] ; in, It reflects the degree of deviation between the actual motion of the ship and the preset uniform speed or uniform acceleration model; This is the observation matrix, whose function is to convert the predicted state into the predicted observation value.

[0054] Constructing target movement trend prediction factors As a quantitative indicator of state prediction bias, the ratio of the sum of squared residuals to the covariance is calculated using the Mahalanobis Distance principle: ; in, For the normalized sum of squared residuals, For the mobility threshold, This is the new information covariance matrix. When the ship maintains steady-state navigation... The filter converges with a normal step size; when the ship undergoes a sharp turn or change in speed... During a surge, It grows rapidly, anticipating that the target is entering a period of high mobility.

[0055] Using target movement trend prediction factors Real-time magnified state prediction covariance matrix : ; in, For the first The state prediction covariance matrix at time t. For the first The state transition matrix at time t, For the first The state estimation covariance matrix at time t. This indicates transpose.

[0056] By increasing In the next moment, the algorithm automatically reduces its reliance on the motion model and increases the weight of real-time radar observations, i.e., increases the Kalman gain. This ensures that the trajectory can quickly keep up with the target at critical moments such as when the ship turns, preventing correlation breaks caused by excessive prediction deviations.

[0057] If the target has associated AIS data, its static attributes are directly extracted using the unique MMSI (Maritime Mobile Service Identity). If the target does not have AIS enabled, its physical dimensions and vessel type are inferred by using the trajectory after the prediction factor has stabilized and a multi-frame smoothing algorithm.

[0058] Finally, confidence scores were calculated to establish a comprehensive scoring model. : ; in, Continuous tracking duration for the target This represents the average motion deviation. If the score exceeds the threshold, a temporary ID is assigned and the system is added to the obstacle avoidance monitoring system. , These are the weighting coefficients.

[0059] This algorithm completes the dynamic information (speed, heading) and static information (ship type, size) of ships without AIS, generating a preliminary fusion target (including a unique target ID and confidence score). For conflicting radar and AIS data, such as positional discrepancies, weights are dynamically assigned based on sensor accuracy for weighted correction: with the ship as the center, distances less than 1 nautical mile are considered close range, and greater than 3 nautical miles are considered long range. Between 1 and 3 nautical miles, weights are dynamically adjusted using linear interpolation. At close range, the radar weight is 0.7 and the AIS weight is 0.3; at long range, the radar weight is 0.3 and the AIS weight is 0.7.

[0060] (2) Second-level fusion unit: heterogeneous data enhancement fusion (fusion target + video).

[0061] The enhanced YOLOv8 algorithm is used to identify targets such as ships and obstacles in videos. This enhanced algorithm is optimized for water navigation scenarios. a) Introduce the P2 micro-target detection head.

[0062] In water scenes, distant ships occupy only a small number of pixels in the image. Traditional YOLOv8's downsampling factor is too large, causing the loss of geometric details of these tiny objects in the deep feature maps. The algorithm adds a P2 feature layer (4x downsampling) to the Neck part, fusing the high-resolution features from the shallow layer with the semantic features from the deep layer through skip connections, enabling the model to better capture these details. It has a stronger sensitivity to distant obstacles below the pixel level.

[0063] b) Integrated coordinate attention mechanism.

[0064] Specular reflections from sea waves are often misidentified as buoys or small boats. To enable the model to automatically filter out sea clutter, the algorithm introduces a Coordinate Attention (CA) mechanism. The CA operator is embedded after the C2f module in the backbone network. This operator captures cross-channel orientation information by performing global pooling in both horizontal and vertical directions. This allows the network to focus more on the horizontal contours and vertical structure of ships when extracting features, rather than the random wave flickering in the background, thus effectively suppressing false detections caused by strong sea surface reflections.

[0065] c) Improve the loss function.

[0066] Aquatic environments are often accompanied by fog or water mist, causing target edges to blur. Traditional CIoU loss functions are overly sensitive to the quality of the bounding boxes, making them prone to oscillations during convergence. The localization loss function is replaced with Wise-IoUv3. Wise-IoUv3 introduces an outlier evaluation metric based on a dynamic non-monotonic focusing mechanism. : ; in, This is the gradient gain adjustment coefficient, set to 3.0. The weighting adjustment factor is set to 1.9. The baseline IoU loss component is dimensionless. By dynamically allocating weights, the algorithm can focus on learning typical samples of moderate quality, accelerating the convergence speed of the model under complex sea conditions and improving the regression accuracy of the target bounding box when the ship is swaying.

[0067] The BoT-SORT+Kalman filter joint tracking algorithm generates video target trajectories: BoT-SORT uses camera motion compensation and bounding box smoothing to improve tracking stability, while Kalman filtering predicts the position of the next frame based on the target motion model. The combination of the two can effectively deal with problems such as ship maneuvering and occlusion, and generate continuous and stable video target trajectories.

[0068] Spatial alignment between radar / AIS fusion targets and video targets is achieved through precise world coordinate-pixel coordinate conversion. This conversion is based on intrinsic and extrinsic parameter matrices obtained from camera calibration. The intrinsic parameter matrix includes focal length and principal point coordinates, while the extrinsic parameter matrix includes the camera's installation position and angle relative to the ship's hull. The conversion first transforms the world coordinates to the camera coordinate system, then obtains pixel coordinates through projection transformation, and performs real-time corrections based on ship attitude data. After alignment, visual features extracted from the video targets, such as ship outline, color, and superstructure features, are used to enhance the attributes of the fusion targets: cargo ships and passenger ships are distinguished based on the aspect ratio of the outline and the superstructure shape; fishing boats are identified based on the ship's color and deck features, thus outputting an enhanced fusion target containing richer attribute information.

[0069] (3) Third-level fusion unit: situational correlation fusion (static and dynamic targets + new energy ship data).

[0070] The enhanced fusion target output from the second stage is spatiotemporally correlated with static target data obtained from electronic charts, such as bridges, shoals, and recommended channels; sudden static obstacle data output in real time by the radar and video joint identification module, such as fishing nets, floating objects, and construction zones; and electric propulsion status data (response delay, maximum braking torque) and energy consumption data obtained from new energy-specific sensors. This constructs a four-dimensional fusion data system of "environment-target-ship-energy consumption." The workflow of the radar and video joint identification module is as follows: a) Extraction of features from heterogeneous targets.

[0071] The radar-side algorithm automatically compares real-time point cloud data with electronic nautical chart data, eliminating known static targets such as bridges and reefs. For the remaining isolated echo points, its reflection intensity, radial velocity (those close to 0 are considered static), and spatial distribution characteristics are extracted. On the video side, the enhanced YOLOv8 algorithm performs real-time target detection on the full-view image, extracting the category probability and mask features of potential obstacles.

[0072] b) Extraction and mapping of Region of Interest (ROI).

[0073] Using the aforementioned spatial coordinate transformation matrix, the coordinates of the suspected static target detected by the radar are... The image is projected onto a real-time video frame to generate a corresponding Region of Interest (ROI). This step effectively avoids interference from the complex background of the open sea on the visual algorithm.

[0074] c) Multi-source joint identification.

[0075] The system performs two-level verification on targets within the ROI area: 1) It calculates the overlap between the center of the bounding box of the target identified in the video and the center point of the radar projection. If the deviation is within a threshold (0.05, i.e., 5% of the image size), it is determined to be a high-value potential obstacle. 2) It uses a deep learning model to classify the target. If it is identified as a fishing net buoy or floating object, its physical coordinates are locked using radar ranging information; if it is identified as a construction area, its physical edge contour is extracted using semantic segmentation technology and mapped to a global raster map.

[0076] d) Confidence-based decision-making.

[0077] If the radar has an echo but the video cannot identify it due to insufficient lighting, a conservative unknown obstacle mark is given based on the radar reflective area; conversely, if the visual system detects an obstacle but the radar does not have an echo, such as a semi-submersible floating object, a close-range warning is given based on the visual trajectory.

[0078] 3. Static and dynamic integrated situational awareness construction module.

[0079] (1) Dynamic target status: Output target ID, real-time position (accuracy ≤ 0.3m), speed, heading, ship type, motion trend (predicting the position in the next 5s based on historical trajectory), confidence score, and sort them by priority: dangerous goods ship > large ship > small ship > floating object.

[0080] (2) Static target situation: Based on the basic data of electronic nautical chart, sudden static obstacles identified by radar and video are superimposed (the type, size and distance of the obstacle are marked). Through visual augmented reality (AR) technology, information such as bridge clearance height, shoal depth, lock location and temporary no-navigation zone are simultaneously marked in the electronic nautical chart and video. The static target data update frequency is ≤1s.

[0081] (3) Ship's own status: Integrate compass attitude data, electric propulsion system status data, and energy consumption data to display information such as ship heading, roll / tune angle, current energy consumption level, and estimated remaining range in real time.

[0082] 4. New energy-adaptive dynamic risk warning and energy efficiency auxiliary decision-making module.

[0083] (1) Dynamic risk early warning mechanism: Based on four-dimensional fusion data, a four-factor risk assessment model of "distance-speed-ship characteristics-environment" is constructed. This model optimizes the traditional DCPA / TCPA calculation: First, the system aligns solid-state radar, enhanced AIS, and compass data to obtain the motion vectors of the ship and the fused target in a unified coordinate system. Traditional DCPA calculations are based solely on ground velocity, while this model uses dedicated new energy sensors to calibrate the ship's electric propulsion status data in real time, calculating a precise relative velocity vector considering dynamic loads. and relative distance vector .

[0084] Secondly, traditional DCPA treats the ship as a point mass, resulting in insufficient accuracy in close-range obstacle avoidance. This algorithm utilizes the ship size information from the fused target output of the second-level unit, combined with static target data acquired by the third-level unit, to expand the ship's safe zone. The calculation method is as follows: ; in, Dynamically floating based on grid properties, in narrow waterways or complex navigational obstructions. The value was increased from 1.0 to 1.3, thereby enabling a dynamic shift of the warning threshold toward safety redundancy, ensuring accurate characterization of collision risk during extremely close-range detection. This is the total length of the ship; The estimated length of the target vessel is obtained by the second-level fusion unit through AIS data extraction or video feature recognition; The relative bearing of the two ships.

[0085] Furthermore, an optimized TCPA model combining time delay and braking performance loss is established. To make the warning time practically meaningful for operation, the algorithm subtracts the system sensing and communication delay from the traditional geometric time. And the dynamic braking compensation item constrained by battery SOC: ; in, To achieve the maximum braking deceleration of the propulsion system, This is an efficiency operator based on the battery's state of charge. When When the amount of electricity decreases, the last term in the formula increases, leading to a change in the calculated value. The corresponding timeframe is shortened, thus triggering a high-level early warning in advance.

[0086] The final risk level classification criteria are as follows: Low risk: >1 nautical mile or If the interval exceeds 30 minutes, the corresponding warning method will be an indicator light.

[0087] Medium risk: 0.5 nautical miles ≤1 nautical mile and 10 minutes For intervals of ≤30 minutes, the corresponding warning method is a combination of sound and indicator light.

[0088] High risk: 0.1 nautical miles ≤0.5 nautical miles and 3 minutes For intervals of ≤10 minutes, the corresponding warning methods are pop-up window + sound + vibration.

[0089] Emergency Risk: ≤0.1 nautical miles or If the time is ≤3 minutes, the corresponding warning method is for the linkage propulsion system to issue a deceleration prompt.

[0090] The aforementioned thresholds are dynamically adjusted based on the ship's braking performance in high-speed navigation or narrow waterway scenarios, with an adjustment range not exceeding 30%. When an emergency risk warning is triggered, the system sends a deceleration request signal to the electric propulsion system via the Controller Area Network (CAN) bus. The propulsion system automatically executes deceleration operations based on the current motor status and battery SOC, and simultaneously feeds back the execution result to the warning module, forming a closed-loop control. The warning response time is ≤800ms.

[0091] (2) Energy efficiency-assisted decision-making: Based on the fusion of waterway information (such as water flow direction and shoal location) and ship energy consumption data, the optimal energy-efficient navigation path and speed range are recommended in real time through reinforcement learning algorithms.

[0092] a) Constructing the state space .

[0093] To enable the agent to fully perceive the navigation situation, a state vector is defined. It includes the following multi-dimensional quantitative indicators: real-time water flow vector, shoal boundary distance and recommended channel deviation extracted from the fusion situation of the channel features; dynamic constraints are the real-time ID, position, speed and estimated motion trend of all surrounding sensed and fused targets; power characteristics include motor torque output, system response delay coefficient, braking performance parameters and current heading; energy status is the real-time feedback of energy consumption data (current, voltage, instantaneous power) and battery remaining state of charge.

[0094] b) Action space Discretized definition.

[0095] The system outputs decision commands designed to recommend optimal navigation parameters and motion vectors in real time. Including expected heading correction: Recommended speed range: Based on the current energy consumption rate, the optimal power level or speed range of the electric propulsion system is recommended to match the current energy efficiency target.

[0096] c) Reward function.

[0097] Task progress rewards Used to guide ships to navigate efficiently along recommended routes, combining the position and target waypoint in four-dimensional data.

[0098] ; in, The real-time ground velocity is obtained from situational data. The angle between the ship's current course and the centerline of the recommended channel. This represents the current distance from the target point. The decision time step is typically set to 1 second.

[0099] Real-time energy efficiency benefits Power data acquired by sensors in new energy ships enables optimal energy consumption.

[0100] ; in, For the system in Real-time instantaneous power at any given moment This refers to the physical distance the ship traveled during that period. This is the energy efficiency sensitivity coefficient, typically taken as 0.8. This represents the instant when an agent observes the state of its environment and takes an action. This refers to the duration of a single action by the agent, i.e., the decision step size.

[0101] Dynamic safety constraint penalties The system constrains ship navigation based on the four-level risk level output by the dynamic risk warning module.

[0102] ; in, This is a warning and penalty item; This is a penalty term. If an energy efficiency path attempt exceeds a safe threshold, a high negative score penalty will be imposed, ensuring that energy efficiency optimization does not come at the expense of safety. (Finally defined...) The total reward at any given moment is a weighted sum of multiple evaluation dimensions.

[0103] 5. Visual interaction and data traceability module.

[0104] (1) Multi-mode visualization interface: Supports switching between electronic chart mode, video AR overlay mode, and data dashboard mode, and displays four-dimensional fusion situational data, risk level, and energy efficiency recommendations in real time. The interface uses color coding: green indicates safe / low energy consumption, yellow indicates medium risk, and red indicates high risk / high energy consumption, intuitively presenting key information.

[0105] (2) Full-process data traceability: All sensor raw data, fusion data, situation data, early warning records and energy consumption data are stored according to UTC timestamps for a storage time of ≥180 days; 1x / 2x / 4x / 8x speed playback is supported, which can accurately locate risk warning nodes and energy consumption mutation nodes, providing data support for fault diagnosis and algorithm optimization.

[0106] Example 2 This embodiment is basically the same as Embodiment 1, except that an alternative solution is adopted for the selection of the core sensing equipment to adapt to different navigation environments or cost requirements.

[0107] 1. Sensor replacement configuration.

[0108] (1) Navigation Radar: FMCW (Frequency Modulated Continuous Wave) radar is used to replace solid-state navigation radar. FMCW radar has the characteristics of low transmission power, no blind zone, and high range resolution. It operates at a frequency of 77 GHz, with a minimum detection range of 10 m and a maximum detection range of 20 nautical miles. It does not need to switch operating modes when detecting at close range, and is suitable for scenarios such as narrow inland waterways that require continuous high-precision ranging.

[0109] (2) Visual sensor: Thermal imaging cameras are used instead of multispectral network cameras. Thermal imaging cameras operate in the 8-14μm band and are not affected by lighting conditions. They can still provide clear images in extremely low visibility environments such as night, fog, rain, and snow, and can effectively identify thermal targets such as floating objects on the water surface and people who have fallen into the water. Thermal imaging cameras are also distributed in the four directions of the ship, front, rear, left, and right, to achieve 360° full coverage.

[0110] (3) Other sensors: AIS terminal, electronic compass, new energy-specific sensor are the same as in Example 1.

[0111] 2. Software adaptation and adjustment.

[0112] Since the thermal imaging camera outputs a single-channel thermal image, in the second-level fusion, the input layer of the YOLOv8 enhanced algorithm is adjusted to use convolutional kernels adapted to the single-channel image, and thermal imaging samples are added to the training data to maintain recognition accuracy. The remaining software modules are the same as in Example 1.

[0113] 3. Applicable scenarios.

[0114] This embodiment is particularly suitable for environments with extremely poor lighting conditions, such as night navigation, foggy operations, and polar waters, or for scenarios where it is necessary to identify people who have fallen into the water during search and rescue missions.

[0115] Example 3 This embodiment is basically the same as Embodiment 1, except that the core algorithm adopts an alternative solution to adapt to different computing power platforms or improve specific performance indicators.

[0116] 1. Algorithm-based configuration replacement.

[0117] (1) Target tracking algorithm: The DeepSORT algorithm is used instead of the BoT-SORT + Kalman filter joint tracking algorithm. DeepSORT reduces the computational resource consumption by about 30% while maintaining tracking accuracy through cascaded matching and Mahalanobis distance metric, making it suitable for small ships or edge computing devices with limited computing power. DeepSORT also combines Kalman filtering for motion state estimation to ensure tracking continuity.

[0118] (2) Trajectory Prediction Algorithm: An LSTM neural network is introduced for target trajectory prediction. The LSTM network predicts the target's position in the next 5 to 10 seconds based on the target's historical trajectory sequence (time step 10, 0.5s per step). Compared with traditional linear extrapolation or Kalman filter prediction, LSTM can learn the target's motion pattern, improving prediction accuracy by about 15% when the target maneuvers (such as turning or changing speed). The LSTM network is pre-trained offline using a large amount of AIS historical trajectory data and infers in a lightweight manner when running online.

[0119] (3) Risk Assessment Model: The fuzzy comprehensive evaluation method is adopted to replace the four-factor risk assessment model. The fuzzy comprehensive evaluation method uses factors such as DCPA, TCPA, ship response delay, and water flow impact as evaluation indicators, quantifies the risk level of each indicator through membership functions, and then calculates the risk level based on the weight vector. This method can more flexibly handle the problem of fuzzy indicator boundaries, and the core constraints (characteristics of new energy ships, real-time requirements) remain consistent with those in Example 1.

[0120] 2. Software module adjustments.

[0121] In the second-level fusion, the tracking module is replaced with DeepSORT; the motion trend prediction in the dynamic target situation is replaced with LSTM neural network output; and the calculation logic in the risk warning module is replaced with fuzzy comprehensive evaluation algorithm.

[0122] 3. Applicable scenarios.

[0123] This embodiment is applicable to small vessels with strict limitations on computing resources, or high-level security applications that require improved early warning accuracy in target maneuvering scenarios.

[0124] Example 4 This embodiment is basically the same as Embodiment 1, except that the processing hardware adopts an edge computing deployment scheme to adapt to the installation space and cost requirements of small ships.

[0125] 1. Hardware replacement configuration.

[0126] (1) Processing unit: The edge computing box replaces the high-performance processing server. The edge computing box uses NVIDIA Jetson AGX Orin, which integrates 2048 CUDA cores and 64 Tensor cores, with an AI computing power of up to 275 TOPS. It has a size of only 200mm×150mm×50mm and a power consumption that is adjustable from 15 to 60W, making it suitable for small new energy ships with limited space.

[0127] (2) Sensor simplification: For small vessels with a length of less than 20m, the sensor configuration can be simplified: The number of multispectral network cameras has been reduced from four to two (one in the front and one in the back), achieving 360° coverage through pan-tilt rotation. The enhanced AIS terminal is simplified to standard Class B AIS, eliminating the passive detection module; A water flow sensor is optional, and the energy efficiency auxiliary decision module optimizes based solely on energy consumption data when there is no water flow data.

[0128] 2. Software adaptation and adjustment.

[0129] Since the computing power of edge computing boxes is lower than that of server solutions, the software modules are optimized for lightweight design. The YOLOv8 enhanced algorithm has been replaced with the YOLOv5s lightweight version, which slightly reduces the recognition accuracy but increases the inference speed by 2 times. In the three-level fusion module, some calculations (such as long-term trajectory prediction) are downgraded to be executed every other frame, and the update frequency is adjusted from real-time to once every 0.5 seconds. The visualization interface uses electronic chart mode by default, and AR overlay mode can be manually turned on when needed.

[0130] 3. Applicable scenarios.

[0131] This embodiment is particularly suitable for cost-sensitive and space-limited scenarios such as small new energy vessels, short-distance inland river ferries, and scenic tour boats, achieving system miniaturization and low-cost deployment while ensuring core safety functions.

[0132] Example 5 This embodiment is a further combination of embodiments two, three, and four, and can be flexibly configured according to actual needs.

[0133] 1. Example of a combination (extreme environment + small vessel).

[0134] The sensor uses the FMCW radar + thermal imaging camera configuration of Embodiment 2, the processing hardware uses the edge computing box of Embodiment 4, and the algorithm uses the lightweight solution (DeepSORT + simplified LSTM) of Embodiment 3. This combination is suitable for small research vessels or work vessels navigating in polar or foggy areas.

[0135] 2. Example of combination 2 (low-cost inland waterway vessels).

[0136] The sensor setup is simplified to two cameras plus a standard AIS sensor. The processing hardware utilizes an edge computing box, and the algorithm employs YOLOv5s + DeepSORT. The LSTM trajectory prediction and energy efficiency-assisted decision-making modules have been removed, retaining only the core functions of three-level fusion and risk warning. This combination is suitable for short-distance inland waterway transport vessels, minimizing costs while ensuring basic safety.

[0137] 3. Software compatibility instructions.

[0138] Each combination scheme is implemented based on the core architecture of Implementation Example 1, with only the relevant modules replaced or simplified. Upon system startup, the corresponding sensor drivers, algorithm models, and functional modules are automatically loaded according to the configuration file, without requiring recompilation of the entire software.

[0139] It is worth noting that all contents not described in detail in this invention are existing technologies and are well known to those skilled in the art.

[0140] Therefore, the present invention adopts the above-mentioned new energy ship navigation situation perception system, method and medium, which can realize the efficient fusion of multi-source heterogeneous data and the accurate construction of navigation situation, effectively solving the problems of fragmented perception, inefficient fusion, separation of static and dynamic target perception and incompatibility between risk warning and the characteristics of new energy ships in the prior art, and improving the navigation safety and energy efficiency of new energy ships in complex waters.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A new energy ship navigation situational awareness system, characterized in that, include: A multi-source heterogeneous sensor array is used to collect multi-source data on the ship's navigation environment and its own status. The data preprocessing and spatiotemporal synchronization module is used to perform format standardization and spatiotemporal synchronization processing on multi-source data collected by multi-source heterogeneous sensor groups. The multi-source data deep fusion module is used to perform three-level progressive fusion of spatiotemporally synchronized multi-source data. The three-level progressive fusion includes: complementary fusion of homogeneous data, enhanced fusion of heterogeneous data, and situational correlation fusion, generating four-dimensional fused data containing environmental, target, ship self, and energy consumption information. The static and dynamic integrated situation building module is used to build an integrated navigation situation based on four-dimensional fusion data, which includes dynamic target situation, static target situation and the ship's own situation. The risk warning and energy efficiency auxiliary decision-making module is used to conduct dynamic risk warning based on the integrated navigation situation and the characteristics of the electric propulsion system of new energy ships, and to provide energy efficiency optimization decisions based on the correlation between sensing data and energy consumption data.

2. The new energy ship navigation situational awareness system according to claim 1, characterized in that, The multi-source heterogeneous sensor group includes at least: solid-state navigation radar, multispectral network camera, enhanced AIS terminal, marine satellite electronic compass, and new energy-specific sensors for collecting status and energy consumption data of electric propulsion systems; Among them, the enhanced AIS terminal adds a passive detection module. The passive detection module monitors the energy distribution within the working frequency band through real-time fast Fourier transform, calculates the phase difference of the same signal arriving at different antenna array elements using a multi-signal classification algorithm to obtain the radio azimuth line, superimposes the radio azimuth line onto the radar point cloud map, marks targets with radar echoes but no AIS messages as suspected targets without AIS enabled and assigns them a passive signal feature ID, and uses the recursive least squares method to perform cross-intersection calculations on multiple azimuth lines to solve for the target's geographical coordinates.

3. The new energy ship navigation situational awareness system according to claim 1, characterized in that, The data preprocessing and spatiotemporal synchronization module is specifically used for: Design a unified data encapsulation protocol to normalize sensor data of different formats; Time stamp alignment of multi-sensor data is achieved based on satellite timing, with a synchronization error of ≤5ms; Spatial coordinate transformation algorithms are used to map spatial data from different sensors to a unified coordinate system. Spatial coordinate transformation includes radar polar coordinates to rectangular coordinates, AIS geodetic coordinates to geocentric and earth-fixed rectangular coordinates and then to northeast-sky local coordinates, and video pixel coordinates based on the pinhole imaging model are back-projected to sea level coordinates. Adaptive threshold filtering and inter-frame differencing are used to clean and reduce noise in the data.

4. The new energy ship navigation situational awareness system according to claim 1, characterized in that, The multi-source data deep fusion module includes: The first-level fusion unit is used to match and correlate dynamic targets detected by radar with ship information received by AIS. It adopts an improved Kalman filter algorithm and introduces a target motion trend prediction factor. : ； in, For the normalized sum of squared residuals, The mobility threshold; The second-level fusion unit is used to identify targets in the video through target recognition algorithms and generate video target trajectories in combination with tracking algorithms. It spatially aligns the initial fused targets with the video targets and enhances the attribute information of the fused targets using video visual features. The target recognition algorithm adopts the YOLOv8 enhanced version algorithm, which introduces the P2 small target detection head, coordinate attention mechanism, and replaces the loss function with Wise-IoUv3. The tracking algorithm adopts BoT-SORT combined with Kalman filtering algorithm. The third-level fusion unit is used to spatiotemporally correlate the enhanced fusion target output by the second-level fusion unit with static target data obtained from electronic charts, sudden static obstacle data output in real time by the radar and video joint identification module, and electric propulsion status data and energy consumption data obtained from new energy-specific sensors, to construct four-dimensional fusion data that includes environmental, target, ship self, and energy consumption information.

5. A new energy ship navigation situational awareness system according to claim 4, characterized in that, The first-level fusion unit is also used for: for targets without AIS enabled, using the stabilized trajectory of the prediction factors to inversely calculate their physical scale and ship type through a multi-frame smoothing algorithm; and establishing a comprehensive confidence scoring model. ,in Continuous tracking duration for the target The mean motion deviation, , This is a weighting coefficient; when the score exceeds a preset threshold, a temporary ID is assigned and the score is included in the obstacle avoidance monitoring system.

6. The new energy ship navigation situational awareness system according to claim 1, characterized in that, The static and dynamic integrated situational awareness construction module is specifically used for: Output dynamic target situation, including target ID, real-time position, speed, heading, type, motion trend and confidence level; Output static target situation, including sudden static obstacles identified by electronic chart overlay radar and video, and update static target data in real time; Output the ship's own status, including compass attitude data, electric propulsion system status, current energy consumption, and estimated remaining range.

7. The new energy ship navigation situational awareness system according to claim 1, characterized in that, The risk warning and energy efficiency decision support module includes: The dynamic risk warning unit is used to construct a risk assessment model based on four factors: distance, speed, ship characteristics, and environment. It introduces the response delay coefficient and braking performance parameters of the electric propulsion system of new energy ships to dynamically adjust the warning threshold and classifies collision risks into different levels corresponding to different warning methods. The energy efficiency auxiliary decision-making unit is used to recommend the optimal energy-efficient navigation path and speed range in real time based on the fused channel information and energy consumption data through reinforcement learning algorithms, and to mark the energy consumption increment corresponding to different operations.

8. A new energy ship navigation situational awareness system according to claim 1, characterized in that, It also includes a visualization and data traceability module, used for: It provides a variety of visualization interfaces to display the integration status, risk level and energy efficiency recommendations in real time, and uses color coding for intuitive presentation; All raw data, fused data, situational data, early warning records, and energy consumption data are stored according to timestamps, and data playback and traceability are supported.

9. A method for situational awareness of new energy vessels, characterized in that, The system applied to any one of claims 1 to 8 includes the following steps: Multi-source data on the ship's navigation environment and its own status are collected through a multi-source heterogeneous sensor array. The collected multi-source data is standardized in format and spatiotemporally synchronized. The spatiotemporally synchronized multi-source data is fused in a three-level progressive manner to generate four-dimensional fused data that includes information on the environment, target, ship itself, and energy consumption. An integrated navigation situation is constructed based on four-dimensional fusion data, which includes dynamic target situation, static target situation and the ship's own situation; Based on the integrated navigation situation, dynamic risk warnings are carried out in combination with the characteristics of the electric propulsion system of new energy ships, and energy efficiency optimization decisions are provided based on the correlation between sensing data and energy consumption data.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in claim 9.

Images