Ship navigation situation sensing system and sensing method based on panoramic video construction

Abstract

The invention provides a ship navigation situation sensing system and sensing method based on panoramic video construction, and belongs to the technical field of image processing, and a control module collects data through navigation equipment, environment measurement equipment and sensing equipment. The control module preprocesses the collected data. The control module performs feature extraction on the data and detects a static obstacle and a dynamic target. The control module performs association fusion on the static obstacle and the dynamic target through a multi-source data fusion algorithm; and the control module displays the static obstacle and the dynamic target through the display module. The control module evaluates the sailing risk of the ship according to the sailing state of the ship, the position of the static obstacle and the situation of the dynamic target; when the control module predicts danger, the control module gives an alarm through ship alarm equipment, provides adjustment suggestions and records dangerous events at the same time. According to the invention, comprehensive and accurate surrounding environment information is provided for a ship driver, and the safety of ship navigation is improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image processing, and in particular, the present application relates to a ship navigation situation awareness system and method based on panoramic video construction. BACKGROUND

[0002] With the rapid development of global trade and shipping industry, the number of ships has increased significantly, which puts higher requirements on ship navigation safety. The traditional ship navigation system mainly relies on radar, AIS (Automatic Identification System), electronic chart and other devices to provide information, but these systems have their own limitations and are difficult to provide comprehensive and intuitive environmental awareness in complex environments.

[0003] Chinese patent 106816038A discloses an inland water area abnormal behavior ship automatic identification system and method, which acquires AIS message information of inland water area ship navigation, environmental information of hydrological and meteorological department, video image and depth image in real time, analyzes ship abnormal behavior mode category, establishes ship abnormal behavior sample library; establishes a deep learning network model to analyze ship behavior, obtains ship abnormal behavior mode and GPS positioning information; detects the ship in the video image, obtains the three-dimensional spatial information of the ship combined with the depth image, and obtains the video positioning information of the ship; fuse the GPS positioning information, video positioning information, ship abnormal behavior mode and ship detection feature, and perform ship target correlation to automatically identify abnormal behavior ships on the video.

[0004] The prior art does not comprehensively detect the environment around the ship, which affects the safety of ship navigation. SUMMARY

[0005] The present application aims at a ship navigation situation awareness system and method based on panoramic video construction, to provide comprehensive and accurate surrounding environment information for ship drivers and improve the safety of navigation.

[0006] The present application aims to provide a ship navigation situation awareness technology based on panoramic video construction, which can solve the above-mentioned problems existing in the prior art, provide comprehensive, accurate and intuitive environmental information for ship drivers, and improve the safety and efficiency of navigation in adverse weather conditions.

[0007] In order to achieve the above-mentioned purpose, the technical scheme adopted by the present application is as follows:

[0008] The present application provides a ship navigation situation awareness system based on panoramic video construction, which comprises a collection module, a control module and a display module, the output end of the collection module is connected to the input end of the control module, and the output end of the control module is connected to the input end of the display module.

[0009] The collection module comprises a navigation device, an environment measuring device and a perception device, the navigation device comprises an electronic chart display and information system, a global navigation satellite system and an inertial navigation system; the environment measuring device comprises a depth sounder, a weather station and a radar system; the perception device comprises a panoramic video collection system, an infrared / thermal imaging camera, an AIS receiver and a laser radar.

[0010] The control module adopts an industrial computer.

[0011] The display module comprises a multifunctional display screen and a VR glasses.

[0012] The application provides a perception method of a ship navigation situation awareness system constructed based on panoramic video:

[0013] Step one: the control module collects data through the navigation device, the environment measuring device and the perception device;

[0014] Step two: the control module pre-processes the collected data;

[0015] Step three: the control module extracts features of the data and detects static obstacles and dynamic targets;

[0016] Step four: the control module respectively correlates and fuses the static obstacles and the dynamic targets through a multi-source data fusion algorithm;

[0017] Step five: the control module displays the static obstacles and the dynamic targets through the display module;

[0018] Step six: the control module evaluates the ship navigation risk according to the ship navigation state, the position of the static obstacles and the situation of the dynamic targets; when the control module predicts a danger, the control module alarms through a ship alarm device and provides adjustment suggestions, and records the dangerous event.

[0019] In step one, the control module receives the electronic chart data provided by the electronic chart display and information system; the control module collects the water depth data below the ship through the depth sounder; the control module collects the weather data of the environment where the ship is located through the weather station; the control module detects the long-distance target through the radar system; the control module collects the video data of the environment around the ship through the high-definition camera in the panoramic video collection system; the control module collects the thermal image data through the infrared / thermal imaging camera; the control module receives the AIS signal of the surrounding ships through the AIS receiver; and the control module collects the position data of the near-distance obstacles through the laser radar.

[0020] In step two, the control module performs clutter suppression and target detection on radar data collected by the radar system and lidar; the control module performs distortion correction and image enhancement processing on video data collected by the panoramic video acquisition system; the control module analyzes and verifies the AIS signals of the surrounding ships collected.

[0021] In step three, the control module extracts static obstacle information from the electronic chart data; the control module detects and tracks dynamic targets in the video data through a target detection model; the control module detects static obstacles and dynamic targets through infrared and visible light image fusion algorithm; the control module detects static or moving ships through the AIS receiver.

[0022] In step four, the control module correlates and fuses the static obstacles and dynamic targets extracted by different devices in the collection module through a multi-source data fusion algorithm.

[0023] In step six, the control module predicts the navigation state of the ship through a deep learning model, combines the position of the static obstacle and the situation of the dynamic target, and evaluates the navigation risk of the ship through a ship collision risk calculation model.

[0024] The technical effects of the present application are:

[0025] (1) The present application improves the perception ability of the ship's surrounding environment, provides 360-degree omnidirectional environmental perception through panoramic video acquisition system and multi-collection device data fusion technology. Fusing visible light and infrared images significantly improves the environmental perception ability in night and low visibility conditions. Expanding the range of the driver's vision, effectively identifying the targets missed by AIS and radar system.

[0026] (2) The present application is more accurate in detecting and tracking dynamic targets, and the improved YOLOv8-SGW model significantly improves the detection accuracy and speed of marine targets. Multi-source data fusion technology improves the accuracy of target position and attribute. It can accurately detect and track long-distance small targets such as buoys and small ships.

[0027] (3) The present application intuitively enhances the reality navigation, which intuitively superimposes virtual information on the real scene, reduces the cognitive burden of the driver. Provides immersive VR navigation experience, improves the intuitiveness and efficiency of environmental perception. Supports multi-view observation, adapts to different driving scenarios and needs.

[0028] (4) The present application can provide intelligent analysis of environmental situation and collision warning, real-time analysis of ship navigation state and surrounding environmental situation, accurate calculation of ship collision risk, and timely warning and decision-making suggestions.

[0029] (5) The application improves the safety of ship navigation, reduces ship collision accidents caused by visual blind area, and ensures the perception of the environment of the ship under bad weather conditions. The application has a wide range of application scenarios and is suitable for various types of ships, including cargo ships, passenger ships, tugboats, unmanned ships, etc., and can be applied to various navigation environments such as ports, inland rivers, and offshore. It supports future intelligent upgrades, such as autonomous navigation, remote driving, etc. BRIEF DESCRIPTION OF DRAWINGS

[0030] The present specification includes the following drawings, the contents of which are shown as follows:

[0031] Figure 1 A flowchart of a ship navigation situation awareness system and a perception method based on panoramic video construction of the present application. DETAILED DESCRIPTION

[0032] The specific embodiments of the present application will be further described below with reference to the drawings and by describing the embodiments, the purpose is to help those skilled in the art to have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present application, and to help its implementation.

[0033] The present application provides a ship navigation situation awareness system based on panoramic video construction, comprising a collection module, a control module and a display module, the output end of the collection module is connected to the input end of the control module, and the output end of the control module is connected to the input end of the display module.

[0034] The collection module includes a navigation device, an environment measuring device and a perception device, the navigation device includes an electronic chart display and information system, a global navigation satellite system and an inertial navigation system; the environment measuring device includes a depth sounder, a weather station and a radar system; the perception device includes a panoramic video acquisition system, an infrared / thermal imaging camera, an AIS receiver and a laser radar. The control module uses an industrial computer. The display module includes a multifunctional display screen and a VR glasses.

[0035] The present application provides a perception method of a ship navigation situation awareness system based on panoramic video construction:

[0036] Step one: the control module collects data through the navigation device, the environment measuring device and the perception device;

[0037] Step two: the control module pre-processes the collected data;

[0038] Step three: the control module extracts features from the data and detects static obstacles and dynamic targets;

[0039] Step four: the control module respectively correlates and fuses the static obstacles and dynamic targets through a multi-source data fusion algorithm;

[0040] Step five: the control module displays static obstacles and dynamic targets through the display module;

[0041] Step six: the control module evaluates the navigation risk of the ship according to the navigation state of the ship, the position of the static obstacle and the situation of the dynamic target; when the control module predicts a danger, the control module alarms through the ship alarm device and provides adjustment suggestions, and records the dangerous event at the same time.

[0042] In step one, the control module receives the electronic chart data provided by the electronic chart display and information system; the control module collects the water depth data below the ship through the depth sounder; the control module collects the weather data of the environment where the ship is located through the weather station; the control module detects long-distance targets through the radar system; the control module collects video data of the environment around the ship through the high-definition camera in the panoramic video acquisition system; the control module collects thermal image data through the infrared / thermal imaging camera; the control module receives the AIS signals of the surrounding ships through the AIS receiver; the control module collects the position data of the near-distance obstacles through the laser radar.

[0043] In step two, the control module performs clutter suppression and target detection on the radar data collected by the radar system and the laser radar; the control module performs distortion correction and image enhancement processing on the video data collected by the panoramic video acquisition system; the control module analyzes and verifies the AIS signals of the surrounding ships.

[0044] In step three, the control module extracts static obstacle information from the electronic chart data; the control module detects and tracks dynamic targets in the video data through the target detection model; the control module detects heat-emitting static obstacles and dynamic targets through the infrared and visible light image fusion algorithm; the control module detects static obstacles and dynamic targets through the radar system and the laser radar; the control module detects static or moving ships through the AIS receiver.

[0045] In step four, the control module respectively correlates and fuses the static obstacles and dynamic targets extracted by different devices in the acquisition module through the multi-source data fusion algorithm.

[0046] In step six, the control module predicts the navigation state of the ship through the deep learning model, combines the position of the static obstacle and the situation of the dynamic target, and evaluates the navigation risk of the ship through the ship collision risk calculation model.

[0047] The application discloses a ship navigation situation awareness system based on panoramic video construction, which comprises a collection module, a control module and a display module, wherein the collection module comprises a navigation device, an environment measuring device and a perception device, the navigation device comprises an electronic chart display and information system, a global navigation satellite system and an inertial navigation system. The electronic chart display and information system (ECDIS) provides navigation information and an electronic chart, and the electronic chart displays static obstacle information including shoal, reef, sunken ship and the like. The electronic chart display and information system adopts an electronic chart system conforming to the latest IHO S-100 standard, thereby ensuring real-time information. The global navigation satellite system (GNSS) is used for receiving Beidou and GPS satellite signals and providing accurate position and time information. The inertial navigation system (INS) is used for providing continuous navigation information when satellite signals are lost.

[0048] The environment measuring device comprises a depth finder, a weather station and a radar system. The depth finder is used for measuring the water depth below the ship and providing a shallow water early warning. The weather station is used for measuring weather parameters such as wind speed, wind direction, air pressure and temperature. The radar system is used for detecting long-distance targets, especially in bad weather conditions. The radar system is an important equipment for realizing long-distance target detection, especially in bad weather, can penetrate obstacles such as rain, fog and darkness, and obtain information such as distance, direction and speed of the target. The application adopts a combination of X-band and S-band dual-band radars, the X-band radar has high angle resolution and distance resolution, and is suitable for near-distance and high-precision target detection; the S-band radar has strong penetration ability and long detection distance, and is suitable for long-distance target search and tracking. The radar antenna adopts phased array antenna technology, can realize rapid and flexible beam scanning, and can detect a large range area in a short time. When the radar system works, the transmitter generates high-frequency electromagnetic wave signals, which are radiated to space through the antenna. When the electromagnetic wave meets the target, it will be reflected, and the reflected signal is received by the radar antenna, and is amplified, mixed, filtered and processed by the receiver, and then the echo signal of the target is extracted. The time difference between the echo signal and the transmitted signal is used to calculate the distance of the target; the direction of the target is determined by the direction of the antenna beam; the Doppler effect principle is used to analyze the frequency change of the echo signal, and the relative speed of the target is calculated. The obtained target distance, direction and speed information are converted in format and encoded by a data processing unit, and then transmitted to the multi-source data fusion module of the system for fusion analysis with other data.

[0049] The perception device includes a panoramic video acquisition system, an infrared / thermal imaging camera, an AIS receiver, and a laser radar. The panoramic video acquisition system is composed of multiple high-definition cameras arranged in a ring around the ship's bridge and acquires 360-degree panoramic video. The panoramic video acquisition system, as the core hardware of the application, is composed of multiple high-definition cameras, and its performance directly affects the perception effect of the system on the environment around the ship. In the selection of equipment, industrial-grade cameras with high resolution, wide dynamic range, and IP67 protection level are selected. High resolution ensures that the details of the collected images are clear, facilitating subsequent target detection and recognition; wide dynamic range allows the camera to clearly capture the picture under complex lighting conditions such as strong light, backlight, or low light; IP67 protection level ensures that the camera works stably in harsh environments such as sea moisture, salt fog, and rain. The number of cameras is reasonably configured according to the size of the ship and the monitoring requirements, usually 6-8 are arranged uniformly in a ring around the ship's bridge, ensuring that 360-degree panoramic video can be collected in all directions without dead angles, covering all directions where targets may appear during ship navigation. In terms of installation layout, the high-definition cameras should be installed at a height higher than the ship's deck by a certain distance to avoid obstruction of the view by the ship's structure, cargo, etc., while ensuring that the view of the high-definition camera covers the area above the ship's waterline and a certain range around it. During installation, each high-definition camera is accurately calibrated and fixed to ensure that the field of view between adjacent high-definition cameras has a certain overlapping area, providing the necessary conditions for subsequent image stitching. The working principle is that each high-definition camera collects video data of the environment around the ship in real time, converts the optical signal to an electrical signal, and converts the analog video signal to a digital video signal through digital signal processing technology. In the collected raw video data, there may be lens distortion and other problems, so the video needs to be corrected for distortion first. A lens distortion correction algorithm based on Zhang Zhengyou's calibration method is used to calibrate the camera and obtain the lens distortion parameters, then the coordinates of each pixel point in the video image are transformed to restore the true geometric shape of the scene. After distortion correction, the video image is geometrically transformed to convert the images collected by different cameras to the same coordinate system, preparing for image stitching. Finally, the SSIM-EW algorithm is used for image stitching, which is based on the structural similarity (SSIM) metric, and assigns adaptive weights to different images to generate seamless and clear panoramic video, providing basic visual information for subsequent dynamic target detection, multi-source data fusion, and dynamic target situation analysis of the system.

[0050] The infrared / thermal imaging camera provides clear images at night and in low-visibility conditions. The infrared / thermal imaging camera is used to capture thermal radiation information of the environment around the ship in low-visibility weather conditions such as at night, heavy fog, heavy rain, etc., to generate thermal images, and to make up for the insufficient performance of visible light cameras in such environments. A long-wave infrared thermal imaging camera with high sensitivity and high resolution is selected, which has a detection wavelength range of 8-14 pm, and can effectively detect small temperature differences. Even in complete darkness or adverse weather conditions, the thermal profile of the surrounding target can be clearly presented. The camera is equipped with an electric zoom lens, which can adjust the observation distance and range according to actual needs, meeting the detection requirements of targets at different distances. When installing, it is reasonable to arrange the visible light camera in the panoramic video acquisition system and the infrared / thermal imaging camera to ensure that their field of view ranges are basically the same, which is convenient for subsequent infrared and visible light image fusion processing. When working, the camera receives the infrared radiation emitted by the target object through the infrared detector, and after focusing by the optical system, an electrical signal is generated on the photosensitive element of the detector. After signal amplification, filtering, and analog-to-digital conversion, the electrical signal is converted into digital thermal image data, which is transmitted to the image processing unit of the system for further analysis and processing.

[0051] The AIS receiver receives the AIS signals of surrounding ships and obtains static and dynamic information of the ships. The AIS (Automatic Identification System) receiver is used to receive the AIS signals of surrounding ships and obtain their static and dynamic information, including the name, type, size, heading, speed, position, and navigation state of the ship. An AIS receiver that meets the relevant standards of the International Maritime Organization (IMO) is selected, which has a dual-channel receiving function and can simultaneously receive AIS signals on VHF 87B (161.975 MHz) and 88B (162.025 MHz) channels, ensuring the completeness and reliability of information reception. The AIS receiver receives the AIS signals emitted by surrounding ships through an antenna, and after filtering, amplification, and frequency conversion by the radio frequency front end, the signals are converted into intermediate frequency signals, which are then demodulated and decoded by the digital signal processing module to extract various information data of the ship. After verification and format conversion, the obtained data is transmitted to the multi-source data fusion module of the system, associated with and matched with other sensor data such as radar and video, and fused to enrich the situational awareness information of the system for surrounding ships, providing an important basis for ship collision avoidance decision-making and navigation situation analysis.

[0052] The laser radar (LiDAR) is used to accurately detect obstacles, especially near-range obstacles.

[0053] The control module adopts an industrial computer. Each device is converted into a network signal through a protocol converter, and then the data is transmitted to the industrial computer by an Ethernet switch. The industrial computer processes the data, detects static obstacles and dynamic targets, and superimposes them into a panoramic video. Meanwhile, the industrial computer performs navigation risk assessment on the ship, and issues an alarm through an alarm device when a danger is predicted. The model of the industrial computer in the embodiment of the application is Yanchang P17B.

[0054] The display module includes a multifunctional display screen and a VR glasses. The control module displays the panoramic video after multi-source data fusion through the VR glasses. The multifunctional display screen is used to display an electronic chart, water depth data under the ship, meteorological data of the environment where the ship is located, a radar image, a panoramic video, a thermal image, and position information and navigation conditions of surrounding ships.

[0055] The application provides a ship navigation situation awareness method based on a panoramic video.

[0056] The control module collects data through navigation devices, environment measuring devices and sensing devices. The control module receives electronic chart data provided by an electronic chart display and information system. The control module collects water depth data under the ship through a depth finder. The control module collects meteorological data of the environment where the ship is located through a meteorological station. The control module detects long-distance targets through a radar system. The control module collects video data of the environment around the ship through a high-definition camera in a panoramic video collection system. The control module collects thermal image data through an infrared / thermal imaging camera. The control module receives AIS signals of surrounding ships through an AIS receiver. The control module collects position data of near-distance obstacles through a laser radar. The operating system of the industrial computer in the application is a Linux operating system with high stability and strong real-time performance, which provides reliable bottom environment support for various software applications and algorithm running of the system.

[0057] The control module pre-processes the collected data. The control module performs clutter suppression and target detection on radar data collected by the radar system and the laser radar. The control module performs distortion correction and image enhancement processing on video data collected by the panoramic video collection system. The control module analyzes and verifies AIS signals of surrounding ships.

[0058] The control module extracts features from the data and detects static obstacles and dynamic targets. Specifically, in the implementation of the static obstacle extraction algorithm based on electronic chart data, first, from the electronic chart data conforming to the IHO S-100 standard, the geometric coordinates, attributes, and other information of static navigation hazards such as shoals, reefs, and wrecks are extracted using geographic information system (GIS) related techniques. The electronic chart area is divided into different levels of sub-regions using quadtree spatial indexing technology, a spatial indexing structure is constructed, and the retrieval efficiency of static obstacle information is improved. Then, according to the extracted static obstacle geometric information, a three-dimensional model of the static obstacle is established using a three-dimensional modeling software (such as Blender) or related three-dimensional modeling algorithms, and appropriate materials and textures are assigned, which are used for subsequent augmented reality display, enabling the driver to more intuitively understand the distribution and morphology of the surrounding static obstacles.

[0059] The control module detects static obstacles and dynamic targets in the video data through a target detection model. The present application improves the target detection model (YOLOv8). The size of the detection head in the target detection algorithm is increased to improve the detection capability of small targets. The GhostNet network structure is introduced to reduce the computational load and parameter quantity of the model without reducing the detection accuracy of the model, improve the detection speed, and replace the traditional regression loss function with the WIoU loss function to enhance the accuracy of the model in positioning dynamic targets. The improved model is trained by a large amount of marine target image data (including ships, floating objects, navigation marks, etc.), and the data needs to be labeled, including the category, position, size, etc. of the dynamic target. During the training process, random data augmentation techniques (such as random cropping, flipping, scaling, adding noise, etc.) are used to expand the data set and improve the generalization ability of the model. When the target detection model (YOLOv8) detects dynamic targets, the video images captured by the high-definition camera in real time are input into the trained model, and the model outputs the detected dynamic target category, position, and confidence information. In the dynamic target tracking stage, combined with the target tracking algorithm (DeepSort), the target features (such as appearance features, position information, etc.) in the target detection results are used to associate and track the targets in consecutive video frames, handle the target occlusion and disappearance problem, and realize continuous monitoring of dynamic targets.

[0060] The control module detects the static obstacles and dynamic targets by infrared and visible light image fusion algorithm. In the implementation process of the visible light image fusion algorithm (APFusion), a multi-level adaptive perception loss function is first designed. Based on the shallow information (such as pixel intensity, edge, etc.) and deep features (such as semantic information, target structure, etc.) of the image, the function extracts features from the infrared and visible light images through a convolutional neural network (CNN) to obtain feature maps of different scales. According to the information entropy, contrast, and other indicators of the feature maps, the adaptive weight is assigned to the original image, so that the algorithm can adaptively fuse the information of the infrared and visible light images according to the different characteristics of the images. The designed loss function is used to train the feature guidance network, and the network structure adopts an encoder-decoder architecture. The encoder extracts and encodes the features of the input infrared and visible light images, and the decoder decodes the fused feature maps into a fused image. During the training process, the network parameters are constantly adjusted to make the fused image optimal in terms of visual information fidelity, mutual information, difference correlation coefficient, and spatial frequency. After the training is completed, the real-time collected infrared and visible light images are input into the trained network to obtain the fused image, thereby improving the detection and recognition ability of the static obstacles and dynamic targets in adverse weather conditions. The visible light image is the video image collected by the panoramic video acquisition system.

[0061] The control module detects the static obstacles and dynamic targets by radar system and laser radar.

[0062] The control module respectively associates and fuses the static obstacles and the dynamic targets through a multi-source data fusion algorithm. In the data fusion, the initial collection time of the data needs to be synchronized, and the application adopts a synchronization method based on a time stamp to record and compare the collection times of different devices in the collection module, and through interpolation or resampling and the like, adjusts the data to the same time reference. In the data fusion, spatial registration is also needed, and a three-dimensional coordinate conversion algorithm based on a seven-parameter conversion model is used to unify the coordinate systems of different devices in the collection module to the ship coordinate system. In the association process of the static obstacles or the dynamic targets, a fuzzy correlation matrix calculation method based on fuzzy mathematics is adopted to calculate the similarity between the targets detected by different devices, to construct a fuzzy correlation matrix, and to determine the association relationship between the targets according to the similarity values in the matrix. In the data fusion stage, the convex combination and covariance cross fusion algorithm is combined to fuse the associated target data, to update the position, speed, attribute and the like of the target, and to improve the accuracy and reliability of the target state estimation. The multi-source data fusion algorithm of the application is based on an asynchronous communication mechanism of a message queue, uses the Protobuf protocol to serialize and deserialize the data, reduces the data transmission amount, and improves the data transmission efficiency. In the implementation of the data fusion algorithm, the previous multi-source data fusion algorithms, including the fuzzy correlation matrix calculation method based on fuzzy mathematics, the convex combination and covariance cross fusion algorithm and the like, are integrated, the data collected by different devices are time-synchronized, spatially registered and fused, and accurate data basis is provided for subsequent navigation situation analysis and decision support. The application detects the static obstacles and the motion targets through different devices, the same static obstacle or motion target can be detected multiple times or only once, the redundant static obstacle and motion target detection mechanism improves the completeness of the detection results, and improves the safety of the ship.

[0063] The control module displays static obstacles and dynamic targets through the display module. Specifically, the control module displays the static obstacles and dynamic targets after multi-source data fusion through the VR glasses and superimposes them into the panoramic video collected by the panoramic video collection system, so that the driver can more intuitively observe the environment around the ship through the VR glasses, and the control module superimposes the ship's route information into the panoramic video, so that the driver can more comprehensively grasp the navigation state of the ship. At the same time, the control module displays the electronic chart, the water depth data below the ship, the meteorological data of the environment in which the ship is located, the radar image, the panoramic video, the thermal image, and the position information and navigation state of the surrounding ships through the multifunctional display screen. In the present application, the construction and display of the panoramic video are developed based on the OpenCV and OpenGL libraries. The OpenCV library provides rich image processing and computer vision algorithms for realizing the functions of video stream collection, distortion correction, image stitching, etc.; the OpenGL library is used to realize the efficient rendering and display of the panoramic video, supports multi-view switching, video scaling, rotation, and interactive operations such as zooming in on the region of interest. The video data of each camera is acquired in real time, image processing and display control are sequentially performed according to the pre-set process, and an intuitive and clear panoramic view of the environment around the ship is provided for the driver. The image display in the VR glasses is developed based on the Unity 3D game engine, combined with augmented reality development platforms such as ARCore or ARKit, to realize precise superposition and immersive display of virtual information and real scenes. Through three-dimensional modeling and scene rendering of the ship, the route, static obstacles and dynamic targets are displayed on the panoramic video in an intuitive and easy-to-understand manner, enhancing the driver's perception and understanding of the navigation environment.

[0064] The control module evaluates the navigation risk of the ship according to the navigation state of the ship, the position of the static obstacle and the situation of the dynamic target; when the control module predicts a danger, the control module alarms through the ship alarm device and provides adjustment suggestions, and records the dangerous event. Specifically, the navigation state of the ship is predicted through a deep learning model, and the LSTM or Transformer model is selected as the prediction model. Before the model is used, the historical navigation data of the ship is collected, including the position, heading, speed, time and other information of the ship, as well as the surrounding environment data (such as weather conditions, current information, etc.), and the data is preprocessed, such as cleaning and normalization. The preprocessed data is divided into training set, validation set and test set, the training set is used to train the model, in the training process, the parameters of the model (such as learning rate, number of hidden layer neurons, etc.) are adjusted to optimize the prediction performance of the model, the validation set is used to evaluate and adjust the parameters of the model to prevent overfitting. After training, the test set is used to test the generalization ability of the model. In actual application, the state information (position, heading, speed, time) of the current ship and the environmental data (weather conditions, current information) are input into the trained model, and the model outputs the position of the ship and the prediction result of the surrounding situation change in the future period of time, providing forward-looking information support for the navigation decision of the ship.

[0065] The evaluation of the navigation risk of the ship is realized by a ship collision danger degree calculation model, first, the factors affecting the ship collision danger degree are determined, including the position of the static obstacle or the distance, direction, relative speed and motion change of the dynamic target, and these factors are used as the input parameters of the model. The input parameters are fuzzified, different fuzzy sets (such as near, medium, far; small angle, medium angle, large angle; low speed, medium speed, high speed, etc.) are divided according to the value range and actual meaning of the parameters, and the membership function is defined for each fuzzy set, and the accurate input parameters are converted into membership values in the fuzzy set. A fuzzy rule base based on expert experience is established, for example, "if the dynamic target is close and the relative speed is fast and the motion change is large, the collision danger degree is high". In the fuzzy reasoning process, according to the fuzzy membership values of the input parameters and the fuzzy rule base, the Mamdani reasoning method or Larsen reasoning method is used for reasoning, and the fuzzy result of the collision danger degree is obtained. Finally, through the de-fuzzification operation, the fuzzy result is converted into a clear danger degree level (such as low, medium, high), which is used for collision warning and auxiliary decision. When a dangerous situation is detected, an alarm is issued through a loudspeaker, an alarm lamp or a display screen icon, and according to the current situation and the prediction result, an avoidance suggestion and a route adjustment suggestion are provided. The control module records and stores key events and data for post-analysis and system optimization.

[0066] The navigation auxiliary decision support software module is developed by combining expert systems with machine learning. An expert knowledge base in the field of ship navigation is established, containing knowledge such as collision avoidance rules and operating experience in various navigation scenarios. At the same time, historical navigation data is used to train the machine learning model, so that it can automatically generate collision avoidance suggestions and route adjustment suggestions based on the current navigation situation and prediction results. Through risk assessment and effect analysis of different decision schemes, the optimal decision reference is provided for the driver, and the navigation data is recorded and played back, which is convenient for post-analysis and system optimization, as well as driver training and learning.

[0067] The beneficial effects of the present application are described in detail below.

[0068] The present application improves the perception ability of the surrounding environment of the ship, provides 360-degree dead angle-free environmental perception through the panoramic video acquisition system and multi-acquisition device data fusion technology. The visible light and infrared images are fused, which significantly improves the environmental perception ability in night and low visibility conditions. The driver's field of view is expanded, and the targets missed by AIS and radar systems are effectively identified.

[0069] The present application is more accurate in detecting and tracking dynamic targets, and the improved YOLOv8-SGW model significantly improves the detection accuracy and speed of marine targets. The multi-source data fusion technology improves the accuracy of target position and attributes. It can accurately detect and track small targets at a long distance, such as buoys and small ships.

[0070] The present application intuitively enhances the reality navigation, which intuitively superimposes virtual information on the real scene, reducing the cognitive burden of the driver. It provides an immersive VR navigation experience, improving the intuitiveness and efficiency of environmental perception. It supports multi-view observation, adapting to different driving scenarios and needs.

[0071] The present application can provide intelligent analysis of environmental situation and collision warning, real-time analysis of ship navigation state and surrounding environmental situation, accurate calculation of ship collision risk, and timely warning and decision-making suggestions.

[0072] The present application improves the safety of ship navigation, reduces ship collision accidents caused by visual dead zones, and also ensures the perception of the environment of the ship in bad weather conditions.

[0073] The application scenario of the present application is wide, and it is suitable for various types of ships, including cargo ships, passenger ships, tugboats, unmanned ships, etc., and can be applied to port, inland, offshore and other navigation environments.

[0074] Supports future intelligent upgrades, such as autonomous navigation, remote driving, etc.

[0075] The application is described above with reference to the drawings. It is obvious that the specific implementation of the application is not limited to the above-mentioned manner. As long as various non-essential improvements are made by using the method concept and technical solutions of the application, or the above-mentioned concept and technical solutions of the application are directly applied to other occasions without improvement, they are all within the protection scope of the application.

Claims

1. A ship navigation situational awareness system based on panoramic video, characterized in that: It includes a data acquisition module, a control module, and a display module. The output of the data acquisition module is connected to the input of the control module, and the output of the control module is connected to the input of the display module.

2. The ship navigation situational awareness system based on panoramic video as described in claim 1, characterized in that: The acquisition module includes navigation equipment, environmental measurement equipment, and sensing equipment. The navigation equipment includes an electronic chart display and information system, a global navigation satellite system, and an inertial navigation system. The environmental measurement equipment includes a depth sounder, a weather station, and a radar system. The sensing equipment includes a panoramic video acquisition system, an infrared / thermal imaging camera, an AIS receiver, and a lidar.

3. The ship navigation situational awareness system based on panoramic video as described in claim 1, characterized in that: The control module uses an industrial computer.

4. The ship navigation situational awareness system based on panoramic video as described in claim 1, characterized in that: The display module includes a multi-functional display screen and VR glasses.

5. A perception method for a ship navigation situational awareness system based on panoramic video as described in any one of claims 1-4, characterized in that: Step 1: The control module collects data through navigation equipment, environmental measurement equipment, and sensing equipment; Step 2: The control module preprocesses the collected data; Step 3: The control module extracts features from the data and detects static obstacles and dynamic targets; Step 4: The control module uses a multi-source data fusion algorithm to correlate and fuse static obstacles and dynamic targets respectively; Step 5: The control module displays static obstacles and dynamic targets through the display module; Step Six: The control module assesses the ship's navigation risks based on the ship's navigation status, the position of static obstacles, and the situation of dynamic targets. When the control module predicts a danger, it issues an alarm through the ship's alarm equipment, provides adjustment suggestions, and records the dangerous event.

6. The perception method of a ship navigation situational awareness system based on panoramic video as described in claim 5, characterized in that: In step one, the control module receives electronic chart data from the electronic chart display and information system; the control module collects water depth data below the ship using a depth sounder; the control module collects meteorological data of the ship's environment using a weather station; the control module scans and detects distant targets using a radar system; the control module collects video data of the ship's surrounding environment using a high-definition camera in a panoramic video acquisition system; the control module collects thermal image data using an infrared / thermal imaging camera; the control module receives AIS signals from surrounding ships using an AIS receiver; and the control module collects position data of nearby obstacles using a lidar system.

7. The perception method of a ship navigation situational awareness system based on panoramic video as described in claim 5, characterized in that: In step two, the control module performs clutter suppression and target detection on the radar data collected by the radar system and lidar; the control module performs distortion correction and image enhancement processing on the video data collected by the panoramic video acquisition system; and the control module analyzes and verifies the AIS signals of surrounding ships collected.

8. The perception method of a ship navigation situational awareness system based on panoramic video as described in claim 5, characterized in that: In step three, the control module extracts static obstacle information from the electronic chart data; the control module detects and tracks dynamic targets in the video data through a target detection model; the control module detects heated static obstacles and dynamic targets through an infrared and visible light image fusion algorithm; the control module detects static obstacles and dynamic targets through a radar system and lidar; and the control module's AIS receiver detects stationary or moving ships.

9. The perception method of a ship navigation situational awareness system based on panoramic video as described in claim 5, characterized in that: In step four, the control module uses a multi-source data fusion algorithm to associate and fuse static obstacles and dynamic targets extracted by different devices in the acquisition module.

10. The perception method of a ship navigation situational awareness system based on panoramic video as described in claim 5, characterized in that: In step six, the control module predicts the ship's navigation status through a deep learning model, and assesses the ship's navigation risk by combining the position of static obstacles and the situation of dynamic targets through a ship collision risk calculation model.

Images