Ship collision early warning method and device and storage medium

Abstract

The invention provides a ship collision early warning method and device and a storage medium, and belongs to the technical field of ship early warning, and the method comprises the steps: importing the dynamic information of a moving target around a ship, obtaining an original ship environment image from a multi-channel video sensor disposed on the periphery of the ship, and obtaining original distance data from a laser radar; performing fusion analysis on each original ship environment image and each original distance data to obtain a target panoramic video and preprocessed distance data; and performing collision analysis on the target panoramic video, the preprocessed distance data and the dynamic information to obtain collision data. According to the invention, the perception, detection and collision avoidance early warning capabilities of the periphery of the ship are greatly improved, full-view coverage is realized, the problem of existence of a monitoring blind area is solved, meanwhile, fusion of multi-source data is realized, the target identification precision is improved, the probability of misjudgment and missing judgment is reduced, and the intelligence of collision early warning is realized.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the field of ship collision warning technology, and in particular to a ship collision warning method and device and a storage medium. BACKGROUND

[0002] At present, there is no mature and integrated video perception collision avoidance warning method in the field of ships. The existing scattered perception schemes (such as single video monitoring and independent laser radar detection) have obvious problems of functional fragmentation and performance short board, which cannot meet the needs of ship safety control. The main deficiencies are as follows: 1. Lack of full-view coverage, with monitoring blind area Existing ships mostly use a combination mode of "local area video monitoring + radar detection": video monitoring only covers local areas such as ship decks and bridge rooms, and does not form a 360° perimeter without dead angle. Especially in key positions such as the bow and stern of the ship, blind spots are prone to occur. Although radar can achieve long-distance detection, it has weak recognition ability for small targets in the near field (such as floating objects and small fishing boats), and cannot provide visual details of the target, making it difficult for the operator to accurately judge the target type and threat level. In the near field scene such as ship berthing and narrow channel navigation, blind spots may miss targets and cause collision risks.

[0003] 2. Multi-source data is not fused, and the situation awareness dimension is single In existing schemes, the data of video sensors, laser radars, AIS (ship automatic identification system) and other devices are independent of each other: video devices only output image information and cannot associate spatial parameters such as target distance and position; laser radars only provide distance data and lack target visual feature auxiliary identification; AIS can only obtain dynamic information of ships with signals, and is completely ineffective for targets without AIS signals (such as unpowered floating objects and small speedboats). The fragmented data makes the operator need to view multiple device terminals at the same time, and cannot form an integrated situation awareness of "vision + space + dynamic", which increases the complexity and delay of situation judgment, and makes it difficult to cope with the rapidly changing marine environment.

[0004] 3. Low target recognition accuracy, with prominent misjudgment and omission The target recognition of existing methods relies on manual observation or simple image comparison, and lacks intelligent algorithm support: in complex environments (such as night, fog, and sea wave reflection), the clarity of video images decreases, and manual identification of target types is difficult; even if some methods introduce basic recognition algorithms, they can only distinguish between "ships / non-ships" and other rough categories, and cannot further identify the specific types of targets (such as cargo ships, fishing boats, and obstacles), and are prone to misjudgment of navigation lights and sea wave shadows as targets, or omission of small objects due to target occlusion, resulting in frequent false alarms and missed alarms, which not only interferes with normal navigation operations, but also may cause safety accidents due to missed targets.

[0005] 4. Collision avoidance early warning lacks intelligence and response lag Existing ship collision avoidance mainly relies on manual judgment of risks by radar distance and visual observation, lacking automatic collision avoidance analysis and hierarchical early warning mechanism: on the one hand, manual judgment is greatly affected by the experience and fatigue degree of the operator, and decision-making errors are prone to occur in high-intensity navigation tasks; on the other hand, a collision risk calculation model based on target dynamic parameters (such as speed and heading) and the ship's navigation state has not been established, which cannot accurately predict the collision time and the closest point of approach distance, and can only issue an early warning when the target is at a very close distance, leaving the operator with a short collision avoidance response time and making it difficult to effectively avoid risks. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a ship collision early warning method, device and storage medium to overcome the shortcomings of the prior art.

[0007] The technical solution of the present application to solve the above technical problem is as follows: a ship collision early warning method, comprising the following steps: Importing dynamic information of a plurality of moving targets around the ship, and obtaining a plurality of original ship environment images from a plurality of video sensors arranged on the periphery of the ship, and obtaining original distance data corresponding to each of the moving targets from a laser radar; Respectively performing fusion analysis on each of the original ship environment images and each of the original distance data to obtain a target panoramic video and preprocessed distance data corresponding to each of the original distance data; Performing collision analysis on the target panoramic video, all the preprocessed distance data and all the dynamic information to obtain collision data corresponding to each of the moving targets; Performing early warning analysis on all the collision data, and performing sound and light alarm according to the analysis result, and sending the analysis result to a designated terminal.

[0008] Another technical solution of the present application to solve the above technical problem is as follows: a ship collision early warning device, comprising: A data obtaining module for importing dynamic information of a plurality of moving targets around the ship, and obtaining a plurality of original ship environment images from a plurality of video sensors arranged on the periphery of the ship, and obtaining original distance data corresponding to each of the moving targets from a laser radar; A fusion analysis module for respectively performing fusion analysis on each of the original ship environment images and each of the original distance data to obtain a target panoramic video and preprocessed distance data corresponding to each of the original distance data; A collision analysis module is configured to perform collision analysis on the target panoramic video, all of the preprocessed distance data, and all of the dynamic information, to obtain collision data corresponding to each of the moving targets. A pre-warning analysis module is configured to perform pre-warning analysis on all of the collision data, to perform sound and light warning according to the analysis result, and to send the analysis result to a designated terminal.

[0009] Based on the above-mentioned ship collision pre-warning method, the application further provides a ship collision pre-warning system.

[0010] Another technical solution for solving the above-mentioned technical problem is as follows: a ship collision pre-warning system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the ship collision pre-warning method as described above is realized.

[0011] Based on the above-mentioned ship collision pre-warning method, the application further provides a computer readable storage medium.

[0012] Another technical solution for solving the above-mentioned technical problem is as follows: a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the ship collision pre-warning method as described above is realized.

[0013] The application has the following beneficial effects: through fusion analysis on original ship environment images and original distance data, target panoramic video and preprocessed distance data are obtained; through collision analysis on the target panoramic video, the preprocessed distance data, and dynamic information, collision data are obtained; through pre-warning analysis on the collision data, sound and light warning is performed according to the analysis result, and the analysis result is sent to a designated terminal, which greatly improves the perception, detection, and collision avoidance pre-warning ability of the ship periphery, realizes full-view coverage, solves the problem of existing monitoring blind area, at the same time, realizes fusion of multi-source data, improves target recognition accuracy, reduces the probability of misjudgment and omission, and realizes the intelligentization of collision pre-warning. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of a ship collision pre-warning method provided by an embodiment of the application is shown in the figure. Figure 2 A module block diagram of a ship collision pre-warning device provided by an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0015] The principles and characteristics of the application are described below in combination with the drawings, and the examples are only used to explain the application, and are not used to limit the scope of the application.

[0016] Figure 1A flowchart of a ship collision early warning method provided by an embodiment of the present application.

[0017] As shown in the figure, a ship collision early warning method includes the following steps: Figure 1 S1: Import dynamic information of multiple moving targets around the ship, and obtain multiple original ship environment images from multiple video sensors arranged on the periphery of the ship, and obtain original distance data corresponding to each of the moving targets from a laser radar; S2: Perform fusion analysis on each of the original ship environment images and each of the original distance data, respectively, to obtain a target panoramic video and preprocessed distance data corresponding to each of the original distance data; S3: Perform collision analysis on the target panoramic video, all of the preprocessed distance data, and all of the dynamic information to obtain collision data corresponding to each of the moving targets; S4: Perform early warning analysis on all of the collision data, and perform audible and visual alarm according to the analysis result, and send the analysis result to a designated terminal.

[0018] In the above embodiment, the target panoramic video and the preprocessed distance data are obtained by fusion analysis of the original ship environment images and the original distance data, the collision data is obtained by collision analysis of the target panoramic video, the preprocessed distance data, and the dynamic information, the early warning analysis of the collision data is performed, and the audible and visual alarm is performed according to the analysis result, and the analysis result is sent to the designated terminal, which greatly improves the perception, detection, and collision avoidance early warning capability of the ship periphery, realizes full-view coverage, solves the problem of existing monitoring blind area, at the same time, realizes the fusion of multi-source data, improves the target recognition accuracy, reduces the probability of misjudgment and omission, and realizes the intelligentization of collision early warning.

[0019] Optionally, as an embodiment of the present application, the process of performing fusion analysis on each of the original ship environment images and each of the original distance data to obtain a target panoramic video includes: Preprocess each of the original ship environment images and each of the original distance data to obtain preprocessed ship environment images corresponding to each of the original ship environment images and preprocessed distance data corresponding to each of the original distance data; Fuse all of the preprocessed ship environment images and all of the preprocessed distance data to obtain a target panoramic video.

[0020] It should be understood that the multi-channel video seamlessly and without color difference spliced by the camera front end arranged around the ship is formed into a 360° panoramic situation image information (i.e., a target panoramic video).

[0021] ​Specifically, the video fusion technology is mainly divided into two levels, which are pre-processing and video image fusion.

[0022] In the above embodiment, each original ship environment image and each original distance data are fused and analyzed respectively to obtain a target panoramic video, full-view coverage is realized, the problem of monitoring blind area is solved, multi-source data fusion is realized, and target recognition accuracy is improved.

[0023] Optionally, as an embodiment of the present application, the process of pre-processing each original ship environment image and each original distance data respectively to obtain a pre-processed ship environment image corresponding to each original ship environment image and a pre-processed distance data corresponding to each original distance data comprises: correcting each original ship environment image respectively to obtain a corrected ship environment image corresponding to each original ship environment image; eliminating noise from each corrected ship environment image respectively by using a filtering algorithm to obtain an eliminated ship environment image corresponding to each original ship environment image; adjusting the brightness of each eliminated ship environment image respectively to obtain an adjusted ship environment image corresponding to each original ship environment image; performing image registration on each adjusted ship environment image respectively to obtain a pre-processed ship environment image corresponding to each original ship environment image; performing denoising on each original distance data respectively to obtain a denoised distance data corresponding to each original distance data; performing coordinate calibration on each denoised distance data respectively to obtain a pre-processed distance data corresponding to each original distance data.

[0024] It should be understood that the pre-processing technology mainly performs geometric correction, noise elimination, color brightness adjustment and registration on the video image (i.e. the original ship environment image).

[0025] Specifically, in the process of registering multiple images (i.e. adjusted ship environment images), the geometric motion models mainly used are: translation model, similarity model, affine model and perspective model. The translation model of the image refers to the displacement of the image in the X direction and the Y direction in the two-dimensional space. If the camera only has a translation motion, the translation model can be used. The similarity model of the image refers to the rotation motion of the camera itself in addition to the translation motion. At the same time, when there is a scaling of the scene, the scaling motion with the scaling factor can also be used for description. Therefore, when the image may have translation, rotation, scaling motion, the similarity model can be used. The affine model of the image, i.e. the general characteristics of parallel line transformation into parallel line and limited point mapping to limited point, can specifically be uniform scale transformation with consistent scale transformation coefficients in each direction, non-uniform scale transformation with inconsistent transformation coefficients, and shear transformation, etc., which can describe translation motion, rotation motion and small range scaling and deformation. The perspective model of the image can perfectly express various transformations and is a most accurate transformation model.

[0026] In the above embodiment, each original ship environment image and each original distance data are preprocessed to obtain preprocessed ship environment images and preprocessed distance data, full-view coverage is realized, the problem of monitoring blind area is solved, and the fusion of multi-source data is realized, thereby improving the target recognition accuracy.

[0027] Optionally, as an embodiment of the present application, the process of fusing all the preprocessed ship environment images and all the preprocessed distance data to obtain a target panoramic video comprises: splicing all the preprocessed ship environment images to obtain an original panoramic video; constructing a virtual scene through a preset three-dimensional modeling rule; fusing the original panoramic video, the virtual scene and all the preprocessed distance data to obtain a target panoramic video.

[0028] It should be understood that the image fusion technology can be generally divided into single resolution technology and multi-resolution technology. In the single resolution technology, there are mainly average method, hat function method, weighted average method and median filter method, etc. The multi-resolution technology mainly has Gaussian pyramid, Laplacian pyramid, contrast pyramid, gradient pyramid and wavelet, etc. For the panoramic camera, the image resolution of the front-end multi-camera is the same, so the fusion only exists single resolution fusion.

[0029] Specifically, the preset three-dimensional modeling rule can be to combine the mechanism modeling method and the data-driven method. At present, the modeling methods mainly include mechanism modeling and data-driven modeling. The former establishes mathematical formulas according to the mechanism characteristics of the research object, and assigns parameters, and then applies numerical calculation method or analytical method for calculation, which is generally suitable for physical systems with clear mechanism. The data-driven modeling adopts statistical and machine learning methods to establish a model, which is suitable for research objects with unclear mechanism or only existing correlation. When mechanism modeling, due to the inevitable assumptions and simplifications, sometimes it will bring an error that cannot be ignored. In this case, if the data is sufficient, data-driven modeling method is also suitable. In addition, when using data-driven method, in order to solve the problems of small sample, unbalanced sample, weak feature and uninterpretable, the combination of mechanism modeling method and data-driven method has certain advantages.

[0030] It should be understood that the video fusion technology mainly fuses one or more image sequence videos (i.e., original panoramic videos) about a scene or a model collected by a video collection device with a virtual scene related thereto, and generates a new virtual scene or model (i.e., target panoramic video) about the scene.

[0031] Specifically, the video image fusion can be divided into pixel-level, feature-level and decision-level fusion from low to high in terms of intelligence. The pixel-level fusion is based on image pixels for splicing fusion. The feature-level fusion is mainly based on the obvious features of graphics, such as lines, buildings and other features, for image splicing fusion. The decision-level fusion uses mathematical algorithms such as Bayesian method and D-S evidence method for probability decision, thereby performing video or image fusion.

[0032] In the above embodiment, all preprocessed ship environment images and all preprocessed distance data are fused to obtain the target panoramic video, so as to realize the fusion of multiple source data, improve the target recognition accuracy, reduce the probability of misjudgment and omission, and realize the intelligentization of collision warning.

[0033] Optionally, as an embodiment of the present application, the dynamic information includes AIS data, and the process of performing collision analysis on the target panoramic video, all the preprocessed distance data and all the dynamic information to obtain collision data corresponding to each mobile target includes: The target panoramic video and each AIS data are identified by a pre-constructed target recognition model to obtain mobile target identity data corresponding to each mobile target and mobile target basic parameters corresponding to each mobile target. The ship target speed and heading and the ship navigation parameters are imported, and the collision risk calculation is performed on the ship target speed and heading, the ship navigation parameters, each pre-processed distance data, the mobile target identity data corresponding to each mobile target, and the mobile target basic parameters corresponding to each mobile target through the pre-constructed collision risk model, to obtain the collision data corresponding to each mobile target.

[0034] It should be understood that the target appearing in the video collected by the multi-channel video sensor (i.e. target panoramic video) is identified, and the target information collected by the laser radar (i.e. AIS data) is analyzed and processed to respectively obtain the target type (i.e. mobile target identity data) and the position information (i.e. mobile target basic parameters) in real time.

[0035] Specifically, the collision prediction is calculated through the collision avoidance detection analysis and early warning algorithm.

[0036] In the above embodiment, the collision data is obtained by performing collision analysis on the target panoramic video, all pre-processed distance data, and all dynamic information, which greatly improves the perception, detection, and collision avoidance warning capability of the ship surrounding, realizes full-view coverage, solves the problem of existing monitoring blind area, at the same time, realizes the fusion of multi-source data, improves the target recognition accuracy, reduces the probability of misjudgment and omission, and realizes the intelligentization of collision warning.

[0037] Optionally, as an embodiment of the present application, the collision data includes collision probability and collision distance, and the process of performing early warning analysis on all the collision data and performing sound and light alarm according to the analysis result includes: If the collision probability is greater than or equal to a preset threshold, the buzzer is controlled to sound, and the flashing light is controlled to flash a preset first light; If the collision distance is less than a preset first warning distance, the buzzer is controlled to sound, and the flashing light is controlled to flash a preset second light; If the collision distance is greater than the preset first warning distance and less than a preset second warning distance, the buzzer is controlled to sound, and the flashing light is controlled to flash a preset third light; If the collision distance is less than the preset second warning distance, the buzzer is controlled to sound, and the flashing light is controlled to flash a preset fourth light.

[0038] It should be understood that the early warning distance: 0-500 meters can be set, and the early warning is graded, and the early warning mode is sound and light alarm.

[0039] In the above embodiments, all collision data are analyzed for early warning, and audible and visual alarms are triggered based on the analysis results. This achieves full-view coverage, solves the problem of blind spots in monitoring, and simultaneously realizes the fusion of multi-source data, improving target recognition accuracy, reducing the probability of false positives and false negatives, and realizing intelligent collision warning.

[0040] Optionally, as an embodiment of the present invention, it further includes: The panoramic video of the target, multiple identity data of the moving targets, multiple basic parameters of the moving targets, multiple collision probabilities, and multiple collision distances are displayed.

[0041] In the above embodiments, the display of the target panoramic video, multiple moving target identity data, multiple moving target basic parameters, multiple collision probabilities, and multiple collision distances can assist operators in making collision avoidance decisions.

[0042] Optionally, as another embodiment of the present invention, the ship video perception collision avoidance and early warning system of the present invention is applied to the situational awareness and monitoring of the ship's surroundings, collision avoidance detection analysis and early warning. It includes the invention of multi-channel video fusion and stitching 360-degree panoramic technology for the ship's perimeter, real-time video and virtual fusion display technology, target recognition technology for the ship's surroundings, and collision avoidance detection analysis and early warning algorithm technology, which improves the ship's situational awareness and collision avoidance early warning capabilities. It can be widely used in various types of ships to realize intelligent applications such as panoramic monitoring of the surrounding situation, identification and alarm of approaching targets, collision avoidance early warning, and target ranging of ships while berthing and sailing.

[0043] Optionally, as another embodiment of the present invention, the ship video perception collision avoidance and early warning system of the present invention incorporates ship perimeter multi-channel video fusion and stitching 360-degree panoramic technology, real-time video and virtual fusion display technology, ship surrounding target recognition technology, and ship collision avoidance detection analysis and early warning algorithm technology. The background technologies involved include digital twin technology, VR technology, panoramic video stitching technology, etc. However, in the ship video perception collision avoidance and early warning system, the related background technologies have been further innovated, forming multi-channel video fusion and stitching 360-degree panoramic technology, real-time video and virtual fusion display technology, etc.

[0044] This real-time video and virtual fusion display technology is an innovative design based on digital twin and VR technologies. Digital twin technology is maturing rapidly and is widely used in smart city construction, rail transit, airports, high-speed rail, schools, and production workshops. Through a digital twin platform, 3D visualization of data on people, places, events, objects, and groups in various scenarios can be achieved, enabling full-process dynamic monitoring and real-time attitude control. However, it is not yet widely used on naval vessels, especially in a domestically produced environment. Given the rapid development of key domestic hardware and software in recent years, the performance of core electronic components such as processors, graphics cards, MCUs, and FPGAs, as well as key software such as operating systems and databases, has reached the current application capabilities and achieved independent control.

[0045] VR technology is a crucial component in building digital twin systems. With the support of VR technology, the manufacturing, operation, and maintenance status of physical entities can be presented in a surreal way, providing an immersive virtual reality experience through visual, auditory, and tactile senses. By acquiring physical equipment parameters, capturing images of the physical equipment, and consulting other auxiliary materials, the necessary data for building the digital twin can be accumulated.

[0046] Multi-channel video fusion and stitching 360-degree panoramic technology is an innovative development based on the original panoramic video stitching technology. The two key steps of panoramic video stitching technology are registration and fusion. Registration aims to register images to the same coordinate system based on a geometric motion model; fusion combines the registered images into a single large stitched image.

[0047] In the process of multi-image registration, the geometric motion models mainly used include: translation model, similarity model, affine model, and perspective model. The translation model refers to an image that has only undergone displacement in the X and Y directions in two-dimensional space. If the camera only undergoes translational motion, the translation model can be used. The similarity model refers to situations where the camera itself may undergo rotational motion in addition to translation. Furthermore, when there is scene scaling, scaling factors can be used to describe the motion. Therefore, when an image may undergo translation, rotation, or scaling, the similarity model can be used. The affine model of an image possesses the general properties of transforming parallel lines into parallel lines and mapping finite points to finite points. Specifically, it can represent uniform scale transformations with consistent transformation coefficients in all directions, non-uniform scale transformations with inconsistent transformation coefficients, and shearing transformations, etc. It can describe translational motion, rotational motion, and small-scale scaling and deformation. The perspective model of an image can perfectly represent various transformations and is one of the most accurate transformation models.

[0048] Image fusion techniques can generally be divided into two categories: single-resolution techniques and multi-resolution techniques. Single-resolution techniques mainly include the averaging method, hat function method, weighted averaging method, and median filtering method. Multi-resolution techniques mainly include Gaussian pyramid, Laplacian pyramid, contrast pyramid, gradient pyramid, and wavelet fusion. However, for panoramic cameras, the images acquired by multiple cameras at the front end have the same resolution; therefore, fusion only involves single-resolution fusion.

[0049] Currently, panoramic video stitching technology still has problems with color difference and seam width, with many images showing obvious color differences and relatively obvious seams.

[0050] The technology of ship collision avoidance detection analysis and early warning algorithm is an innovative collision avoidance detection analysis and early warning algorithm technology based on the near-field target information around the ship detected by lidar.

[0051] Optionally, as another embodiment of the present invention, the purpose of the present invention is to provide various types of ships with 360-degree panoramic situational monitoring, video target perception and collision avoidance early warning, forming ship video perception collision avoidance early warning equipment, and in order to address the shortcomings of technologies such as panoramic video stitching, to innovatively form multi-channel video fusion stitching 360-degree panoramic technology, real-time video and virtual fusion display technology, surrounding target recognition technology, and ship collision avoidance detection analysis and early warning algorithm technology.

[0052] Optionally, as another embodiment of the present invention, the ship video perception collision avoidance and early warning system of the present invention consists of a detection front end (lidar, video sensor), video stitching and fusion equipment, transmission equipment, video perception collision avoidance and early warning platform, ship video perception collision avoidance and early warning terminal, etc. It integrates multi-channel video fusion stitching 360-degree panoramic technology, real-time video and virtual fusion display technology, surrounding target recognition technology, ship collision avoidance detection analysis and early warning algorithm technology, so as to realize intelligent applications such as panoramic monitoring of the surrounding situation, identification and alarm of approaching targets, collision avoidance early warning, and target ranging of ships while berthing and sailing.

[0053] Optionally, as another embodiment of the present invention, the video perception collision avoidance and early warning platform software of the present invention is developed based on a domestic operating system, database and hardware environment. It integrates the data collected from the front end and maps it to the virtual space in a 1:1 manner through the 3D modeling of the device. It integrates various business subsystems, monitoring data and other information using virtual simulation technology, big data technology, Internet of Things technology, AI algorithms, etc., and quickly and intuitively displays them in the 3D virtual visualization system. It presents the relevant real-world scene in the form of digital twin and 3D visualization, and realizes target recognition analysis and collision avoidance detection analysis and early warning.

[0054] Optionally, as another embodiment of the present invention, the detection front end of the present invention mainly consists of video sensors, lidar, etc., which can realize the collection of various types of information at the front end and transmit them to the back-end processing device for data fusion via Ethernet.

[0055] (I) Main tactical and technical indicators 1) The software platform enables 1:1 digital modeling and high-fidelity 3D display; 2) The software supports switching between virtual and real-world visuals; 3) Supports AIS and radar navigation signal access; 4) Supports 360° panoramic situational monitoring around ships; 5) Image resolution is no less than 1080P; 6) Detection distance: not less than 100m; 7) Detection and ranging accuracy: ±0.1m; 8) Detection angle: 360° horizontally, and no less than 15° vertically; 9) Vessel identification distance: not less than 100m (visible light); 10) Warning distance: adjustable from 0 to 500 meters, with tiered warning systems; 11) Warning method: audible and visual alarm.

[0056] (II) Key Technologies and Implementation Approaches Ship video-based collision avoidance and early warning systems are comprehensive technologies. Data is the foundation of the technology, models are the core, and platforms or software are the carriers. Through full-domain data identification, precise status perception, real-time data analysis, scientific model-based decision-making, and intelligent and precise execution, the system automatically identifies, mines, and reconstructs 3D data, and endows the data with bidirectional control processing capabilities.

[0057] Optionally, as another embodiment of the present invention, the VR technology of the present invention is an important link in supporting the construction of a digital twin system. With the support of VR technology, the manufacturing, operation, and maintenance status of physical entities can be presented in a surreal form, providing an immersive virtual reality experience from various aspects such as vision, sound, and touch. By acquiring physical equipment parameters, capturing images of the physical equipment, and consulting other auxiliary information, the necessary data for the construction of the digital twin is accumulated.

[0058] Alternatively, as another embodiment of the present invention, the advantages of the present invention and the expected economic and social benefits are analyzed as follows: (1) Analysis of advanced nature Currently, no ship platform of any type is equipped with a video perception collision avoidance and early warning system. Equipping ships with video perception collision avoidance and early warning systems while they are docked or underway will greatly enhance their perception, detection, collision avoidance, and early warning capabilities around them. This is the first of its kind in China's intelligent monitoring and early warning system for ships.

[0059] (2) Analysis of economic and social benefits The application of technologies such as multi-channel video fusion and stitching 360-degree panoramic technology, real-time video and virtual fusion display technology, surrounding target recognition technology, and ship collision avoidance detection analysis and early warning algorithms has become the main development direction of ship intelligent monitoring, collision avoidance and early warning, which has improved the detection and early warning capabilities of ships. The research results of this invention can be widely used in unmanned ships and manned ships.

[0060] This invention is designed for both unmanned and manned vessels. All hardware components must be suitable for the actual operating environment of the vessel. Combined with years of experience in the research and production of naval equipment, it possesses unique technological advantages. Once this technology is fully mastered, it can be applied to numerous vessels, showing broad development prospects.

[0061] Optionally, as another embodiment of the present invention, the ship video perception collision avoidance and early warning system of the present invention realizes the perception, monitoring and collision avoidance early warning of near-field targets and surrounding situation of the ship through the following steps: ① Ship video perception collision avoidance and early warning is a system that uses video sensors and lidar to monitor, detect and identify objects and targets around the ship, and collects information on the surrounding field of view and objects and targets. ② After the video sensor collects the field-of-view image information around the ship, the video stitching and fusion equipment merges and stitches the multiple videos together, performs geometric correction, noise elimination, color and brightness adjustment and registration on the video images, and forms a 360-degree panoramic real-time video around the ship, which is then transmitted to the video perception collision avoidance and early warning platform through the transmission equipment. ③ The video perception collision avoidance and early warning platform integrates the ship's 360-degree panoramic real-time video with a related virtual scene to form a more comprehensive 360-degree panoramic real-time video of the ship. The 360° panoramic situational image information is presented on the ship's video perception collision avoidance and early warning terminal to monitor the situation around the ship in real time. ④ The video perception collision avoidance and early warning platform identifies targets appearing in videos collected by multiple video sensors, analyzes and processes target information collected by lidar, and obtains target type and location information in real time. ⑤ The collision prediction is calculated by the collision detection analysis and early warning algorithm of the video perception collision avoidance early warning platform. If the collision prediction exceeds the set threshold, a collision avoidance early warning is issued.

[0062] Alternatively, as another embodiment of the present invention, the protection points of the present invention are as follows: 1. Implementation methods for ship video perception collision avoidance and early warning; 2. Ship collision avoidance detection analysis and early warning algorithm; 3. 360-degree panoramic technology for fusion and stitching of multi-channel video around the ship's perimeter; 4. Real-time video and virtual fusion display technology.

[0063] Figure 2 This is a block diagram of a ship collision warning device provided in an embodiment of the present invention.

[0064] Alternatively, as another embodiment of the present invention, such as Figure 2 As shown, a ship collision early warning device includes: The data acquisition module is used to import dynamic information of multiple moving targets around the ship, obtain multiple original ship environment images from multiple video sensors set on the ship perimeter, and obtain original distance data corresponding to each of the moving targets from the lidar. The fusion analysis module is used to perform fusion analysis on each of the original ship environment images and each of the original distance data to obtain the target panoramic video and the preprocessed distance data corresponding to each of the original distance data. The collision analysis module is used to perform collision analysis on the target panoramic video, all the preprocessed distance data and all the dynamic information to obtain collision data corresponding to each of the moving targets. The early warning analysis module is used to perform early warning analysis on all the collision data, and to issue audible and visual alarms based on the analysis results, and send the analysis results to the designated terminal.

[0065] Optionally, another embodiment of the present invention provides a ship collision warning system, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the ship collision warning method as described above. This system can be a computer or similar system.

[0066] Optionally, another embodiment of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the ship collision warning method as described above.

[0067] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of the present invention, depending on actual needs.

[0071] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for early warning of ship collisions, characterized in that, Includes the following steps: The system imports dynamic information of multiple moving targets around the ship, obtains multiple original ship environment images from multiple video sensors set on the ship's perimeter, and obtains original distance data corresponding to each moving target from the lidar. The original ship environment images and original distance data are fused and analyzed to obtain the target panoramic video and the preprocessed distance data corresponding to each of the original distance data. Collision analysis is performed on the target panoramic video, all the preprocessed distance data, and all the dynamic information to obtain collision data corresponding to each of the moving targets. All collision data are analyzed for early warning, and audible and visual alarms are triggered based on the analysis results. The analysis results are then sent to a designated terminal.

2. The ship collision early warning method according to claim 1, characterized in that, The process of fusing and analyzing each of the original ship environment images and each of the original distance data to obtain the target panoramic video includes: Each of the original ship environment images and each of the original distance data are preprocessed to obtain preprocessed ship environment images and preprocessed distance data corresponding to each of the original ship environment images and each of the original distance data. All the preprocessed ship environment images and all the preprocessed distance data are fused to obtain the target panoramic video.

3. The ship collision early warning method according to claim 2, characterized in that, The process of preprocessing each of the original ship environment images and each of the original distance data to obtain preprocessed ship environment images corresponding to each of the original ship environment images and preprocessed distance data corresponding to each of the original distance data includes: Each of the original ship environment images is corrected to obtain a corrected ship environment image corresponding to each of the original ship environment images. The noise reduction process is performed on each of the corrected ship environment images using a filtering algorithm to obtain the noise-reduced ship environment images corresponding to each of the original ship environment images. The brightness of each of the eliminated ship environment images is adjusted to obtain an adjusted ship environment image corresponding to each of the original ship environment images. Each of the adjusted ship environment images is subjected to image registration processing to obtain a preprocessed ship environment image corresponding to each of the original ship environment images. Each of the original distance data is denoised to obtain denoised distance data corresponding to each of the original distance data. Each of the denoised distance data is subjected to coordinate calibration to obtain preprocessed distance data corresponding to each of the original distance data.

4. The ship collision early warning method according to claim 2, characterized in that, The process of fusing all the preprocessed ship environment images and all the preprocessed distance data to obtain the target panoramic video includes: All the preprocessed ship environment images are stitched together to obtain the original panoramic video; A virtual scene is constructed by pre-setting 3D modeling rules; The original panoramic video, the virtual scene, and all the preprocessed distance data are fused together to obtain the target panoramic video.

5. The ship collision early warning method according to claim 1, characterized in that, The dynamic information includes AIS data. The process of performing collision analysis on the target panoramic video, all the preprocessed distance data, and all the dynamic information to obtain collision data corresponding to each of the moving targets includes: The pre-built target recognition model is used to identify the target panoramic video and each of the AIS data respectively, to obtain the mobile target identity data and the mobile target basic parameters corresponding to each of the mobile targets; Import the ship target's speed and heading, as well as the ship's navigation parameters. Using a pre-built collision risk model, calculate the collision risk for the ship target's speed and heading, the ship's navigation parameters, each of the pre-processed distance data, the identity data of each moving target, and the basic parameters of each moving target to obtain the collision data corresponding to each moving target.

6. The ship collision early warning method according to claim 5, characterized in that, The collision data includes collision probability and collision distance. The process of performing early warning analysis on all the collision data and issuing audible and visual alarms based on the analysis results includes: If the collision probability is greater than or equal to a preset threshold, the buzzer is controlled to provide an audible alert, and the flashing light is controlled to flash a preset first light. If the collision distance is less than the preset first warning distance, the buzzer is controlled to provide an audible alert, and the flashing light is controlled to flash a preset second light. If the collision distance is greater than the preset first warning distance and less than the preset second warning distance, then the buzzer is controlled to provide an audible warning, and the flashing light is controlled to flash a preset third light. If the collision distance is less than the preset second warning distance, the buzzer will be controlled to provide an audible alert, and the flashing light will be controlled to flash a preset fourth light.

7. The ship collision early warning method according to claim 6, characterized in that, Also includes: The panoramic video of the target, multiple identity data of the moving targets, multiple basic parameters of the moving targets, multiple collision probabilities, and multiple collision distances are displayed.

8. A ship collision early warning device, characterized in that, include: The data acquisition module is used to import dynamic information of multiple moving targets around the ship, obtain multiple original ship environment images from multiple video sensors set on the ship perimeter, and obtain original distance data corresponding to each of the moving targets from the lidar. The fusion analysis module is used to perform fusion analysis on each of the original ship environment images and each of the original distance data to obtain the target panoramic video and the preprocessed distance data corresponding to each of the original distance data. The collision analysis module is used to perform collision analysis on the target panoramic video, all the preprocessed distance data and all the dynamic information to obtain collision data corresponding to each of the moving targets. The early warning analysis module is used to perform early warning analysis on all the collision data, and to issue audible and visual alarms based on the analysis results, and send the analysis results to the designated terminal.

9. A ship collision warning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the ship collision warning method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the ship collision warning method as described in any one of claims 1 to 7.

Images