Ship navigation collision risk early warning method and system, electronic equipment and storage medium

Abstract

The invention provides a ship navigation collision risk early warning method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining environment data and video data in a harbor basin, and carrying out the preprocessing of the environment data and the video data; performing target identification and tracking on the preprocessed video data through a deep reinforcement learning algorithm to determine a dynamic target; performing multi-source data fusion according to the dynamic target data, the environment data, the radar positioning data and the AIS data to obtain a fusion result; and based on the fusion result, an optimal navigation path is generated through a path planning algorithm, and ship auxiliary berthing is controlled according to the optimal navigation path and the collision early warning signal. According to the invention, the functions of ship piloting and auxiliary berthing can be realized, the navigation capability and early warning capability of the ship in a complex environment are improved, and the ship navigation collision risk is reduced.

Description

Technical Field

[0001] The present application belongs to the field of ship navigation warning technology, and in particular relates to a ship navigation collision risk warning method, system, electronic equipment and storage medium. Background Art

[0002] With the development of science and technology, the demand for safe navigation of ships is increasing, especially in adverse weather conditions such as at night and in fog. Due to the large size and length of ultra-large ships, the wind and complex flow fields cause the bow and stern of the ship to be affected by wind currents of different intensities and directions, which reduces the ship's steering efficiency, directly affecting the ship's course and making it difficult for the ship to turn. In particular, nearshore currents can have a significant impact on tugboat operations. This has prompted the rapid development of intelligent assistance systems for night and fog navigation based on visual enhancement and target recognition. Intelligent assisted navigation systems can perceive the hydrological and meteorological environment, the ship's motion status, other ships and obstacles in real time, and achieve beyond-line-of-sight perception in fog and low visibility conditions. Through multi-format visual enhancement displays, they can realize real-time judgment of navigation status and dangers.

[0003] However, existing assistance systems have poor perception and decision-making capabilities at night or in extremely foggy conditions, with a narrow sensing range and low reliability. Furthermore, high-speed government vessels rely on pilots and conventional observation devices to monitor the surroundings at night and in fog, resulting in poor observation and inadequate collision risk warnings. Furthermore, existing technologies focus solely on single devices or algorithms, lacking a comprehensive assistance system. Summary of the Invention

[0004] In view of this, the present application aims to propose a ship navigation collision risk warning method, system, electronic device and storage medium to solve at least one of the above problems.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows: In a first aspect, the present application provides a method for warning of ship collision risk, comprising: Acquiring environmental data and video data within the harbor, and preprocessing the environmental data and the video data respectively; Performing target recognition and tracking on the pre-processed video data using a deep reinforcement learning algorithm to determine dynamic targets; Performing multi-source data fusion based on the dynamic target data, environmental data, radar positioning data, and AIS data to obtain a fusion result; Based on the fusion result, an optimal navigation path is generated through a path planning algorithm, and the ship is controlled to assist in berthing according to the optimal navigation path and the collision warning signal.

[0006] Secondly, based on the same inventive concept, the present application also provides a ship navigation collision risk warning system, comprising: a data acquisition module configured to acquire environmental data and video data within the harbor, and pre-process the environmental data and the video data respectively; A target recognition module is configured to perform target recognition and tracking on the pre-processed video data using a deep reinforcement learning algorithm to determine dynamic targets; a data fusion module configured to perform multi-source data fusion based on the dynamic target data, environmental data, radar positioning data, and AIS data to obtain a fusion result; The path planning module is configured to generate an optimal navigation path based on the fusion result and through a path planning algorithm, and control the auxiliary berthing of the ship according to the optimal navigation path and the collision warning signal.

[0007] In a third aspect, based on the same inventive concept, the present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0008] In a fourth aspect, based on the same inventive concept, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in the first aspect.

[0009] Compared with the prior art, the ship navigation collision risk warning method, system, electronic device and storage medium described in this application have the following beneficial effects: The ship navigation collision risk warning method described in this application dynamically perceives the external environment through real-time collected environmental data and video image data, which can help crew members better perceive the environment around the ship, identify potential dangers, and assist the ship in safe route planning and navigation operations. This method can realize the functions of ship piloting and assisted mooring, improve the navigation and warning capabilities of ships in complex environments, and reduce the risk of ship navigation collisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings: Figure 1 This is a flow chart of a method for warning of ship collision risk according to an embodiment of the present application; Figure 2 This is a schematic structural diagram of a ship navigation collision risk warning system according to an embodiment of the present application; Figure 3 This is a schematic diagram of the hardware structure of the electronic device described in an embodiment of the present application. DETAILED DESCRIPTION

[0011] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0012] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0013] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0014] See also Figure 1 As shown, this embodiment provides a method for warning of ship collision risk, which specifically includes the following steps: Step S101: Acquire environmental data and video data in the harbor, and pre-process the environmental data and video data respectively.

[0015] Step S102: Perform target recognition and tracking on the pre-processed video data using a deep reinforcement learning algorithm to determine dynamic targets.

[0016] Step S103: Perform multi-source data fusion based on the dynamic target data, environmental data, radar positioning data, and AIS data to obtain a fusion result.

[0017] Step S104: Based on the fusion result, an optimal navigation path is generated through a path planning algorithm, and the ship is controlled to assist in berthing according to the optimal navigation path and the collision warning signal.

[0018] A ship navigation collision risk warning method of the present application dynamically perceives the external environment through real-time collected environmental data and video image data, which can help crew members better perceive the environment around the ship, identify potential dangers, and assist the ship in safe route planning and navigation operations. This method can realize the functions of ship piloting and assisted mooring, improve the navigation and warning capabilities of ships in complex environments, and reduce the risk of ship navigation collisions.

[0019] In some embodiments, the environmental data includes meteorological data and hydrological data, and data cleaning and filling are performed on the meteorological data and hydrological data respectively, wherein the meteorological data includes wind speed, wind direction, and visibility, and the hydrological data includes flow velocity, flow direction, and water depth; The video data is processed by an image quality optimization algorithm to obtain an enhanced video image; Specifically, in step S101 of this embodiment, for environmental data, this embodiment uses meteorological and hydrological sensors to obtain data such as wind direction, wind speed, flow direction, flow speed, visibility, etc. in the harbor, and transmits the data to a remote computer via a cable to provide environmental data for the next step of data preprocessing.

[0020] Environmental data preprocessing: Check the data for obvious errors, such as values ​​outside the reasonable range for wind direction, wind speed, and other data, logical errors in flow direction and velocity data, etc. Mark and delete these erroneous data, and use interpolation to refill the deleted data to obtain complete data values.

[0021] For video data, this embodiment uses high-definition cameras and infrared cameras to capture the dynamics of ships in the harbor and extract the position and dynamic information of each ship and small target in the harbor, such as movement speed and direction.

[0022] Video data preprocessing: At night, in heavy rain or fog, video images often appear blurry and have low contrast. This results in high noise and low data accuracy when extracting data, making it difficult to directly use. Therefore, this application proposes an image quality enhancement technology suitable for conditions with poor visibility. The basic idea is to use an image with good daytime visibility as a model, overlap and compare it with an image with poor visibility, and migrate key feature points to the image with good visibility to achieve image quality optimization.

[0023] In some implementations, the image quality optimization algorithm includes: Input a first video image under poor visibility conditions and a second video image under good visibility conditions, traverse the first video image and the second video image frame by frame, and convert each frame image after traversal into a first image matrix and a second image matrix respectively; Enumerate the coordinates of each pixel in the first image matrix and the second image matrix, and define the coordinates of the center point of each frame of image; Defining a neighborhood matrix of pixel points in the first video image and the second video image, and calculating the Euclidean distance between the coordinates of each pixel in the neighborhood matrix and the coordinates of the center point using a distance formula, so that the positions of the pixel points of the two video images correspond to each other; Removing repeated backgrounds from the next moment image and the current moment image from the first video image and retaining feature points to obtain a third image matrix; Adding the first image matrix to the second image matrix to obtain a fourth image matrix; The fourth image matrix corresponding to each frame of image is reconverted into a new video image.

[0024] The specific steps of the image quality optimization algorithm are as follows: In some embodiments, the DQN algorithm based on the front-end database grids the new video image, determines the longitude and latitude of each grid, traverses each frame of the image, and subtracts the current frame image from the next frame to determine the dynamic target.

[0025] Specifically, in step S102 of this embodiment, the DQN algorithm cannot be directly used for target recognition and tracking because the original algorithm is only applicable to single-target tasks. Target recognition and tracking is divided into two parts, recognition and tracking, which are multi-target tasks. Furthermore, there are many targets at sea, and many static objects do not need to be recognized. Therefore, this application improves the original algorithm and proposes a DQN algorithm based on a pre-database. The specific algorithm is as follows: In some embodiments, multi-source data fusion is performed based on dynamic target data, environmental data, radar positioning data, and AIS data to obtain a fusion result, including: Perform data cleaning and normalization on dynamic target data, environmental data, radar positioning data, and AIS data, and assign matching weights to each data type; The normalized data and the corresponding weights are weighted and the weighted data are summed to obtain the data fusion result.

[0026] Specifically, in step S103 of this embodiment, the importance of different data is assigned corresponding weights using the weighted average method, and then the weighted values ​​are averaged to obtain the fusion result. The specific steps are as follows: 1. Data preprocessing The input dynamic target data, environmental data, radar positioning data, AIS data, etc. are cleaned to remove obviously erroneous or abnormal data points. The data are then normalized and data of different dimensions are converted to the range of [0, 1] to ensure that the weights of different types of data are comparable during the fusion process.

[0027] 2. Determine the weight Assign a weight to each data type based on the importance, reliability, or relevance of the data. The weight is determined through empirical judgment. The weights set in this embodiment are specifically: Weight value corresponding to dynamic target data =0.05, weight value corresponding to environmental data , the weight value corresponding to the radar positioning data =0.3 and the weight value corresponding to AIS data =0.2.

[0028] 3. Weighted calculation Multiply the normalized data by their corresponding weights to obtain weighted data. Assume that the normalized dynamic target data is , the environmental data is , the radar positioning data is , AIS data is , then the weighted dynamic target data is , the weighted environmental data is , the weighted radar positioning data is , the weighted AIS data is .

[0029] 4. Fusion result calculation Sum the weighted data to get the result of data fusion This result integrates information from multiple factors and can comprehensively reflect the navigation status of the ship at sea.

[0030] It should be noted that since environmental data contains multiple sub-data such as wind, waves, and currents, they all need to be considered comprehensively when setting the corresponding weights and weighted calculations to reflect the overall environmental status of the ship.

[0031] In some embodiments, the path planning algorithm includes: Generate a risk matrix based on the current environmental state, select an action based on the current environmental state, and after executing the selected action, generate a new risk matrix, environmental state, and updated data fusion results; The temporal target of the path planning is calculated based on the temporal difference algorithm, and the loss function is calculated based on the temporal target and the output of the pre-built critic network. The parameters of the critic network are updated based on the obtained loss function and the gradient descent method; Iterative update outputs the optimal path.

[0032] Specifically, in step S104 of this embodiment, the "temporal difference method" (TDlearning) in reinforcement learning is used to dynamically update the strategy and path planning results. The following is a complete technical solution that summarizes the various steps: In some embodiments, the relative distance between ships or between a ship and a shore is determined based on shore-based sensor signals to generate an early warning signal.

[0033] It should be noted that the above description is limited to some embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in an order different from that described in the above embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0034] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides a ship navigation collision risk warning system.

[0035] like Figure 2 As shown, the ship navigation collision risk warning system includes: The data acquisition module 11 is configured to acquire environmental data and video data in the harbor and pre-process the environmental data and video data respectively; The target recognition module 12 is configured to perform target recognition and tracking on the pre-processed video data using a deep reinforcement learning algorithm to determine dynamic targets; The data fusion module 13 is configured to perform multi-source data fusion based on dynamic target data, environmental data, radar positioning data and AIS data to obtain a fusion result; The path planning module 14 is configured to generate an optimal navigation path based on the fusion result and through a path planning algorithm, and control the ship to assist in berthing according to the optimal navigation path and the collision warning signal.

[0036] For the convenience of description, the above system is described as being divided into various modules according to their functions. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0037] The system of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0038] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, an embodiment of the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0039] Figure 3 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0040] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0041] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0042] The input / output interface 1030 is used to connect to an input / output module to enable information input and output. The input / output module can be configured as a component within the device (not shown) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc. Output devices may include a display, speaker, vibrator, indicator light, etc.

[0043] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, Wi-Fi, Bluetooth, etc.).

[0044] The bus 1050 comprises a pathway for transmitting information between various components of the device, such as the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 .

[0045] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0046] The electronic device of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0047] Based on the same inventive concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the method described in any of the above embodiments.

[0048] The computer-readable media of this embodiment includes permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0049] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0050] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0051] In addition, to simplify the description and discussion, and to avoid obscuring the understanding of the embodiments of the present application, well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided figures. Furthermore, devices may be shown in block diagram form to avoid obscuring the understanding of the embodiments of the present application, and this also takes into account the fact that the implementation details of these block diagram devices are highly dependent on the platform on which the embodiments of the present application will be implemented (i.e., these details should be fully understood by those skilled in the art). Where specific details (e.g., circuits) are set forth to describe the exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with variations therefrom. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0052] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the discussed embodiments.

[0053] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A ship navigation collision risk warning method, characterized in that: include: Acquiring environmental data and video data within the harbor, and preprocessing the environmental data and the video data respectively; Performing target recognition and tracking on the pre-processed video data using a deep reinforcement learning algorithm to determine dynamic targets; Performing multi-source data fusion based on the dynamic target data, environmental data, radar positioning data, and AIS data to obtain a fusion result; Based on the fusion result, an optimal navigation path is generated through a path planning algorithm, and the ship is controlled to assist in berthing according to the optimal navigation path and the collision warning signal.

2. The method according to claim 1, wherein: The environmental data includes meteorological data and hydrological data, and the meteorological data and the hydrological data are cleaned and filled respectively, wherein the meteorological data includes wind speed, wind direction, and visibility, and the hydrological data includes flow velocity, flow direction, and water depth; The video data is processed using an image quality optimization algorithm to obtain an enhanced video image.

3. The method according to claim 2, characterized in that The image quality optimization algorithm includes: Input a first video image under poor visibility conditions and a second video image under good visibility conditions, traverse the first video image and the second video image frame by frame, and convert each frame of the traversed image into a first image matrix and a second image matrix respectively; Enumerate the coordinates of each pixel in the first image matrix and the second image matrix, and define the coordinates of the center point of each frame of image; Defining a neighborhood matrix of pixels in the first video image and the second video image, and calculating the Euclidean distance between each pixel coordinate in the neighborhood matrix and the center point coordinate using a distance formula, so that the pixel positions of the two video images correspond to each other; Removing repeated backgrounds from the next moment image and the current moment image from the first video image and retaining feature points to obtain a third image matrix; Adding the first image matrix and the second image matrix to obtain a fourth image matrix; The fourth image matrix corresponding to each frame of image is reconverted into a new video image.

4. The method according to claim 3, wherein: The DQN algorithm based on the front-end database grids the new video image, determines the longitude and latitude of each grid, traverses each frame of the image, and makes a difference between the current frame and the next frame to determine the dynamic target.

5. The method according to claim 1, wherein The multi-source data fusion is performed according to the dynamic target data, environmental data, radar positioning data and AIS data to obtain a fusion result, including: Performing data cleaning and normalization on the dynamic target data, environmental data, radar positioning data, and AIS data, and assigning a matching weight to each data according to the data type; The normalized data and the corresponding weights are weighted and the weighted data are summed to obtain the data fusion result.

6. The method according to claim 1, characterized in that The path planning algorithm includes: Generate a risk matrix based on the current environmental state, select an action based on the current environmental state, and after executing the selected action, generate a new risk matrix, environmental state, and updated data fusion results; The temporal target of the path planning is calculated based on the temporal difference algorithm, and the loss function is calculated based on the temporal target and the output of the pre-built critic network. The parameters of the critic network are updated based on the obtained loss function and the gradient descent method; Iteratively update and output the optimal path.

7. The method according to claim 1, wherein: The relative distance between ships or between ships and shore is determined based on the shore-based sensor signal to generate an early warning signal.

8. A ship collision risk warning system, characterized in that: include: a data acquisition module configured to acquire environmental data and video data within the harbor, and pre-process the environmental data and the video data respectively; A target recognition module is configured to perform target recognition and tracking on the pre-processed video data using a deep reinforcement learning algorithm to determine dynamic targets; a data fusion module configured to perform multi-source data fusion based on the dynamic target data, environmental data, radar positioning data, and AIS data to obtain a fusion result; The path planning module is configured to generate an optimal navigation path based on the fusion result and through a path planning algorithm, and control the auxiliary berthing of the ship according to the optimal navigation path and the collision warning signal.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the program.

10. A non-transitory computer-readable storage medium, characterized in that in, The non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.

Images