A ferry early warning and collision avoidance implementation method based on video stream analysis

Abstract

The invention discloses a ferry early warning and collision avoidance implementation method based on video stream analysis, which relates to the technical field of ferry early warning. The method comprises the following steps: using a target detection algorithm to detect collision targets on a preprocessed video stream, and obtaining position information of the collision targets; calculating the density of the collision targets, and if the density exceeds expectations, collecting the operation status data of the target equipment and calculating the coordination degree, and if the coordination degree is not lower than expectations, using a ferry collision digital twin model to predict the movement of the ferry and the collision targets, and obtaining corresponding prediction data; calculating the time taken for the risk targets to enter the safe distance of the ferry, and constructing a ferry navigation risk data set after aggregation, and constructing a navigation risk coefficient from the ferry navigation risk data set. If the risk coefficient exceeds a safety threshold, an automatic avoidance strategy is provided for the ferry; by providing an emergency strategy, the heading and speed of the ferry are adjusted, and possible collision risks are avoided, thereby ensuring the safety of the ferry navigation.

Description

Technical Field

[0001] The invention relates to the technical field of ferry warning, and in particular to a method for realizing ferry warning and collision avoidance based on video stream analysis. Background Art

[0002] Ferries can be used to transport passengers as well as cargo (including dangerous goods, livestock, and even vehicles). Ferries can be scheduled or they can only depart when they are full. Among them, passenger ferries are used to carry passengers and their belongings across rivers, lakes, and straits, and some also carry non-motor vehicles and small motor vehicles. Car ferries are used to carry cars across rivers, lakes, and straits, and there are two types: end-to-end and side-to-side. Train ferries are used to carry railway vehicles across rivers and straits. Railway ferries are developed based on traditional train ferries, and are actually multi-purpose ships that carry trains and passengers across straits.

[0003] In the Chinese invention patent with application publication number CN109064775A, a ship collision warning method based on photoelectric equipment is disclosed, including a channel trajectory image acquisition unit, the channel trajectory image acquisition unit is connected to the trajectory image analysis unit, the trajectory image analysis unit includes a target positioning module, a target tracking module, and a trajectory feature extraction module, the target positioning module is electrically connected to the target tracking module, the target tracking module is electrically connected to the trajectory feature extraction module, the trajectory image analysis unit is electrically connected to the trajectory preprocessing unit, the trajectory preprocessing unit includes a front-window calibration module, an image compression module and an image filtering module, and the sensor calibration module is electrically connected to the image compression module; the advantage of the present invention is that it allows users to intuitively obtain ship collision information through video information.

[0004] Combined with the above application and the contents of the prior art:

[0005] When a ferry is sailing on a waterway, it must avoid or maintain a certain distance from objects that may cause collision risks, such as ships and buoys on the waterway, to avoid collisions and ensure the safety of the ferry's navigation. However, when visibility on the waterway is low and the number of ships is large, the risk of collision of the ferry will greatly increase. In order to perform emergency avoidance when a collision may occur, the existing method is to identify the acquired video image information, and then screen out the target objects that may cause the next collision through target detection. After determining the trajectory and speed of the target object, the current course and speed of the ferry are readjusted to achieve the final emergency avoidance. However, when performing emergency avoidance under navigation conditions with low visibility and high ship density, it is difficult for the ship avoidance system to quickly screen out risk targets with a higher collision risk. Moreover, after screening out the risk targets and executing the avoidance strategy, if the course and speed of the risk targets cannot be accurately predicted, it will also cause the emergency avoidance to be difficult to achieve the expected effect.

[0006] To this end, the present invention provides a method for realizing ferry early warning and collision avoidance based on video stream analysis. Summary of the invention

[0007] 1. Technical issues to be solved

[0008] In view of the shortcomings of the prior art, the present invention provides a ferry early warning and collision avoidance implementation method based on video stream analysis. The density of the collision target is calculated. If the density exceeds the expectation, the operating status data of the target equipment is collected and the coordination degree is calculated. If the coordination degree is not lower than the expectation, the movement of the ferry and the collision target is predicted using the ferry collision digital twin model to obtain the corresponding prediction data. The time taken for the risk target to enter the safe distance of the ferry is calculated, and the ferry navigation risk data set is constructed after aggregation. The navigation risk coefficient is constructed from the ferry navigation risk data set. If the risk coefficient exceeds the safety threshold, an automatic avoidance strategy is provided for the ferry. By providing an emergency strategy, the heading and speed of the ferry are adjusted to avoid possible collision risks, thereby solving the technical problems recorded in the background technology.

[0009] (II) Technical solution

[0010] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for realizing ferry early warning and collision avoidance based on video stream analysis, comprising: monitoring the channel state in the direction of the ferry's advance, constructing a corresponding navigation area condition state data set from the acquired monitoring data, constructing a busy coefficient Fm of the channel from the navigation area condition state data set, and issuing an early warning instruction to the outside if the busy coefficient Fm exceeds expectations;

[0011] The sensor network of the ferry collects data along the ferry's sailing direction, and after acquiring the video stream, uses the target detection algorithm to detect collision targets on the pre-processed video stream to obtain the location information of the collision target;

[0012] A ferry navigation digital twin model is constructed based on sample data, collision targets in the target area are marked on the ferry navigation digital twin model, and the density Mo of collision targets is calculated. If the density Mo exceeds expectations, a device collaborative analysis instruction is issued to the outside.

[0013] Collect the operating status data of the target equipment, and build a target equipment operating data set after aggregation. The coordination degree Xo is calculated from the target equipment operating data set. If the coordination degree Xo is not lower than expected, use the ferry collision digital twin model to predict the movement of the ferry and the collision target and obtain the corresponding prediction data;

[0014] The time taken for the risk target to enter the safe distance of the ferry is calculated, and the ferry navigation risk data set is constructed after aggregation. The navigation risk coefficient Fxs is constructed from the ferry navigation risk data set. If the risk coefficient Fxs exceeds the safety threshold, an automatic avoidance strategy is provided and executed for the ferry to achieve the avoidance of the ferry. The method of constructing the risk coefficient Fxs is as follows: the risk duration Ft is linearly normalized in the following manner:

[0015]

[0016] Where i = 1, 2, ..., p, p is the number of risk targets; weight coefficient: 0 ≤ F 1 ≤1,0≤F 2 ≤1 and F 2 +F 1 =1; Ft i is the risk duration of the ith risk target, is the tolerable value of risk duration.

[0017] Furthermore, the starting point, end point and current location of the ferry are determined according to the ferry mission, and the path between the starting point and the end point is used as the waterway; several monitoring areas are set up on the waterway in front of the ferry, and the current navigation conditions of the waterway are monitored at each monitoring area, and the acquired monitoring data are summarized to construct a corresponding navigation area condition status data set.

[0018] Furthermore, the busy coefficient Fm of the waterway is constructed from the data in the navigation area condition data set, wherein the ship density Cm and visibility Km are linearly normalized, and the corresponding data values ​​are mapped to [0,1], and according to the following formula,

[0019]

[0020] Weight coefficient: 0≤β≤1, 0≤α≤1, and α+β=1; i=1, 2, …M, M is the number of monitoring areas, Cm i is the value of ship density in the i-th monitoring area, Cm is its mean value, is the qualified reference value of ship density, Km i is the visibility value in the ith monitoring area, Km avg is its mean value, It is a qualified reference value for visibility.

[0021] Furthermore, after receiving the early warning command, a distributed sensor network is arranged outside the ferry. After the ferry enters the navigation state, the sensor network is used to collect data along the navigation direction of the ferry, and the collected navigation and related data are pre-processed; after obtaining the pre-processed data, the target detection algorithm is used to perform collision target detection on the pre-processed video stream to identify potential collision targets in the video.

[0022] Furthermore, the detected collision target is tracked, and its motion trajectory and speed are analyzed and obtained; the data collected and pre-processed by the sensor network are summarized, and a waterway status data set is constructed after data fusion; some data is extracted from the waterway status data set as sample data, and the sample data is used to train and construct a digital twin model of ferry navigation;

[0023] The monitoring area in front of the ferry is taken as the target area, and the collision targets in the target area are marked on the ferry navigation digital twin model. The density Mo of the collision targets is calculated based on their position information in the following way:

[0024]

[0025] Among them, k is the number of collision targets, S ij is the shortest distance from the ith collision target to the jth collision target, S avg is the mean of the shortest distances.

[0026] Furthermore, after receiving the equipment collaborative analysis instruction, the corresponding equipment participating in the execution of the avoidance strategy is taken as the target equipment, and the operating status data of the target equipment is collected, and a target equipment operating data set is constructed after aggregation; the coordination degree Xo between the target equipment is calculated based on the data in the target equipment operating data set. If the coordination degree Xo is lower than the preset coordination degree threshold, the distributed control system is redesigned; if the coordination degree Xo is not lower than expected, a risk assessment instruction is issued to the outside.

[0027] Furthermore, the coordination degree Xo is constructed as follows: after linear normalization of the response time Xp and the task achievement rate Dp when the target device is used, the corresponding data values ​​are mapped in the interval [0,1] according to the following formula:

[0028]

[0029] Where i = 1, 2, ... m, m is the number of target devices; k 2 , k 1 is the weight coefficient, 0≤k 1 ≤1,0≤k 2 ≤1, and k 1 +k 2 =1,Xp iis the response time of the i-th target device, Xp avg is the mean response time; Dp i The task achievement rate of the i-th target device, Dp avg is the mean of the task completion rate.

[0030] Furthermore, after receiving the risk assessment instruction, the ferry navigation data, channel water flow data, and the speed and trajectory data of the collision target are collected, and then the ferry collision digital twin model is trained to obtain the ferry collision digital twin model; the ferry collision digital twin model is used to predict the movement of the ferry and the collision target, and the driving trajectory and corresponding driving speed between the ferry and the collision target are obtained.

[0031] Furthermore, a safe distance is pre-set for the ferry. If the distance between the collision target and the ferry is within the safe distance, the collision target is taken as a risk target. The time taken for each risk target to enter the safe distance of the ferry is calculated based on the predicted driving trajectory and driving speed. The time is taken as the risk time Ft, and the risk time of each risk target is summarized to construct a ferry navigation risk data set.

[0032] Furthermore, a navigation risk coefficient Fxs is constructed from the predicted data in the ferry navigation risk data set. If the risk coefficient Fxs exceeds the safety threshold, an alarm command is issued to the outside. The corresponding dynamic programming model is constructed by dynamic programming algorithm training. After receiving the alarm command, the dynamic programming model provides an automatic avoidance strategy for the ferry, including adjusting the current navigation path, direction and speed of the ferry; executing the automatic avoidance strategy, and outputting the automatic avoidance strategy to be executed to the risk target within a safe distance.

[0033] (III) Beneficial effects

[0034] The present invention provides a method for realizing ferry early warning and collision avoidance based on video stream analysis, which has the following beneficial effects:

[0035] 1. Evaluate whether the waterway is busy based on the busy coefficient Fm, and estimate the collision risk of the ferry during navigation. If the waterway is currently too busy, there are many ships sailing, and visibility is low, the ships currently sailing in the waterway should be dispatched in time to reduce navigation risks.

[0036] 2. Install a sensor network for the ferry to collect various data of the ferry, identify collision targets in the vicinity of the ferry through target detection, and obtain the location information of each collision target through detection and positioning. After screening out the collision targets, if there is a collision risk at present, it is convenient to handle it in time.

[0037] 3. Through the ferry navigation digital twin model, the driving status of the ferry is predicted and tested, and the risk of ferry collision is judged and evaluated. By tracking each collision target, timely avoidance is facilitated when a collision risk occurs.

[0038] 4. Construct the density Mo of collision targets in the monitoring area. The obtained density Mo can be used to estimate the size of the collision risk of the ferry in the monitoring area. If the density Mo is greater than expected, the speed of the ferry can be reduced. When the speed is reduced to a low level, the current collision risk can be reduced.

[0039] 5. Use the established coordination degree Xo to judge whether each device can respond quickly and whether it can quickly execute the avoidance strategy. If the device cannot be effectively controlled, redesign or install the distributed control system to ensure that it can be executed normally after obtaining the avoidance strategy.

[0040] 6. Train and acquire the ferry collision digital twin model, predict the driving and moving status data of the ferry and the collision target, determine the possibility of collision between the ferry and the collision target based on the acquired driving trajectory and speed, and further screen out risk targets based on the actual possibility of collision, so as to make more accurate treatment.

[0041] 7. According to the size of the risk factor Fxs, the collision risk of the ferry is further evaluated within a safe distance. After gradually narrowing the risk assessment range, the result of the collision risk assessment is more reliable. At this time, an alarm command is issued to remind the ferry driver to ensure the safety of the ferry.

[0042] 8. Use the trained dynamic programming model to provide an automatic avoidance strategy for the ferry. By executing the avoidance strategy, when the collision risk between the ferry and the risk target is higher than expected, the corresponding emergency strategy is quickly given to adjust the ferry's heading and speed, avoid possible collision risks, and ensure the safety of the ferry's navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The present invention is a flow chart of a method for implementing ferry early warning and collision avoidance based on video stream analysis. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] See also Figure 1 The present invention provides a method for realizing ferry early warning and collision avoidance based on video stream analysis, comprising:

[0046] Step 1: monitor the waterway status in the ferry's forward direction, construct a corresponding navigation area condition status data set based on the acquired monitoring data, and construct a busy coefficient Fm of the waterway based on the navigation area condition status data set. If the busy coefficient Fm exceeds expectations, issue an early warning instruction to the outside;

[0047] The step 1 includes the following contents:

[0048] Step 101: If the ferry is about to enter a sailing state, determine the starting point, end point and current location of the ferry according to the ferry mission, take the path between the starting point and the end point as a waterway, and construct an electronic map covering the ferry waterway;

[0049] Several monitoring areas are set up on the channel ahead of the ferry, and the current navigation conditions of the channel are monitored at each monitoring area, including: obtaining the number of ships in each monitoring area and then calculating the ship density, and detecting the visibility in the monitoring area, aggregating the obtained monitoring data, and constructing a corresponding navigation area condition status data set;

[0050] Step 102: construct the busy coefficient Fm of the waterway from the data in the navigation area condition data set, wherein the ship density Cm and visibility Km are linearly normalized, and the corresponding data values ​​are mapped to [0,1], and according to the following formula,

[0051]

[0052] Weight coefficient: 0≤β≤1, 0≤α≤1, and α+β=1. The weight coefficient is obtained by referring to the hierarchical analysis method;

[0053] Where i = 1, 2, ... M, M is the number of monitoring areas, Cm i is the value of ship density in the i-th monitoring area, Cm is its mean value, is the qualified reference value of ship density, Km i is the visibility value in the ith monitoring area, Km avg is its mean value, is the qualified reference value of visibility,

[0054] A busy threshold is pre-set based on historical data and safety management of ferry navigation; if the busy coefficient Fm exceeds expectations, that is, exceeds the busy threshold, it means that when the ferry navigates in the waterway, it is difficult to navigate and the risk of collision is also high. At this time, an early warning instruction is issued to the outside;

[0055] When using, combine the contents in steps 101 and 102:

[0056] When the ferry is about to enter the sailing state, the waterway status data of the monitoring area in front of the ferry is monitored. On the basis of the monitoring data, the ship density and visibility in each monitoring area are used as representative parameters. The busy coefficient Fm in the waterway is constructed according to the representative parameters. The busyness of the waterway is evaluated based on the busy coefficient Fm, and then the collision risk of the ferry during navigation can be estimated. At this time, if the waterway is currently too busy, there are many ships sailing and the visibility is low, the ships currently sailing in the waterway can be dispatched in time to reduce the navigation risk.

[0057] Combined with the above application and the contents of the prior art:

[0058] When a ferry is sailing on a waterway, it must avoid or maintain a certain distance from objects that may cause collision risks, such as ships and buoys on the waterway, to avoid collisions and ensure the safety of the ferry's navigation. However, when visibility on the waterway is low and the number of ships is large, the risk of collision of the ferry will greatly increase. In order to perform emergency avoidance when a collision may occur, the existing method is to identify the acquired video image information, and then screen out the target objects that may cause the next collision through target detection. After determining the trajectory and speed of the target object, the current course and speed of the ferry are readjusted to achieve the final emergency avoidance. However, when performing emergency avoidance under navigation conditions with low visibility and high ship density, it is difficult for the ship avoidance system to quickly screen out risk targets with a higher collision risk. Moreover, after screening out the risk targets and executing the avoidance strategy, if the course and speed of the risk targets cannot be accurately predicted, it will also cause the emergency avoidance to be difficult to achieve the expected effect.

[0059] Step 2: The sensor network of the ferry collects data along the sailing direction of the ferry, and after acquiring the video stream, uses the target detection algorithm to detect the collision target on the pre-processed video stream to obtain the position information of the collision target;

[0060] The step 2 includes the following contents:

[0061] Step 201: after receiving the warning instruction, a distributed sensor network is arranged outside the ferry, including radar, camera, sonar, etc.; a ferry model is constructed according to the shape and specification data of the ferry, and the corresponding sensors are marked on the model; after the ferry enters the navigation state, the sensor network is used to collect data along the navigation direction of the ferry, and the collected navigation and related data are pre-processed, for example, the collected video stream is processed in real time, including compression, format conversion, etc.;

[0062] Step 202: After obtaining the preprocessed data, a collision standard of the collision target is pre-set according to the size of the target object that may cause a collision risk to the ferry, and a target detection algorithm is used to perform collision target detection on the preprocessed video stream to identify various potential collision targets in the video, such as other ships, buoys or other large obstacles, and obtain the position information of the collision target;

[0063] When using, combine the contents in steps 201 to 203:

[0064] When the current waterway is relatively busy and there is a potential risk of collision, a sensor network is installed on the ferry to collect various data of the ferry. Through target detection, collision targets in the vicinity of the ferry are identified, and the location information of each collision target is obtained through detection and positioning. After the collision targets are screened out, if there is a current collision risk, it is convenient to handle it in a timely manner.

[0065] Step 3: construct a ferry navigation digital twin model based on sample data, mark the collision targets in the target area on the ferry navigation digital twin model, and calculate the density Mo of the collision targets. If the density Mo exceeds expectations, issue a device collaborative analysis instruction to the outside.

[0066] The step three includes the following contents:

[0067] Step 301: Use the Kalman filter algorithm or other similar tracking algorithms to track the detected collision target, analyze and obtain its motion trajectory and speed, etc.;

[0068] Aggregate the data collected and pre-processed by the sensor network, as well as the motion trajectory and speed of the collision target, and build a channel status data set after data fusion; extract part of the data from the channel status data set as sample data, and build an initial model with a convolutional neural network. After testing and training the initial model with the sample data, build the corresponding ferry navigation digital twin model;

[0069] When in use, the ferry navigation digital twin model can predict and test the ferry's driving status, and then judge and evaluate the risk of ferry collision. At the same time, by tracking each collision target, it is also convenient to avoid collision in time when there is a risk of collision.

[0070] Step 302: The monitoring area in front of the ferry is taken as the target area, and the collision targets in the target area are marked on the ferry navigation digital twin model. The density Mo of the collision targets is calculated based on the position information of the collision targets in the following manner:

[0071]

[0072] Among them, k is the number of collision targets, S ij is the shortest distance from the ith collision target to the jth collision target, S avg is the mean of the shortest distances.

[0073] Based on historical data and safety management expectations for ferry collision risks, a density threshold is pre-set; if the density Mo exceeds expectations, that is, exceeds the density threshold, then the risk of collision between the ferry and the collision target is high. When a collision actually occurs, it needs to be handled promptly and quickly. In order to improve processing efficiency, a device collaborative analysis instruction is issued to the outside;

[0074] When used, combine the contents in steps 301 and 302:

[0075] When the ferry enters the monitoring area ahead, the location information of the collision targets in the monitoring area is obtained, and then the density Mo of the collision targets in the monitoring area is constructed. The obtained density can be used to estimate the size of the collision risk of the ferry in the monitoring area. If the density Mo is greater than expected, the speed of the ferry can be reduced. When the speed is reduced to a low level, the current collision risk can be reduced.

[0076] Step 4: Collect the operating status data of the target equipment, and construct a target equipment operating data set after aggregation. The coordination degree Xo is calculated from the target equipment operating data set. If the coordination degree Xo is not lower than expected, use the ferry collision digital twin model to predict the movement of the ferry and the collision target, and obtain the corresponding prediction data;

[0077] The step 4 includes the following contents:

[0078] Step 401: In order to reduce the collision risk of the ferry, when receiving the control instruction, it is necessary to make the navigation equipment, engine control and other equipment on the ferry work together, and when controlling several equipments, after receiving the equipment collaborative analysis instruction, the corresponding equipment participating in the execution of the avoidance strategy is taken as the target equipment, the operation status data of the target equipment is collected, and the target equipment operation data set is constructed after being aggregated;

[0079] Step 402: Calculate the coordination degree Xo between target devices from the data in the target device operation data set in the following manner: After linear normalization of the response time Xp and the task achievement rate Dp when the target device is in use, map the corresponding data values ​​in the interval [0,1] according to the following formula:

[0080]

[0081] Where i = 1, 2, ... m, m is the number of target devices; k 2 , k 1 is the weight coefficient, 0≤k1 ≤1,0≤k 2 ≤1, and k 1 +k 2 =1, the weight coefficient can be obtained by referring to the hierarchical analysis method; Xp i is the response time of the i-th target device, Xp avg is the mean response time; Dp i The task achievement rate of the i-th target device, Dp avg is the mean of the task completion rate,

[0082] If the coordination degree Xo is lower than the preset coordination degree threshold, it means that it may be difficult to achieve the expected effect when actually controlling the ferry. In this case, the distributed control system should be redesigned to improve the coordination degree Xo between the target devices. For details, please refer to the following:

[0083] Select a suitable communication protocol: Nodes in a distributed control system need to communicate with each other. Selecting a suitable communication protocol can ensure reliable message delivery and efficient processing.

[0084] Choose a suitable distributed algorithm: According to the system requirements and scale, choose a suitable distributed algorithm, such as consistent hashing algorithm, distributed lock, etc., to ensure the performance and consistency of the distributed system.

[0085] Choose a suitable distributed database: According to the system's data characteristics and access patterns, choose a suitable distributed database, such as a relational database, NoSQL database, etc., to improve data storage and access efficiency.

[0086] System structure: Distributed control systems are generally divided into three layers: process control layer, production monitoring layer and centralized management layer. The process control layer at the bottom layer generally implements data collection and control on-site by decentralized field control stations, data acquisition stations, etc., and transmits data to the production monitoring layer computer through the data communication network. The production monitoring layer centrally monitors and manages the data from the process control layer, and can transmit the data to the centralized management layer for higher-level decision-making and planning.

[0087] Technology selection: Distributed control systems integrate computer technology, network technology, communication technology, and automatic control technology. Therefore, it is necessary to select appropriate technologies and tools based on specific application scenarios and requirements.

[0088] Security considerations: When designing a distributed control system, you also need to consider the security of the system, including data security, communication security, and system security. Appropriate security measures such as data encryption, identity authentication, access control, etc. need to be taken to ensure the safe operation of the system.

[0089] If the coordination degree Xo is not lower than expected, a risk assessment instruction is issued to the outside;

[0090] When in use, when it is judged that the ferry needs to be avoided, the ferry and its various equipment need to respond. If the equipment responds too slowly, the ferry may not be able to effectively execute the avoidance strategy after obtaining it. Therefore, the coordination degree Xo is constructed with the response time and the task achievement rate as representative parameters. The coordination degree Xo is used to judge whether each device can respond quickly and whether it can quickly execute the avoidance strategy. If the device cannot be effectively controlled, the distributed control system is redesigned or installed to ensure that the avoidance strategy can be normally executed after it is obtained.

[0091] Step 403: after receiving the risk assessment instruction, ferry navigation data, channel water flow data, and speed and trajectory data of the collision target are collected and used as sample data. After the initial model is constructed by the machine learning algorithm, the sample data is used to train the initial model to obtain the ferry collision digital twin model; the ferry collision digital twin model is used to predict the movement of the ferry and the collision target, and the driving trajectory and corresponding driving speed between the ferry and the collision target are obtained;

[0092] When used, combine the contents in steps 401 to 403:

[0093] After training and acquiring the ferry collision digital twin model, after determining the monitoring area, that is, the area where the ferry is about to sail, the driving movement status data of the ferry and the collision target are predicted. Based on the obtained driving trajectory and speed, the possibility of collision between the ferry and the collision target can be determined. According to the actual possibility of collision, the risk targets can be further screened out to facilitate more accurate processing.

[0094] Step 5: Calculate the time it takes for the risk target to enter the safe distance of the ferry, and then summarize and construct a ferry navigation risk data set. The navigation risk coefficient Fxs is constructed from the ferry navigation risk data set. If the risk coefficient Fxs exceeds the safety threshold, provide and execute an automatic avoidance strategy for the ferry to achieve the avoidance of the ferry.

[0095] The step five includes the following contents:

[0096] Step 501: pre-set a safety distance for the ferry. If the distance between the collision target and the ferry is within the safety distance, the collision target is regarded as a risk target.

[0097] The time taken for each risk target to enter the safe distance of the ferry is calculated based on the predicted driving trajectory and driving speed, and the time is taken as the risk time Ft. The shorter the risk time, the greater the risk of collision. The risk time of each risk target is summarized to construct a ferry navigation risk data set.

[0098] When in use, a safety distance is set in advance. After the ferry enters the safety distance, the collision target that enters the safety distance is regarded as a risk target, so that emergency treatment can be carried out in time to reduce the probability of collision.

[0099] Step 502: construct a navigation risk coefficient Fxs from the predicted data in the ferry navigation risk data set, and use the navigation risk coefficient Fxs to evaluate the current potential collision risk of the ferry, wherein the risk duration is linearly normalized, and then in the following manner:

[0100]

[0101] Where i = 1, 2, ..., p, p is the number of risk targets; weight coefficient: 0 ≤ F 1 ≤1,0≤F 2 ≤1 and F 2 +F 1 =1; Ft i is the risk duration of the ith risk target, is the tolerable value of risk duration;

[0102] A safety threshold is pre-set based on historical data and management expectations of ferry navigation safety risks. If the risk factor Fxs exceeds the safety threshold, it means that the current risk of collision is high and needs to be handled in a timely manner. At this time, the alarm system is automatically triggered to issue an alarm command to the outside. The alarm command can be a visual signal, a sound prompt, or directly linked to the ferry's navigation system for early warning.

[0103] When in use, the movement of the risk target is predicted, and the risk coefficient Fxs is constructed from the predicted data. According to the size of the risk coefficient Fxs, the collision risk of the ferry is further evaluated within a safe distance. After gradually narrowing the risk assessment range, the result of the collision risk assessment is made more reliable, and an alarm command is issued at this time to remind the ferry driver to ensure the safety of the ferry.

[0104] Step 503: A corresponding dynamic programming model is constructed by training a dynamic programming algorithm. After receiving an alarm instruction, the dynamic programming model provides an automatic avoidance strategy for the ferry, including adjusting the current navigation path, direction and speed of the ferry; executing the automatic avoidance strategy, and outputting the automatic avoidance strategy to be executed to the risk target within a safe distance;

[0105] When using, combine the contents in steps 501 to 503:

[0106] After receiving the alarm command, based on the prediction data output by the ferry collision digital twin model, the trained dynamic programming model is used to provide an automatic avoidance strategy for the ferry. By executing the avoidance strategy, when the collision risk between the ferry and the risk target is higher than expected, the corresponding emergency strategy is quickly given to adjust the ferry's heading and speed, avoid possible collision risks, and ensure the safety of the ferry's navigation.

[0107] It should be noted that the Analytic Hierarchy Process (AHP) is a decision-making method that decomposes complex problems into multiple levels or factors and evaluates the relative importance of these factors through qualitative and quantitative analysis. This method was proposed by American operations researcher Satie in the early 1970s and is mainly used to solve decision-making problems of target systems with hierarchical and staggered evaluation indicators. The basic steps of the AHP include:

[0108] Establish a hierarchical model: divide the decision-making goals, factors to be considered (decision-making criteria) and decision-making objects into the highest level (goal level), middle level (criteria level) and lowest level (plan level) according to their mutual relationships, and draw a hierarchical diagram.

[0109] Construct a judgment (paired comparison) matrix: When determining the weights between factors at each level, use a pairwise comparison method to assess the levels according to a relative scale, and construct a judgment matrix based on this.

[0110] Hierarchical single sorting and its consistency test: By solving the eigenvector of the judgment matrix, the weight of each factor relative to its upper-level factor is obtained, and a consistency test is performed to ensure the reliability of the results.

[0111] Hierarchical total ranking and its consistency test: Based on the results of hierarchical single ranking, calculate the weight of the lowest level factor relative to the highest level target, and perform a consistency test on the total ranking.

[0112] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0113] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only some logical function divisions. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0116] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0118] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0119] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0120] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A ferry early warning and collision avoidance implementation method based on video stream analysis, characterized in that: include, Monitor the waterway status in the ferry's forward direction, construct a corresponding navigation area condition status data set based on the acquired monitoring data, and construct the busy coefficient Fm of the waterway based on the navigation area condition status data set. If the busy coefficient Fm exceeds expectations, issue an early warning instruction to the outside; The sensor network of the ferry collects data along the ferry's sailing direction, and after acquiring the video stream, uses the target detection algorithm to detect collision targets on the pre-processed video stream to obtain the location information of the collision target; A ferry navigation digital twin model is constructed based on sample data, collision targets in the target area are marked on the ferry navigation digital twin model, and the density Mo of collision targets is calculated. If the density Mo exceeds expectations, a device collaborative analysis instruction is issued to the outside. Collect the operating status data of the target equipment, and build a target equipment operating data set after aggregation. The coordination degree Xo is calculated from the target equipment operating data set. If the coordination degree Xo is not lower than expected, use the ferry collision digital twin model to predict the movement of the ferry and the collision target and obtain the corresponding prediction data; The time taken for the risk target to enter the safe distance of the ferry is calculated, and the ferry navigation risk data set is constructed after aggregation. The navigation risk coefficient Fxs is constructed from the ferry navigation risk data set. If the risk coefficient Fxs exceeds the safety threshold, an automatic avoidance strategy is provided and executed for the ferry to achieve the avoidance of the ferry. The method of constructing the risk coefficient Fxs is as follows: the risk duration Ft is linearly normalized in the following manner: Where i = 1, 2, ..., p, p is the number of risk targets; weight coefficient: 0 ≤ F1 ≤ 1, 0 ≤ F2 ≤ 1 and F2 + F1 = 1; Ft i is the risk duration of the ith risk target, is the tolerable value of risk duration.

2. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 1 is characterized in that: The starting point, end point and current location of the ferry are determined according to the ferry mission, and the path between the starting point and the end point is used as the waterway; several monitoring areas are set up on the waterway in front of the ferry, and the current navigation conditions of the waterway are monitored in each monitoring area. The acquired monitoring data are summarized to construct a corresponding navigation area condition status data set.

3. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 1 is characterized in that: The busy coefficient Fm of the waterway is constructed from the data in the navigation area condition data set, where the ship density Cm and visibility Km are linearly normalized, and the corresponding data values ​​are mapped to [0,1], and according to the following formula, Weight coefficient: 0≤β≤1, 0≤α≤1, and α+β=1; i=1, 2, …M, M is the number of monitoring areas, Cm i is the value of ship density in the i-th monitoring area, Cm is its mean value, is the qualified reference value of ship density, Km i is the visibility value in the ith monitoring area, Km avg is its mean value, It is a qualified reference value for visibility.

4. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 3 is characterized in that: After receiving the early warning command, a distributed sensor network is arranged outside the ferry. After the ferry enters the navigation state, the sensor network is used to collect data along the navigation direction of the ferry, and the collected navigation and related data are pre-processed; after obtaining the pre-processed data, the target detection algorithm is used to perform collision target detection on the pre-processed video stream to identify potential collision targets in the video.

5. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 4 is characterized in that: Track the detected collision targets, analyze and obtain their motion trajectory and speed; summarize the data collected and pre-processed by the sensor network, and build a channel status data set after data fusion; extract part of the data from the channel status data set as sample data, and use the sample data to train and build a digital twin model of ferry navigation; The monitoring area in front of the ferry is taken as the target area, and the collision targets in the target area are marked on the ferry navigation digital twin model. The density Mo of the collision targets is calculated based on their position information in the following way: Among them, k is the number of collision targets, S ij is the shortest distance from the ith collision target to the jth collision target, S avg is the mean of the shortest distances.

6. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 5 is characterized in that: After receiving the equipment collaborative analysis instruction, the corresponding equipment participating in the execution of the avoidance strategy is taken as the target equipment, and the operating status data of the target equipment is collected. After summarizing, a target equipment operating data set is constructed; the coordination degree Xo between the target equipment is calculated based on the data in the target equipment operating data set. If the coordination degree Xo is lower than expected, the distributed control system is redesigned. If the coordination degree Xo is not lower than expected, a risk assessment instruction is issued to the outside.

7. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 6 is characterized in that: The coordination degree Xo is constructed as follows: After linear normalization of the response time Xp and the task achievement rate Dp when the target device is used, the corresponding data values ​​are mapped in the interval [0,1] according to the following formula: Where i = 1, 2, ... m, m is the number of target devices; k2, k1 are weight coefficients, 0≤k1≤1, 0≤k2≤1, and k1+k2=1, Xp i is the response time of the i-th target device, Xp avg is the mean response time; Dp i The task achievement rate of the i-th target device, Dp avg is the mean of the task completion rate.

8. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 7 is characterized in that: After receiving the risk assessment instruction, the ferry navigation data, channel water flow data, and the speed and trajectory data of the collision target are collected, and then the ferry collision digital twin model is trained to obtain the ferry collision digital twin model; the ferry collision digital twin model is used to predict the movement of the ferry and the collision target, and the driving trajectory and corresponding driving speed between the ferry and the collision target are obtained.

9. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 8, characterized in that: A safe distance is pre-set for the ferry. If the distance between the collision target and the ferry is within the safe distance, the collision target is taken as a risk target. The time taken for each risk target to enter the safe distance of the ferry is calculated based on the predicted driving trajectory and driving speed. The time is taken as the risk duration Ft. The risk duration of each risk target is summarized to construct a ferry navigation risk data set.

10. The method for realizing ferry early warning and collision avoidance based on video stream analysis according to claim 9, characterized in that: The navigation risk coefficient Fxs is constructed from the predicted data in the ferry navigation risk data set. If the risk coefficient Fxs exceeds the safety threshold, an alarm instruction is issued to the outside. The corresponding dynamic programming model is constructed by dynamic programming algorithm training. After receiving the alarm command, the dynamic programming model provides an automatic avoidance strategy for the ferry, including adjusting the current navigation path, direction and speed of the ferry; executing the automatic avoidance strategy, and outputting the automatic avoidance strategy to be executed to the risk target within a safe distance.

Images