Ship safe navigation monitoring method based on video images

Abstract

The invention discloses a ship safe navigation monitoring method based on a video image, and the method comprises the steps: 1, obtaining a video frame in real time through a ship-borne camera, and carrying out the preprocessing of the video frame; step 2, identifying and positioning a target position in a video frame through a fusion positioning algorithm; 3, calculating the distance between the ship and the target according to the detected target position; 4, track prediction is carried out on the detected target, and the future position and the motion path of the target are estimated; 5, calculating potential collision time according to the relative positions of the ship and the target and the speed and trajectory of the target, and if the potential collision time is smaller than a preset safety threshold value, judging that a collision risk exists; and step 6, when a potential collision risk is detected, the system generates an early warning signal to remind a ship driver to take actions. According to the invention, the accuracy of target identification on the ship route is improved, so that the shipping safety is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of ship intelligent management, and particularly relates to a method for monitoring the safe navigation of ships based on video images. Background Art

[0002] With the linear increase in water surface transport capacity and volume, ship intelligent navigation systems are becoming increasingly important. However, due to the special and complex water surface environment, there is currently a lack of effective technical means for ship intelligent safety supervision, and the problem of ship safe navigation is becoming increasingly prominent.

[0003] The main technical difficulty is that due to weather conditions or the characteristics of the target itself being not obvious, the accuracy of target recognition on the ship's navigation route in the prior art is insufficient, and accurate, effective, and timely target positioning cannot be carried out, and thus recognition and early warning judgment results cannot be given. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for monitoring the safe navigation of ships based on video images for improving the accuracy of target recognition on the ship's route and thus enhancing the shipping safety in view of the deficiencies in the above prior art.

[0005] To solve the above technical problem, the present invention is based on the technologies and systems of traditional ship collision prevention monitoring, adds cameras, and uses computer vision technology to analyze the video stream of the ship for real-time monitoring and early warning. These systems can be installed on the ship to provide real-time visual feedback to help judge the distance and position between the ship and the dock and between ships, and to record the water surface conditions and waterborne ship information in real time by video.

[0006] Based on a proper association with the Tesla collision prevention system, multiple cameras, ultrasonic sensors, and forward radars around the ship are used to detect and avoid collisions.

[0007] At the same time, a positioning algorithm for multi-feature fusion recognition is adopted to greatly improve the accuracy of target recognition and enhance the reliability of the ship navigation monitoring system.

[0008] Specifically, the present invention provides a method for monitoring the safe navigation of ships based on video images, including the following steps:

[0009] Step 1, image acquisition: real-time video frames are obtained through an on-board camera, and the video frames are preprocessed;

[0010] Step 2, target detection: the target position is recognized and located in the video frame through a fusion positioning algorithm;

[0011] Step 3, distance estimation: according to the detected target position, the distance between the ship and the target is calculated;

[0012] Step 4, Trajectory prediction: Perform trajectory prediction on the detected target to estimate the future position and movement path of the target;

[0013] Step 5, Collision risk assessment: Calculate the potential collision time based on the relative position, target speed, and trajectory of the ship and the target. If the potential collision time is less than the preset safety threshold, it is determined that there is a collision risk;

[0014] Step 6, Warning generation: When a potential collision risk is detected, the system generates a warning signal to remind the ship's driver to take action.

[0015] Preferably, in the said Step 1, the preprocessing of the video frame includes: image scaling, denoising, and color space conversion.

[0016] Preferably, in the said Step 2, the fusion positioning algorithm includes:

[0017] Step 2.1, Extract the chromaticity value of the video frame and establish a target color feature histogram;

[0018] Step 2.2, Construct the texture image of the video frame and establish a target texture feature histogram;

[0019] Step 2.3, According to the gradient direction of the edge pixel points in the video frame, and establish a target shape feature histogram;

[0020] Step 2.4, According to the target color feature, texture feature, and shape feature, construct a target fusion probability distribution map H(i, j) to realize the recognition and positioning of the target in the video frame, where i represents the video frame row number and j represents the target row number;

[0021] Step 2.5, Continuously update the target fusion probability distribution map H(i, j) in the video frame to realize the positioning of the target in the video.

[0022] Preferably, in the said Step 2.4, the steps of constructing the target fusion probability distribution map include:

[0023] Step 2.4.1, According to the target color feature, calculate the probability distribution map C(i, j) of the target color feature appearing in the video frame;

[0024] Step 2.4.2, According to the target texture feature, calculate the probability distribution map T(i, j) of the target texture feature appearing in the video frame;

[0025] Step 2.4.3, According to the target shape feature, calculate the probability distribution map S(i, j) of the target shape feature appearing in the video frame;

[0026] Step 2.4.4, The way to construct the target fusion probability distribution map is:

[0027] H(i, j) = uC(i, j) + vT(i, j) + wS(i, j)

[0028] Wherein, u is the fusion coefficient of the probability distribution map C(i, j), v is the fusion coefficient of the probability distribution map T(i, j), w is the fusion coefficient of the probability distribution map S(i, j), and u, v, w ∈ [0, 1].

[0029] Preferably, in the step 2.5, the method for continuously updating the fusion probability distribution map H(i, j) of the target in the video frame is to continuously update the magnitudes of u, v, and w according to the real-time target color feature, texture feature, and shape feature of the target.

[0030] Preferably, let Where A i,j is the cumulative difference of the color feature histogram of the j-th target in the i-th video frame, and ΔA is the set value of the cumulative difference of the color feature histogram; B i,j,l is the l-th histogram vertical coordinate amplitude in the color feature histogram of the j-th target in the i-th video frame, p is the number of histograms in the color feature histogram, ΔB is the set value of the single histogram variance in the color feature histogram; k1, k2 are weight coefficients;

[0031] Preferably, let Where A' i,j is the cumulative difference of the texture feature histogram of the j-th target in the i-th video frame, and ΔA' is the set value of the cumulative difference of the texture feature histogram; B' i,j,l is the l-th histogram vertical coordinate amplitude in the texture feature histogram of the j-th target in the i-th video frame, p is the number of histograms in the texture feature histogram, ΔB' is the set value of the single histogram variance in the texture feature histogram; k3, k4 are weight coefficients;

[0032] Preferably, let Where A'' i,j is the cumulative difference of the shape feature histogram of the j-th target in the i-th video frame, and ΔA'' is the set value of the cumulative difference of the shape feature histogram; B'' i,j,l is the l-th histogram vertical coordinate amplitude in the shape feature histogram of the j-th target in the i-th video frame, p is the number of histograms in the shape feature histogram, ΔB'' is the set value of the single histogram variance in the shape feature histogram; k5, k6 are weight coefficients;

[0033] Preferably, u=k7(u'+1-v'-w'), v=k8(v'+1-u'-w'), w=k9(w'+1-u'-v'), and k7, k8 and k9 are weight coefficients.

[0034] Preferably, in step 3, a camera is used to measure the distance of the target object. The monocular camera can calculate the distance based on the known size of the target and the size in the video frame; the binocular camera can calculate the depth information using the parallax principle; in step 4, a Kalman filter or a prediction model is used to estimate the future position and motion path of the target.

[0035] The beneficial effects of the present invention are:

[0036] The present invention provides a method for monitoring safe navigation of ships based on video images, which solves the problems of poor data availability, low image quality, unclear image quality, low data utilization efficiency, lack of networking function, poor system reliability, etc. of traditional safety systems.

[0037] The system of the present invention uses a new image algorithm and can be redundant with the ship's own sensors to achieve double-layer protection. It is more intelligent and can identify targets and automatically track targets. It can be connected to the network, upload data in real time, and conduct remote monitoring, which can be carried out 24 hours a day. It can provide richer visual information and higher image quality, which helps the crew better understand the surrounding environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the configuration of the acquisition module of the present invention;

[0039] Figure 2 The present invention is a flow chart of a method for monitoring safe navigation of a ship based on video images. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below in conjunction with embodiments so that those skilled in the art can implement the invention with reference to the description.

[0041] It should be understood that the terms such as “having”, “including” and “comprising” used herein do not exclude the existence or addition of one or more other elements or combinations thereof.

[0042] The test methods used in the following examples are conventional methods unless otherwise specified. The materials and reagents used in the following examples are all commercially available unless otherwise specified. In the following examples, if no specific conditions are specified, the experiments were carried out under conventional conditions or conditions recommended by the manufacturer. The reagents or instruments used, if the manufacturer is not specified, are all conventional products that can be purchased commercially.

[0043] The present invention provides a method for monitoring the safe navigation of ships based on video images, comprising the following steps:

[0044] Step 1, image acquisition: Real-time video frames are obtained through an on-board camera, and preprocessing is performed on the video frames, such as image scaling, denoising, and color space conversion (e.g., conversion from RGB to grayscale image or HSV). As Figure 1 shown, a plurality of acquisition modules 2 are arranged around the hull 1. The acquisition module 2 includes a camera, a sensor, and a radar to improve the accuracy and effectiveness of data materials;

[0045] Step 2, target detection: The target position is identified and located in the video frame through a fusion positioning algorithm. The targets include docks, obstacles, and other ships navigating on the navigation route, etc.;

[0046] Step 3, distance estimation: According to the detected target position, the distance between the ship and the target is calculated; a monocular or binocular camera is used to measure the distance of an object. For a monocular camera, the distance can be calculated based on the known size of the target and its size in the image; for a binocular camera, the depth information can be calculated using the parallax principle. Another method is to directly estimate the distance from a monocular image using a deep learning model (such as DNN-based depth estimation).

[0047] Step 4, trajectory prediction: Trajectory prediction is performed on the detected target to estimate the future position and movement path of the target; trajectory prediction is performed on the detected target, and a Kalman filter or other prediction model is used to estimate the future position and movement path of the target.

[0048] Step 5, collision risk assessment: According to the relative position, target speed, and trajectory of the ship and the target, the potential collision time is calculated. If the potential collision time is less than a preset safety threshold, it is determined that there is a collision risk;

[0049] Step 6, warning generation: When a potential collision risk is detected, the system generates a warning signal to remind the driver to take action through sound, visual cues, or other means.

[0050] In the above technical solution, in step 2, the fusion positioning algorithm includes:

[0051] Step 2.1, extract the chromaticity value of the video frame and establish a target color feature histogram;

[0052] Step 2.2, construct the texture image of the video frame and establish a target texture feature histogram;

[0053] Step 2.3: Based on the gradient directions of the edge pixel points in the video frame, construct a target shape feature histogram. The construction methods of the above three histograms are prior arts and will not be elaborated here.

[0054] Step 2.4: Construct a fusion probability distribution map H(i, j) of the target according to the target color feature, texture feature, and shape feature to achieve the recognition and positioning of the target in the video frame, where i represents the video frame sequence number and j represents the target sequence number.

[0055] Step 2.5: Continuously update the fusion probability distribution map H(i, j) of the target in the video frame to achieve the positioning of the target in the video.

[0056] In the prior art, a single feature recognition method is usually adopted for target recognition. Specifically, the distribution probability of the target in the video image is calculated based on a single feature to determine the position of the target. However, due to weather reasons or the low recognition degree of the target itself, etc., it is impossible to accurately and effectively recognize multiple targets. Therefore, the present invention proposes a target recognition method that fuses multiple recognition features to solve the problems in the prior art.

[0057] Specifically, the present invention combines and uses the target color feature, texture feature, and shape feature, greatly reducing the difficulty of recognizing the target features caused by external factor interference, complementing the three recognition features, and improving the recognition accuracy of the target. The principle is: calculate the difference between the histograms of a single recognition feature in the previous and the next two video frames, and judge the stability and effectiveness under this recognition feature. If the difference between the front and the back is large, it indicates that the accuracy under this recognition feature mode is poor, and the weight of this index feature in the fusion algorithm is lowered. On the contrary, the weight of this index feature in the fusion algorithm is increased, thereby ensuring the overall recognition accuracy of the target.

[0058] In the said Step 2.4, the specific steps for constructing the target fusion probability distribution map include:

[0059] Step 2.4.1: Calculate the probability distribution map C(i, j) of the target color feature appearing in the video frame according to the target color feature.

[0060] Step 2.4.2: Calculate the probability distribution map T(i, j) of the target texture feature appearing in the video frame according to the target texture feature.

[0061] Step 2.4.3: Calculate the probability distribution map S(i, j) of the target shape feature appearing in the video frame according to the target shape feature.

[0062] Step 2.4.4: The method for constructing the target fusion probability distribution map is:

[0063] H(i, j) = uC(i, j) + vT(i, j) + wS(i, j)

[0064] Where u is the fusion coefficient of the probability distribution map C(i, j), v is the fusion coefficient of the probability distribution map T(i, j), w is the fusion coefficient of the probability distribution map S(i, j), and u, v, w ∈ [0, 1]. After obtaining the target fusion probability distribution map, the target can be recognized in the video frame.

[0065] The target fusion probability distribution map fuses the probability distributions of three recognition features. Even if the recognition accuracy of any feature is reduced due to environmental reasons, it does not affect the overall recognition accuracy of the target. For example, when it is difficult to recognize the target by texture features in the video frame, according to the fusion coefficient algorithm, the probability of the probability distribution map T(i, j) of the texture features appearing in the video frame will be automatically reduced, while the probability distribution maps C(i, j) of the target color features and S(i, j) of the shape features appearing in the video frame will be increased; the overall stability and accuracy of the fusion probability distribution map can be stabilized through the fusion coefficient algorithm.

[0066] In step 2.5, the method of continuously updating the fusion probability distribution map H(i, j) of the target in the video frame is to continuously update the magnitudes of u, v, and w according to the real-time target color features, texture features, and shape features of the target, that is, continuously update the target fusion probability distribution map, which realizes the continuous recognition of the target. Furthermore, the future position, movement path, distance from the ship, and potential collision time of the target can be estimated through a Kalman filter or other prediction models.

[0067] Let where A i,j is the cumulative difference of the color feature histogram of the j-th target in the i-th video frame, and ΔA is the set value of the cumulative difference of the color feature histogram; B i,j,l is the l-th histogram vertical coordinate amplitude in the color feature histogram of the j-th target in the i-th video frame, p is the number of histograms in the color feature histogram, and ΔB is the set value of the single histogram variance in the color feature histogram; k1, k2 are weight coefficients.

[0068]

[0069] Let where A' i,j is the cumulative difference of the texture feature histogram of the j-th target in the i-th video frame, and ΔA' is the set value of the cumulative difference of the texture feature histogram; B' i,j,lis the lth histogram ordinate amplitude of the jth target in the texture feature histogram in the i-th video frame, p is the number of histograms in the texture feature histogram, ΔB' is the set value of a single histogram difference in the texture feature histogram; k3 and k4 are weight coefficients;

[0070]

[0071] set up Among them, A i,j is the cumulative difference of the shape feature histogram of the jth target in the i-th video frame, ΔA” is the set value of the cumulative difference of the shape feature histogram; B” i,j,l is the lth histogram ordinate amplitude of the jth target in the shape feature histogram in the i-th video frame, p is the number of histograms in the shape feature histogram, ΔB” is the set value of a single histogram difference in the shape feature histogram; k5 and k6 are weight coefficients;

[0072]

[0073] In step 2.4.4, u=k7(u'+1-v'-w'), v=k8(v'+1-u'-w'), w=k9(w'+1-u'-v'), k7, k8 and k9 are weight coefficients, so that a real-time updated fusion probability distribution map can be obtained for target recognition and effective tracking.

[0074] As mentioned above, the three identification features in the target fusion probability distribution map are complementary and associated to stabilize the stability and accuracy of the target fusion probability distribution map, and ultimately improve the recognition accuracy of the target, so as to accurately grasp the target's trajectory, calculate the distance between the ship and the target and the potential collision time, issue an early warning in time, and realize the safe navigation monitoring of the ship.

[0075] The present invention provides a method for monitoring safe navigation of ships based on video images, which solves the problems of poor data availability, low image quality, unclear image quality, low data utilization efficiency, lack of networking function, poor system reliability, etc. of traditional safety systems.

[0076] The system of the present invention uses a new image algorithm and can be redundant with the ship's own sensors to achieve double-layer protection. It is more intelligent and can identify targets and automatically track targets. It can be connected to the network, upload data in real time, and conduct remote monitoring, which can be carried out 24 hours a day. It can provide richer visual information and higher image quality, which helps the crew better understand the surrounding environment.

[0077] Although the embodiments of the present invention have been disclosed as above, they are not limited to the applications listed in the specification and embodiments. It can be fully applied to various fields suitable for the present invention. For those skilled in the art, additional modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to specific details.

Claims

1. A method for monitoring safe navigation of a ship based on video images, characterized in that: The following steps are involved: Step 1: Image acquisition: obtaining video frames in real time through the ship-borne camera and preprocessing the video frames; Step 2: Target detection: identifying and locating the target position in the video frame by fusion positioning algorithm; Step 3: Distance estimation: Calculate the distance between the ship and the target based on the detected target position; Step 4: trajectory prediction: perform trajectory prediction on the detected target and estimate the target's future position and motion path; Step 5: Collision risk assessment: calculate the potential collision time based on the relative position of the ship and the target, the target speed and trajectory. If the potential collision time is less than the preset safety threshold, it is determined that there is a collision risk; Step 6: Warning generation: When a potential collision risk is detected, the system generates a warning signal to remind the ship driver to take action.

2. The method for monitoring safe navigation of a ship based on video images according to claim 1, characterized in that: In the step 1, preprocessing the video frames includes: image scaling, denoising and color space conversion.

3. The method for monitoring safe navigation of a ship based on video images according to claim 1, characterized in that: In step 2, the fusion positioning algorithm includes: Step 2.1, extracting the chromaticity value of the video frame and establishing a target color feature histogram; Step 2.2, constructing a texture image of the video frame and establishing a texture feature histogram of the target; Step 2.3, establishing a target shape feature histogram according to the gradient direction of the edge pixel points in the video frame; Step 2.4, constructing a fusion probability distribution map H(i, j) of the target based on the target color features, texture features and shape features, to achieve recognition and positioning of the target in the video frame, where i represents the video frame ranking and j represents the target ranking; Step 2.5: Continuously update the fusion probability distribution map H(i, j) of the target in the video frame to achieve the positioning of the target in the video.

4. The method for monitoring safe navigation of a ship based on video images according to claim 3 is characterized in that: In step 2.4, the step of constructing the target fusion probability distribution map includes: Step 2.4.1, according to the target color feature, calculate the probability distribution map C(i, j) of the target color feature appearing in the video frame; Step 2.4.2, based on the target texture feature, calculate the probability distribution map T(i, j) of the target texture feature appearing in the video frame; Step 2.4.3, according to the target shape features, calculate the probability distribution map S(i, j) of the target shape features appearing in the video frame; Step 2.4.4: Construct the target fusion probability distribution map as follows: H(i,j)=uC(i,j)+vT(i,j)+wS(i,j) Among them, u is the fusion coefficient of the probability distribution graph C(i, j), v is the fusion coefficient of the probability distribution graph T(i, j), w is the fusion coefficient of the probability distribution graph S(i, j), u, v, w∈[0, 1].

5. The method for monitoring safe navigation of a ship based on video images according to claim 4 is characterized in that: In the step 2.5, the method for continuously updating the fusion probability distribution map H(i, j) of the target in the video frame is to continuously update the sizes of u, v, w according to the real-time target color features, texture features and shape features of the target.

6. The method for monitoring safe navigation of a ship based on video images according to claim 5, characterized in that: set up Among them A i,j is the cumulative difference of the color feature histogram of the jth target in the i-th video frame, ΔA is the set value of the cumulative difference of the color feature histogram; B i,j,l is the lth histogram ordinate amplitude of the jth target in the i-th video frame in the color feature histogram, p is the number of histograms in the color feature histogram, ΔB is the set value of a single histogram difference in the color feature histogram; k1 and k2 are weight coefficients; 7. The method for monitoring safe navigation of a ship based on video images according to claim 5, characterized in that: set up Where A' i,j is the cumulative difference of the texture feature histogram of the jth target in the i-th video frame, ΔA' is the set value of the cumulative difference of the texture feature histogram; B' i,j,l is the lth histogram ordinate amplitude of the jth target in the texture feature histogram in the i-th video frame, p is the number of histograms in the texture feature histogram, ΔB' is the set value of a single histogram difference in the texture feature histogram; k3 and k4 are weight coefficients; 8. The method for monitoring safe navigation of a ship based on video images according to claim 6, characterized in that: set up Among them, A i,j is the cumulative difference of the shape feature histogram of the jth target in the i-th video frame, ΔA” is the set value of the cumulative difference of the shape feature histogram; B” i,j,l is the lth histogram ordinate amplitude of the jth target in the shape feature histogram in the i-th video frame, p is the number of histograms in the shape feature histogram, ΔB” is the set value of a single histogram difference in the shape feature histogram; k5 and k6 are weight coefficients; 9. The method for monitoring safe navigation of a ship based on video images according to claim 8, characterized in that: In step 2.4.4, u=k7(u'+1-v'-w'), v=k8(v'+1-u'-w'), w=k9(w'+1-u'-v'), and k7, k8 and k9 are weight coefficients.

10. The method for monitoring safe navigation of a ship based on video images according to claim 1, characterized in that: In step 3, a camera is used to measure the distance of the target object. A monocular camera can calculate the distance based on the known size of the target and the size in the video frame; a binocular camera can calculate the depth information using the parallax principle; in step 4, a Kalman filter or a prediction model is used to estimate the future position and motion path of the target.

Images