A bridge anti-collision active warning method and system based on image recognition

Abstract

The present invention relates to the field of image processing, and the present invention relates to an active bridge anti-collision warning method and system based on image recognition, the method comprising: during the ship's travel, establishing a three-dimensional model of the ship on the horizontal plane at the current moment, and starting a drone carried by the ship to go to a designated position to collect point cloud data of the bridge, thereby constructing a three-dimensional bridge model; converting the three-dimensional ship model and the three-dimensional bridge model into a ship voxel grid and a bridge voxel grid respectively, aligning the ship voxel grid and the bridge voxel grid and performing an intersection test, and issuing a collision warning message if there is an intersection between the ship voxel grid and the bridge voxel grid. The present invention continuously adjusts the shooting position and angle of the drone to achieve the optimal selection, thereby obtaining high-quality bridge point cloud data, integrating the three-dimensional ship model and the three-dimensional bridge model into a common reference system, thereby improving the safety of ship navigation through real-time monitoring and early warning.

Description

Technical Field

[0001] The present invention relates to the field of image processing and to an active bridge anti-collision warning method and system based on image recognition. Background Art

[0002] With the development of the water transport industry, the foundation safety of bridges is one of the most important topics in bridge protection. In water transport, ships usually need to pass through bridges. However, due to the large size of ships, complex operations, and the complexity of the water transport environment, collisions between ships and bridges occur from time to time. These accidents may cause damage to ships and casualties, and may also cause structural damage to bridges and traffic interruptions, causing serious impacts on waterway safety and transportation.

[0003] At present, the patent application document with publication number CN106530837A discloses a method for active bridge ship collision prevention based on machine vision. The method includes: selecting appropriate bridge piers to arrange cameras, using machine vision technology to perform static and dynamic identification of ships on the river, and comprehensively analyzing the speed, heading, location and other information of the ship to provide a bridge-ship collision risk assessment and issue a timely warning before a collision occurs.

[0004] The above method is mainly aimed at using machine vision technology to monitor and warn sailing ships from the perspective of the bridge. However, since the camera used is at a high place, although it can cover a wider area, the high perspective may lead to insufficient detection of small ships. In addition, since the position and angle of the camera are fixed, a visual blind spot may be formed under the bridge pier or in the distance, making it impossible to accurately judge the height and width of the ship on the horizontal plane, which may make it impossible to accurately assess whether the ship can safely pass through the bridge hole. Summary of the invention

[0005] In order to solve the technical problem that when installing cameras on bridge piers to identify sailing ships, there are certain field of view limitations and it is impossible to accurately assess whether the ships can safely pass through the bridge tunnel, the present invention provides solutions in the following aspects.

[0006] In a first aspect, a bridge anti-collision active warning method based on image recognition comprises:

[0007] During the ship's travel, a 3D model of the ship at the current moment on the horizontal plane is established, and the drone carried by the ship is started to go to the designated location to collect point cloud data of the bridge, thereby constructing a 3D model of the bridge;

[0008] The three-dimensional ship model and the three-dimensional bridge model are converted into a ship voxel grid and a bridge voxel grid respectively, the ship voxel grid and the bridge voxel grid are aligned and an intersection test is performed, and if there is an intersection between the ship voxel grid and the bridge voxel grid, a collision warning message is issued;

[0009] The process of the drone going to the shooting location to collect the point cloud data of the bridge is as follows:

[0010] The flight area, flight altitude and shooting angle range of the UAV are set, and the initial shooting position and shooting angle are arbitrarily selected within the range as the input of the objective function, and the objective function is positively correlated with the overlap and point cloud density of the point cloud data of the UAV at multiple shooting angles;

[0011] Iterate the objective function, use the optimization algorithm in each iteration, and output the optimal shooting position and angle that makes the objective function value the highest after reaching the preset number of iterations;

[0012] The drone obtains point cloud data of the bridge at the optimal shooting position and angle.

[0013] Beneficial effects: The present invention uses a drone carried by a ship to photograph a bridge. During the photographing process, the drone's photographing position and angle are continuously adjusted through multiple iterations and an optimization algorithm, thereby obtaining high-quality bridge point cloud data to construct an accurate three-dimensional model of the bridge. The three-dimensional model of the ship and the three-dimensional model of the bridge are integrated into a common reference system for better collision detection, and alarms are issued in a timely manner, thereby avoiding possible collisions between ships and bridges through real-time monitoring and early warning, thereby improving the safety of ship navigation.

[0014] In one embodiment, the objective function is:

[0015] ; In the formula, is the objective function, The shooting position of the drone. For shooting angle, , , are weight coefficients, is the point cloud density, is the distance between bridge points, is the overlap of point cloud data under multiple shooting angles; and Indicates the horizontal position of the drone when shooting. Indicates the altitude at which the drone is flying.

[0016] Beneficial effect: By maximizing the point cloud density, the objective function ensures that each frame taken by the drone contains as much point cloud information as possible, which directly improves the accuracy and reliability of the bridge data.

[0017] In one embodiment, aligning the ship voxel grid and the bridge voxel grid comprises:

[0018] The bridge voxel grid is selected as a fixed reference coordinate system, and the optimal transformation matrix between the ship voxel grid and the bridge voxel grid is calculated using an iterative closest point algorithm; wherein the termination condition of the iteration is that the distance between the two voxel grids is less than a set threshold;

[0019] According to the optimal transformation matrix, the ship voxel grid is transformed into the coordinate system of the bridge voxel grid, so that the three-dimensional model of the ship and the three-dimensional model of the bridge are aligned in the fixed reference coordinate system.

[0020] Beneficial effects: By aligning the two voxel grids, the system can determine the relative position and distance between the ship and the bridge with high precision, which is crucial for accurately judging the risk of collision. In a real-time monitoring system, the position of the ship is constantly changing. Aligning the voxel grids can update the relative position of the ship and the bridge in real time, providing timely collision avoidance information to the driver or the automatic navigation system.

[0021] In one embodiment, the intersection test step is:

[0022] Constructing corresponding octrees based on the bridge voxel grid and the ship voxel grid respectively, traversing the two octrees, and if there is no intersection between the root nodes of the two octrees, it is confirmed that there is no collision between the ship and the bridge;

[0023] If there is an intersection between the root nodes of the two octrees, continue traversing the intersecting root nodes, and when the traversal reaches a leaf node, discretize the voxels in the leaf nodes of the two octrees into a discrete point set;

[0024] A distance threshold is set, and if the distance between any two discrete points in the two discrete point sets is less than the distance threshold, it is confirmed that a collision occurs between the ship and the bridge.

[0025] Beneficial effects: Using an octree can significantly reduce unnecessary calculations, because non-intersecting parts can be quickly excluded. By setting a discrete point set and a distance threshold, the algorithm can accurately determine the occurrence of a collision at the voxel level.

[0026] In one embodiment, the steps of acquiring the three-dimensional model of the ship are:

[0027] Determine the height of the ship above the horizontal plane at the current moment, and establish a three-dimensional model of the ship above the horizontal plane according to the height, the width of the ship and the length of the ship;

[0028] In the three-dimensional ship model, the horizontal plane is taken as the origin, the x-axis represents the forward direction of the ship, the y-axis represents the lateral position of the ship, and the z-axis represents the height of the ship above the horizontal plane.

[0029] In one embodiment, constructing the three-dimensional model of the bridge further comprises:

[0030] Using a binocular vision system to capture an image pair of the bridge, and using a feature matching algorithm to find a corresponding feature point pair in the image pair;

[0031] Calculating disparity information of the image pair using the feature point pair, and converting the disparity information into three-dimensional coordinates using calibration data of a camera in a binocular vision system to generate the point cloud data;

[0032] The point cloud data is preprocessed and reconstructed into a three-dimensional model of the bridge.

[0033] Beneficial effects: The binocular vision system can adapt to a variety of lighting and environmental conditions, so that the three-dimensional model reconstruction can work stably in different environments, improving the stability and reliability of the system.

[0034] In one embodiment, the origin of the three-dimensional bridge model is defined as the center of gravity of the bridge on the horizontal plane, the x-axis represents the length of the bridge, the y-axis represents the width of the bridge, and the z-axis represents the height of the bridge above the horizontal plane.

[0035] In a second aspect, a bridge collision avoidance active warning system based on image recognition includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned bridge collision avoidance active warning method based on image recognition is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] By reading the following detailed description with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0037] Figure 1 It is a method flow chart of steps S1-S2 in a bridge anti-collision active warning method based on image recognition in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0039] The embodiment of the present invention discloses a bridge anti-collision active warning method based on image recognition, referring to Figure 1 , including steps S1-S2, which are as follows:

[0040] S1: During the ship's travel, a three-dimensional model of the ship on the horizontal plane at the current moment is established, and the drone carried by the ship is started to go to the designated location to collect point cloud data of the bridge, thereby constructing a three-dimensional model of the bridge.

[0041] In the embodiment of the present invention, the application scenario is: a ship equipped with a binocular vision detection system and an unmanned aerial vehicle device equipped with a 3D camera.

[0042] First, when the ship is sailing normally along the route, the GPS signal is combined with map data to detect whether there is a bridge at a certain distance ahead of the ship's route, where the certain distance is set to 100 meters.

[0043] When a bridge is detected 100 meters ahead, a laser rangefinder is installed on the ship at a suitable location, and the laser rangefinder is shot vertically toward the water surface to measure the distance between the laser rangefinder and the horizontal plane. The height of the hull is constant, and the position where the laser detector is installed is fixed, so the draft of the ship at the current moment can be detected, and the height of the ship above the horizontal plane at the current moment can be further obtained. According to the height of the ship above the horizontal plane, the length and width of the ship, a three-dimensional model of the ship above the horizontal plane is established in the computer software, and it is marked as the three-dimensional model of the ship.

[0044] It should be noted that the three-dimensional model of the ship takes the horizontal plane as the origin, the x-axis represents the ship's forward direction, the y-axis represents the lateral position of the ship, and the z-axis represents the height of the ship above the horizontal plane.

[0045] For example, the height of the ship is 60 meters and the hull is 30 meters. The laser rangefinder is installed vertically facing the water surface at a height of 25 meters from the bottom of the ship. At this time, the laser rangefinder detects that the distance from the water surface is 15 meters. The draft of the ship at the current moment is 10 meters, and the height of the ship above the horizontal plane at the current moment is 50 meters.

[0046] In an embodiment of the present invention, a visual system is mounted on a drone, wherein the parking position of the drone is fixed relative to the height of the bottom of the ship. The draft of the ship at the current moment obtained as described above can be used to calculate the height of the drone from the horizontal plane before taking off. The drone can then be controlled to take off and follow its flight path to a fixed shooting position on the route relative to the bridge in front of the ship to obtain three-dimensional information of the bridge.

[0047] Among them, the drone can be helped to locate a flight path and position suitable for collecting data through a pre-set program or by using GPS and navigation tools; the distance and height difference between the current position of the drone and the shooting position can also be calculated to determine the distance the drone needs to fly forward and the height to which it needs to move vertically, and the calculation results can be input into the flight control system of the drone. The control system adjusts the flight attitude and speed of the drone based on the input data, so that it flies forward and moves to the target position. During the flight, sensors can be used to monitor and adjust the drone's position, speed, attitude and other parameters in real time to ensure that the drone can accurately reach the shooting position.

[0048] For example, it is assumed that the ship is 90 meters away from the bridge when the drone takes off, and the drone is positioned on the ship 30 meters from the bottom of the ship. The position of the drone before taking off needs to be 20 meters away from the current horizontal plane. When the drone needs to be controlled to a height of 10 meters from the horizontal plane of the bridge and 5 meters above the horizontal plane to collect three-dimensional information of the bridge, the drone is controlled to fly 80 meters forward based on its current position, and then descend 15 meters in the direction of the height of the drone from the horizontal plane before taking off to reach the shooting position.

[0049] The flight area and flight altitude of the drone are determined, and the range of shooting angles is set based on experience. The initial shooting position and shooting angle are arbitrarily selected within the range as the starting point of the iteration;

[0050] For each combination of shooting position and angle, the objective function is calculated and satisfies the following relationship:

[0051] The objective function is:

[0052] ; In the formula, is the objective function, The shooting position of the drone. For shooting angle, , , are weight coefficients, is the point cloud density, is the distance between bridge points, is the overlap of point cloud data under multiple shooting angles; and Indicates the horizontal position of the drone when shooting. Indicates the altitude at which the drone is flying.

[0053] The shooting position and shooting angle of the drone are used as the input of the objective function, and the objective function is iterated. In each iteration, an optimization algorithm (such as gradient descent method, genetic algorithm, etc.) is used to make slight adjustments to the shooting position and shooting angle in the hope of obtaining better results in the next iteration. The iterative process is repeated until the preset number of iterations is reached, and the optimal shooting position and angle that makes the objective function value the highest is output. Then the drone is allowed to obtain the point cloud data of the bridge at the optimal shooting position and angle.

[0054] The obtained point cloud data of the bridge is filtered to remove random noise and outliers in the point cloud data, and the point cloud data is converted into a three-dimensional model of the bridge using a surface reconstruction algorithm.

[0055] In another embodiment, two monocular cameras with high resolution and fast acquisition speed are installed on a fixed bracket of the ship to build a binocular vision system. The focal length and aperture of the camera lens, as well as the exposure time, gain, white balance and other parameters of the camera are adjusted to adapt to the 100m long-distance shooting under normal environment. The two monocular cameras are calibrated respectively using a suitable calibration plate to calculate the internal and external parameters of the camera. Using a synchronous trigger, the two monocular cameras are triggered to shoot at the same time to ensure that the image acquisition of the two monocular cameras is synchronized.

[0056] The images of the bridge taken simultaneously by two monocular cameras are aligned, and the corresponding feature point pairs are found in the image pair using a feature matching algorithm. The disparity information of the image pair is calculated using the feature point pairs, and the calibration data of the camera in the binocular vision system is used to convert the disparity information into three-dimensional coordinates to generate three-dimensional point cloud data of the bridge;

[0057] The SOR filtering algorithm is used to smooth the three-dimensional point cloud data of the bridge to remove outliers and noise point clouds, and the processed three-dimensional point cloud data of the bridge is reconstructed into a three-dimensional model of the bridge.

[0058] It should be noted that the origin in the three-dimensional model of the bridge is defined as the center of gravity of the bridge on the horizontal plane, the x-axis represents the length of the bridge, the y-axis represents the width of the bridge, and the z-axis represents the height of the bridge above the horizontal plane.

[0059] S2: Convert the three-dimensional ship model and the three-dimensional bridge model into a ship voxel grid and a bridge voxel grid respectively, align the ship voxel grid and the bridge voxel grid and perform an intersection test. If there is an intersection between the ship voxel grid and the bridge voxel grid, issue a collision warning message.

[0060] In an embodiment of the present invention, the three-dimensional model of the ship and the three-dimensional model of the bridge are converted into a ship voxel grid and a bridge voxel grid respectively, the bridge voxel grid is selected as a fixed reference coordinate system, and the iterative closest point algorithm is used to calculate the optimal transformation matrix between the ship voxel grid and the bridge voxel grid; wherein the termination condition of the iteration is that the distance between the two voxel grids is less than a set threshold; the set threshold can be set to 0.01 based on experience.

[0061] According to the optimal transformation matrix, the ship voxel grid is transformed into the coordinate system of the bridge voxel grid, so that the ship 3D model and the bridge 3D model are aligned in a fixed reference coordinate system.

[0062] Based on the bridge voxel grid and the ship voxel grid, the corresponding octrees are constructed respectively, and the two octrees are traversed. If there is no intersection between the root nodes of the two octrees, it is confirmed that there is no collision between the ship and the bridge;

[0063] If there is an intersection between the root nodes of the two octrees, continue traversing the intersecting root nodes. When the traversal reaches the leaf nodes, discretize the voxels in the leaf nodes of the two octrees into a discrete point set respectively;

[0064] The distance threshold is set to 5. If the distance between any two discrete points in the two discrete point sets is less than the distance threshold, it is confirmed that a collision occurs between the ship and the bridge, and then an early warning message is sent to the ship control room in a timely manner.

[0065] The present invention uses a drone carried by a ship to photograph a bridge. During the photographing process, the drone's photographing position and angle are continuously adjusted through multiple iterations and an optimization algorithm, thereby obtaining high-quality bridge point cloud data to construct an accurate three-dimensional model of the bridge. The three-dimensional model of the ship and the three-dimensional model of the bridge are integrated into a common reference system for better collision detection, and an alarm is issued in time, thereby avoiding possible collisions between ships and bridges through real-time monitoring and early warning, thereby improving the safety of ship navigation.

[0066] An embodiment of the present invention also discloses an active bridge collision avoidance warning system based on image recognition, including a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, an active bridge collision avoidance warning method based on image recognition according to the present invention is implemented.

[0067] The system also includes other components familiar to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

[0068] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module or both. Any such computer storage medium may be part of a device or accessible or connectable to a device.

[0069] In the description of this specification, "plurality" or "several" means at least two, such as two, three or more, etc., unless otherwise clearly and specifically defined.

[0070] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

Claims

1. A bridge anti-collision active warning method based on image recognition, characterized in that: include: During the ship's travel, a 3D model of the ship at the current moment on the horizontal plane is established, and the drone carried by the ship is started to go to the designated location to collect point cloud data of the bridge, thereby constructing a 3D model of the bridge; The three-dimensional ship model and the three-dimensional bridge model are converted into a ship voxel grid and a bridge voxel grid respectively, the ship voxel grid and the bridge voxel grid are aligned and an intersection test is performed, and if there is an intersection between the ship voxel grid and the bridge voxel grid, a collision warning message is issued; The process of the drone going to the shooting location to collect the point cloud data of the bridge is as follows: The flight area, flight altitude and shooting angle range of the UAV are set, and the initial shooting position and shooting angle are arbitrarily selected within the range as the input of the objective function, and the objective function is positively correlated with the overlap and point cloud density of the point cloud data of the UAV at multiple shooting angles; Iterate the objective function, use the optimization algorithm in each iteration, and output the optimal shooting position and angle that makes the objective function value the highest after reaching the preset number of iterations; The drone obtains point cloud data of the bridge at the optimal shooting position and angle.

2. The bridge anti-collision active warning method based on image recognition according to claim 1 is characterized in that: The objective function is: ; In the formula, is the objective function, The shooting position of the drone. For shooting angle, , , are weight coefficients, is the point cloud density, is the distance between bridge points, is the overlap of point cloud data under multiple shooting angles; and Indicates the horizontal position of the drone when shooting. Indicates the altitude at which the drone is flying.

3. The bridge anti-collision active warning method based on image recognition according to claim 2 is characterized in that: The aligning of the ship voxel grid and the bridge voxel grid comprises: The bridge voxel grid is selected as a fixed reference coordinate system, and the optimal transformation matrix between the ship voxel grid and the bridge voxel grid is calculated using an iterative closest point algorithm; wherein the termination condition of the iteration is that the distance between the two voxel grids is less than a set threshold; According to the optimal transformation matrix, the ship voxel grid is transformed into the coordinate system of the bridge voxel grid, so that the three-dimensional model of the ship and the three-dimensional model of the bridge are aligned in the fixed reference coordinate system.

4. The bridge anti-collision active warning method based on image recognition according to claim 3 is characterized in that: The steps of the intersection test are: Constructing corresponding octrees based on the bridge voxel grid and the ship voxel grid respectively, traversing the two octrees, and if there is no intersection between the root nodes of the two octrees, it is confirmed that there is no collision between the ship and the bridge; If there is an intersection between the root nodes of the two octrees, continue traversing the intersecting root nodes, and when the traversal reaches a leaf node, discretize the voxels in the leaf nodes of the two octrees into a discrete point set; A distance threshold is set, and if the distance between any two discrete points in the two discrete point sets is less than the distance threshold, it is confirmed that a collision occurs between the ship and the bridge.

5. The bridge anti-collision active warning method based on image recognition according to claim 1 is characterized in that: The steps of obtaining the three-dimensional model of the ship are: Determine the height of the ship above the horizontal plane at the current moment, and establish a three-dimensional model of the ship above the horizontal plane according to the height, the width of the ship and the length of the ship; In the three-dimensional ship model, the horizontal plane is taken as the origin, the x-axis represents the forward direction of the ship, the y-axis represents the lateral position of the ship, and the z-axis represents the height of the ship above the horizontal plane.

6. The bridge anti-collision active warning method based on image recognition according to claim 1 is characterized in that: The constructing of the three-dimensional bridge model further comprises: Using a binocular vision system to capture an image pair of the bridge, and using a feature matching algorithm to find a corresponding feature point pair in the image pair; Calculating disparity information of the image pair using the feature point pair, and converting the disparity information into three-dimensional coordinates using calibration data of a camera in a binocular vision system to generate the point cloud data; The point cloud data is preprocessed and reconstructed into a three-dimensional model of the bridge.

7. The bridge anti-collision active warning method based on image recognition according to claim 6 is characterized in that: The origin in the three-dimensional model of the bridge is defined as the center of gravity of the bridge on the horizontal plane, the x-axis represents the length of the bridge, the y-axis represents the width of the bridge, and the z-axis represents the height of the bridge above the horizontal plane.

8. An active bridge collision avoidance warning system based on image recognition, characterized in that: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, an image recognition-based bridge collision avoidance active warning method according to any one of claims 1 to 7 is implemented.

Images