Ship trajectory recognition and world coordinate conversion method based on adaptive water surface elevation changes

Abstract

The present invention relates to a ship trajectory recognition and world coordinate conversion method that is adaptive to changes in water surface elevation. The method uses a computer vision method to detect ships in monitoring videos, uses a water surface elevation monitoring sensor to obtain the current water surface elevation, and realizes ship detection and tracking in inland waterways through a calibrated spatial conversion relationship between three-dimensional world coordinates and pixel coordinates. The method adaptively adjusts the trajectory points of the ship's pixel coordinates by reverse projection to world coordinates according to seasonal changes in water surface elevation. The method has the advantages of accuracy, reliability, economy and high efficiency.

Description

Technical Field

[0001] The invention relates to a ship track recognition and world coordinate conversion method capable of self-adapting to changes in water surface elevation, and belongs to the field of inland waterway ship traffic monitoring. Background Art

[0002] Water transport has become the main form of transportation for domestic and international trade due to its large carrying capacity and eco-friendly characteristics; however, accidents such as ship-to-ship collisions, ship-bridge collisions, and ship groundings are becoming increasingly frequent, posing a serious threat to navigation safety and river ecological protection. Active collision avoidance methods can significantly reduce the probability of ship collisions and groundings by providing early warnings for ships with high collision risks, and ship target detection is one of the core tasks of these active collision avoidance methods.

[0003] In recent years, with the rapid development of computer vision algorithms, compared with synthetic aperture radar (SAR) and automatic identification system (AIS), video surveillance has great application potential in inland ship detection and active collision avoidance system due to its advantages of low equipment cost and high resolution. In the application of computer vision algorithms, binocular or multi-camera cameras are commonly used devices for locating three-dimensional targets. However, the price and time cost of deploying binocular cameras are high, which is not easy to promote in actual projects. Due to the lack of depth information, monocular cameras are also limited in identifying three-dimensional targets. Since the water surface is flat, it can be regarded as a plane. By selecting and measuring the coordinates of some points on the water surface in the horizontal plane and the image plane, the homography transformation matrix of the two planes is established to map the trajectory of the ship from the image plane to the horizontal plane. However, since the water surface elevation changes with the seasons, the world coordinates of each point on the water surface may be offset, or sometimes the world coordinate measurement of the required point is limited due to the inability to approach the water surface.

[0004] Therefore, it is urgent to design a new conversion method to overcome the limitations of on-site measurements and realize ship detection and tracking in inland waterways. Summary of the invention

[0005] The present invention provides a ship trajectory recognition and world coordinate conversion method that is adaptive to changes in water surface elevation, which overcomes the limitations of on-site measurement, realizes ship detection and tracking in inland waterways, and adaptively adjusts the trajectory points of the ship's pixel coordinates by reverse projection to world coordinates according to seasonal changes in water surface elevation.

[0006] The technical solution adopted by the present invention to solve the technical problem is:

[0007] The method for ship trajectory recognition and world coordinate conversion based on adaptive water surface elevation change specifically includes the following steps:

[0008] Step S1: Install video surveillance equipment on the shore of the monitored inland river to obtain videos of monitored inland river vessels;

[0009] Step S2: Use the calibration board to calibrate the camera internal parameters of the video surveillance device to obtain the internal parameters of the video surveillance device where f x 、f y is the pixel focal length of the video surveillance device, c x 、c y is the coordinate of the origin of the video surveillance device coordinate system in the pixel coordinate system;

[0010] Step S3: Based on the video surveillance device installed in step S1, select several fixed points visible in the field of view of the video surveillance device, measure the coordinates of the several fixed points in the world coordinate system and the coordinates of the several fixed points reflected in the video pixel coordinate system, wherein the world coordinate system adopts P w (X w , Y w , Z w ) indicates that the video pixel coordinate system uses P p (u, v) indicates that X in the world coordinate system is specified w Y w The plane is parallel to the horizontal plane, the coordinate axis Z w perpendicular to the horizontal plane;

[0011] Based on the world coordinate system coordinates and pixel coordinate system coordinates of the above fixed points, the PnP algorithm and the random sample consistency method are used to calculate the external parameter [RT] of the video surveillance device, which represents the rotation matrix of the video surveillance device. and the translation matrix representing the video surveillance device where r ij is the rotation coefficient, t n is the translation coefficient;

[0012] Step S4: Install a water surface elevation monitoring sensor in the waters of the monitored inland river, and obtain the water surface elevation based on the world coordinate system established in step S3. in Indicates the water surface elevation in the world coordinate system Z w The coordinate values ​​of the axis;

[0013] Step S5: The bounding box of the ship target in the video in the pixel coordinate system is obtained by using a computer vision algorithm based on deep learning target detection and multi-target tracking. The coordinates of the upper left corner of the bounding box are (x1, y1), the width is w, and the height is h. 1 / 2w,y1+ 3 / 4h) as the pixel coordinates of the ship's trajectory P p (u, v), that is, the horizontal midpoint and the vertical lower quarter point of the bounding box are the navigation track positions of the ship;

[0014] Step S6: Based on the water surface elevation E calculated in step S4, the external parameter calibration [RT] obtained in step S3, and the pixel coordinate P obtained in step S5 p (u, v), the world coordinates of the ships in the monitored inland river calculated by back projection;

[0015] As a further preferred embodiment of the present invention, the video surveillance device in step S1 is a monocular camera;

[0016] As a further preferred embodiment of the present invention, the specific steps of obtaining the bounding box of the ship target in the video in the pixel coordinate system in step S5 are:

[0017] Step S51: extracting some frames from the video of monitoring inland ships as a training data set, and marking all ships in the image with a bounding box to form a marked data set;

[0018] Step S52: Loading the labeled data set obtained in step S51 into the ship target detection and multi-target tracking deep learning model to train the model;

[0019] Step S53: Load the video of the monitored inland river vessel obtained in step S1 into the model trained in step S52 for detection and tracking;

[0020] As a further preferred embodiment of the present invention, the back projection calculation formula in step S6 is derived from the conversion formula of pixel coordinates and world coordinates, that is,

[0021]

[0022] Where Z c is the Z coordinate of the camera coordinate system, formula (1) is expanded to

[0023]

[0024] make Formula (2) can be reorganized as

[0025]

[0026] All parameters c in formula (3) ij They have been obtained through the internal parameter calibration in step S2 and the external parameter calibration in step S3. The pixel coordinates u and v are the ship trajectory points identified in step S5. At this time, Z w That is, the water surface elevation E has been obtained through step S4.

[0027] Through the above technical solution, compared with the prior art, the present invention has the following beneficial effects:

[0028] 1. The method for ship trajectory recognition and world coordinate conversion with adaptive water surface elevation change provided by the present invention directly establishes a mapping relationship between a two-dimensional image and a three-dimensional world coordinate within the camera field of view through point pair information of surrounding facilities or structures, thus overcoming the limitations of on-site measurement;

[0029] 2. The method for ship trajectory recognition and world coordinate conversion that is adaptive to water surface elevation changes provided by the present invention uses a water surface elevation monitoring sensor to dynamically adjust the coordinate conversion relationship based on seasonal water surface elevation changes, and more accurately restores the ship's navigation trajectory in three-dimensional world coordinates when the water surface height changes;

[0030] 3. The ship trajectory recognition and world coordinate conversion method for adaptive water surface elevation changes provided by the present invention is easier to deploy, manage and maintain, and has lower economic and labor costs than synthetic aperture radar (SAR) and automatic ship identification system (AIS). BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0032] Figure 1 It is a principle diagram of a method for ship trajectory recognition and world coordinate conversion for adaptive water surface elevation change provided by the present invention;

[0033] Figure 2 It is a flow chart of a method for ship trajectory recognition and world coordinate conversion for adaptive water surface elevation change provided by the present invention;

[0034] Figure 3 This is a case result diagram obtained by the preferred embodiment provided by the present invention. DETAILED DESCRIPTION

[0035] The present invention will now be described in further detail with reference to the accompanying drawings. In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms "left side", "right side", "upper part", "lower part", etc. is based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and "first", "second", etc. do not indicate the importance of the components, and therefore cannot be understood as a limitation on the present invention. The specific dimensions used in this embodiment are only for illustrating the technical solution by example, and do not limit the scope of protection of the present invention.

[0036] As described in the background technology, the conversion means for inland ship detection in the prior art is often a coordinate conversion means through homography transformation. Since the water surface elevation changes with the seasons, the world coordinates of each point on the water surface are always offset, or because it is impossible to approach the water surface, the world coordinate measurement of the required points is restricted, which brings limitations to on-site measurement and cannot achieve accurate detection and tracking of inland waterway ships.

[0037] Based on the above problems, this application provides a method for ship trajectory recognition and world coordinate conversion that is adaptive to changes in water surface elevation. Figure 1 The conversion principle is shown in the figure. It mainly relies on the water surface elevation monitoring sensor to adaptively adjust to seasonal water surface elevation changes dynamically. When the water surface height changes, the navigation trajectory of the ship in the three-dimensional world coordinates can be restored more accurately. Figure 2 As shown, the specific steps of the conversion method provided by this application are:

[0038] Step S1: video surveillance equipment is installed on the bank of the monitored inland river to obtain videos of ships in the monitored inland river; the video surveillance equipment determines the trajectory of the ship in the monitoring video through a monocular camera instead of binocular vision, and adapts to the inverse projection position offset caused by seasonal water surface elevation changes.

[0039] Step S2: Use the calibration board to calibrate the camera internal parameters of the video surveillance device to obtain the internal parameters of the video surveillance device where f x 、f y is the pixel focal length of the video surveillance device, c x 、c y It is the coordinate of the origin of the video surveillance device coordinate system in the pixel coordinate system.

[0040] Step S3: Based on the video surveillance device installed in step S1, select several fixed points visible in the field of view of the video surveillance device, measure the coordinates of the several fixed points in the world coordinate system and the coordinates of the several fixed points reflected in the video pixel coordinate system, wherein the world coordinate system adopts P w (X w , Y w , Z w ) indicates that the video pixel coordinate system uses P p (u, v) indicates that X in the world coordinate system is specified w Y w The plane is parallel to the horizontal plane, the coordinate axis Z w perpendicular to the horizontal plane;

[0041] Based on the world coordinate system coordinates and pixel coordinate system coordinates of the above fixed points, the PnP algorithm and the random sample consistency method are used to calculate the external parameter [RT] of the video surveillance device, which represents the rotation matrix of the video surveillance device. and the translation matrix representing the video surveillance device where r ij is the rotation coefficient, t n is the translation coefficient.

[0042] Step S4: Install a water surface elevation monitoring sensor in the waters of the monitored inland river, and obtain the water surface elevation based on the world coordinate system established in step S3. in Indicates the water surface elevation in the world coordinate system Z w The coordinate values ​​of the axis.

[0043] Step S5: The bounding box of the ship target in the video in the pixel coordinate system is obtained by using a computer vision algorithm based on deep learning target detection and multi-target tracking. The coordinates of the upper left corner of the bounding box are (x1, y1), the width is w, and the height is h. 1 / 2w,y1+ 3 / 4h) as the pixel coordinates of the ship's trajectory P p (u, v), that is, the horizontal midpoint and the vertical lower quarter point of the bounding box are the navigation track positions of the ship;

[0044] The specific steps of the bounding box of the ship target in the video in the pixel coordinate system are:

[0045] Step S51: extracting some frames from the video of monitoring inland ships as a training data set, and marking all ships in the image with a bounding box to form a marked data set;

[0046] Step S52: Loading the labeled data set obtained in step S51 into the ship target detection and multi-target tracking deep learning model to train the model;

[0047] Step S53: Load the video of the monitored inland vessel obtained in step S1 into the model trained in step S52 for detection and tracking.

[0048] Step S6: Based on the water surface elevation E calculated in step S4, the external parameter calibration [RT] obtained in step S3, and the pixel coordinate P obtained in step S5 p (u, v), the world coordinates of the monitored ships in the inland river are calculated by back projection.

[0049] The back projection calculation formula in this step is derived from the conversion formula of pixel coordinates and world coordinates, that is,

[0050]

[0051] Where Z c is the Z coordinate of the camera coordinate system, formula (1) is expanded to

[0052]

[0053] make Formula (2) can be reorganized as

[0054]

[0055] All parameters c in formula (3) ij They have been obtained through the internal parameter calibration in step S2 and the external parameter calibration in step S3. The pixel coordinates u and v are the ship trajectory points identified in step S5. At this time, Z w That is, the water surface elevation E has been obtained through step S4.

[0056] Figure 3 The figure shows a specific implementation case provided by the present application based on the conversion method described above. The figure shows the position of the selected point pairs in the world coordinate system, the monitoring field of view water range restored by inverse transformation, and the detection and tracking of the navigation track of a certain ship in the world coordinate system.

[0057] From the above description, it can be seen that the method of ship trajectory recognition and world coordinate conversion with adaptive water surface height change provided by this application uses a monocular camera to realize the positioning and tracking of ships in the world coordinate system. Compared with synthetic aperture radar (SAR), ship automatic identification system (AIS) and dual / multi-camera, the equipment cost is low, and the layout and maintenance are more convenient. Compared with the coordinate conversion method through homography transformation in the prior art, it can realize the correction of the navigation trajectory in three-dimensional world coordinates according to seasonal water surface elevation changes, and has great application potential in inland ship detection and active collision avoidance systems, and is suitable for promotion in related fields.

[0058] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0059] The meaning of "and / or" described in this application means that the situations where each exists alone or both exist at the same time are included.

[0060] The term “connection” as used in this application may mean a direct connection between components or an indirect connection between components via other components.

[0061] Based on the above ideal embodiments of the present invention, the relevant staff can make various changes and modifications without departing from the technical concept of the present invention through the above description. The technical scope of the present invention is not limited to the contents of the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A method for ship trajectory recognition and world coordinate conversion based on adaptive water surface elevation changes, characterized in that: The specific steps include: Step S1: Install video surveillance equipment on the shore of the monitored inland river to obtain videos of monitored inland river vessels; Step S2: Use the calibration board to calibrate the camera internal parameters of the video surveillance device to obtain the internal parameters of the video surveillance device where f x 、f y is the pixel focal length of the video surveillance device, c x 、c y is the coordinate of the origin of the video surveillance device coordinate system in the pixel coordinate system; Step S3: Based on the video surveillance device installed in step S1, select several fixed points visible in the field of view of the video surveillance device, measure the coordinates of the several fixed points in the world coordinate system and the coordinates of the several fixed points reflected in the video pixel coordinate system, wherein the world coordinate system adopts P w (X w , Y w , Z w ) indicates that the video pixel coordinate system uses P p (u, v) indicates that X in the world coordinate system is specified w Y w The plane is parallel to the horizontal plane, the coordinate axis Z w perpendicular to the horizontal plane; Based on the world coordinate system coordinates and pixel coordinate system coordinates of the above fixed points, the PnP algorithm and the random sample consistency method are used to calculate the external parameter [RT] of the video surveillance device, which represents the rotation matrix of the video surveillance device. and the translation matrix representing the video surveillance device where r ij is the rotation coefficient, t n is the translation coefficient; Step S4: Install a water surface elevation monitoring sensor in the waters of the monitored inland river, and obtain the water surface elevation based on the world coordinate system established in step S3. in Indicates the water surface elevation in the world coordinate system Z w The coordinate values ​​of the axis; Step S5: The bounding box of the ship target in the video in the pixel coordinate system is obtained by using a computer vision algorithm based on deep learning target detection and multi-target tracking. The coordinates of the upper left corner of the bounding box are (x1, y1), the width is w, and the height is h. (x1+1 / 2w, y1+3 / 4h) is used as the pixel coordinates P of the ship's trajectory. p (u,v), that is, the horizontal midpoint and the vertical lower quarter point of the bounding box are the navigation track positions of the ship; Step S6: Based on the water surface elevation E calculated in step S4, the external parameter calibration [RT] obtained in step S3, and the pixel coordinate P obtained in step S5 p (u, v), the world coordinates of the monitored ships in the inland river are calculated by back projection.

2. The method for ship trajectory recognition and world coordinate conversion based on adaptive water surface elevation change according to claim 1, characterized in that: The video surveillance device in step S1 is a monocular camera.

3. The method for ship trajectory recognition and world coordinate conversion based on adaptive water surface elevation change according to claim 1, characterized in that: The specific steps of obtaining the bounding box of the ship target in the video in the pixel coordinate system in step S5 are: Step S51: extracting some frames from the video of monitoring inland ships as a training data set, and marking all ships in the image with a bounding box to form a marked data set; Step S52: Loading the labeled data set obtained in step S51 into the ship target detection and multi-target tracking deep learning model to train the model; Step S53: Load the video of the monitored inland vessel obtained in step S1 into the model trained in step S52 for detection and tracking.

4. The method for ship trajectory recognition and world coordinate conversion based on adaptive water surface elevation change according to claim 1, characterized in that: The back projection calculation formula in step S6 is derived from the conversion formula of pixel coordinates and world coordinates, that is, Where Z c is the Z coordinate of the camera coordinate system, formula (1) is expanded to make Formula (2) can be reorganized as All parameters c in formula (3) ij They have been obtained through the internal parameter calibration in step S2 and the external parameter calibration in step S3. The pixel coordinates u and v are the ship trajectory points identified in step S5. At this time, Z w That is, the water surface elevation E has been obtained through step S4.

Images