Computer vision based ship assisted berthing and unberthing method and related device

Abstract

The application provides a ship assisted berthing and unberthing method based on computer vision and related equipment. The method comprises the following steps: acquiring berth sign image information and berth sign image information board attitude information of a target berth; identifying berth position coordinate information of the target berth based on the berth sign image information; generating a planned navigation path according to the target berth position coordinate information and current position coordinate information of a target ship; and superimposing and displaying the target berth position coordinate information and the planned navigation path in a target video for enhanced display, wherein the target video is acquired by a camera. The method realizes the positioning automation of the target berth in the unberthing and berthing process of the target ship, improves the success rate of the unberthing and berthing of the ship, and saves the labor cost and time cost in the unberthing and berthing process of the ship.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ship technology, and in particular to a ship assisted berthing and unberthing method based on computer vision and related equipment. BACKGROUND

[0002] When a manned / unmanned ship is berthing or unberthing, or is landing or disembarking, the pilot needs to keep track of the situation around the ship, the channel and the hydrological and meteorological information to ensure that the ship can sail on a specific route. Due to the difficulty for the pilot to keep track of the real-time information of the ship speed, position and inertia, or the lack of experience in estimation and judgment or operation technology, the berthing or unberthing, or the landing or disembarking of the ship may fail or a collision with the wharf may occur. With the development of shipbuilding technology and navigation technology, large ships and unmanned ships are increasingly used, but they are more difficult to control and require higher standards for berthing and unberthing operations, especially for ships transporting dangerous goods. If only manual berthing is used, there is a high probability of collision, which may cause serious problems such as equipment loss, environmental pollution and personal safety. Therefore, it is necessary to obtain the relative attitude and position of the ship and the shore in real time to assist in berthing and unberthing.

[0003] The existing ship berthing and unberthing assistance system mainly uses microwave radar positioning technology, infrared positioning technology, laser positioning technology, ship-shore cooperation, satellite navigation positioning, etc. for positioning detection. These positioning technologies have obvious advantages and disadvantages in use. The advantage of microwave radar positioning technology is that it can detect a wide angle and effectively grasp the most dangerous situation in time, but the microwave band is easily affected by weather factors such as rain and snow, the resolution is not high, it is easily affected by false targets, and the effective distance is about 50m. The infrared positioning technology requires high weather visibility, and when the weather conditions are not good, the detection accuracy and distance cannot be guaranteed. Laser positioning technology is the most widely used and relatively effective positioning detection technology, which has the advantages of high detection accuracy, sensitive dynamic response speed and long effective distance, but it also has some disadvantages, such as narrow laser beam, point measurement, inability to fully grasp the overall situation of the ship, and great influence of weather. In actual operation, accidents of ship collision with the wharf may still occur. The ship-shore cooperation method uses the shore-based center to conduct collision risk warning based on sensing information, which is transmitted to the ship after being judged by the shore-based center, and then the ship makes risk disposal decisions and corresponding operations. This method is greatly limited by the rate of wireless transmission network and has a delay problem, which may increase the risk of collision due to wireless transmission problems and easily cause a critical situation. Satellite navigation positioning can monitor the sailing conditions such as speed, heading and distance before the ship berths, but the system is relatively complex. At present, there is no effective solution to the problems in the related technologies. Therefore, a new technical solution is urgently needed to solve the problem. SUMMARY

[0004] A series of simplified concepts are introduced in the summary section, which will be further described in detail in the specific embodiment section. The summary section of the present application does not mean to attempt to define the key features and essential technical features of the claimed technical solutions, nor to attempt to determine the protection scope of the claimed technical solutions.

[0005] In a first aspect, the present application proposes a computer vision-based ship assisted berthing and unberthing method. The computer vision-based ship assisted berthing and unberthing method comprises: acquiring berth sign image information and berth sign image information board attitude information of a target berth; identifying berth position coordinate information of the target berth based on the berth sign image information; generating a planned navigation path according to the target berth position coordinate information and current position coordinate information of a target ship; and superimposing and displaying the target berth position coordinate information and the planned navigation path in a target video for enhanced display, wherein the target video is acquired by a camera.

[0006] Optionally, the berth sign image information comprises ArUco label code information, and the ArUco label code information is used to mark position information of the target berth.

[0007] Optionally, generating the planned navigation path according to the target berth position coordinate information and the current position coordinate information of the target ship comprises: acquiring camera coordinate information, ArUco label code coordinate information and ship coordinate information; determining a first conversion relationship between the camera coordinate information and the ArUco label code coordinate information, wherein the first conversion relationship comprises an angle relationship and / or a displacement relationship corresponding to the camera coordinate information and the ArUco label code coordinate information; determining a second conversion relationship between the ship coordinate information and the camera coordinate information, wherein the second conversion relationship comprises an angle relationship between the ship coordinate information and the camera coordinate information; determining ship relative coordinate information based on the first conversion relationship, the second conversion relationship and the ship coordinate information, wherein the ship relative coordinate information is corresponding position information of the ship in an ArUco label code coordinate system; and generating the planned navigation path based on the ship relative coordinate information.

[0008] Optionally, the method further comprises: acquiring a navigation direction and a ship-shore distance of the target ship; and in a case where the ship-shore distance is less than a preset distance, controlling a berth sign image information board to perform a turning operation based on the navigation direction, so as to make an angle information between the berth sign image information and the navigation direction less than a preset angle, wherein the berth sign image information is displayed on the berth sign image information board.

[0009] Optionally, the berth mark image information is acquired by a camera on the target ship, and in the case of a fixed camera, the turning operation of the berth mark image information board is controlled based on the sailing direction, including: acquiring the field angle range of the fixed camera; acquiring the rotation angle and lifting information of the berth mark image information board coordinate system through a wireless data radio station; and controlling the rotation operation of the berth mark image information board based on the field angle range of the fixed camera, the camera coordinate system, the rotation angle and lifting information of the berth mark image information board coordinate system, and the sailing direction.

[0010] Optionally, the berth mark image information is acquired by a camera on the target ship, and in the case of a rotatable camera, the turning operation of the berth mark image information board is controlled based on the sailing direction, including: acquiring the rotation angle of the rotatable camera; acquiring the rotation angle and lifting information of the berth mark image information board coordinate system through a wireless data radio station; and controlling the rotation operation of the berth mark image information board based on the rotation angle of the rotatable camera, the camera coordinate system, the rotation angle and lifting information of the berth mark image information board coordinate system, and the sailing direction.

[0011] Optionally, the method further includes: in the case of at least two berth mark image information boards of the target berth and a plurality of cameras distributed at different positions of the ship, setting a berth coordinate system with the midpoint of the berth line as the origin, measuring and acquiring the arrangement positions of the plurality of berth mark image information boards and converting them into the coordinates of the berth coordinate system; and acquiring the coordinate systems of different cameras relative to the same berth mark image information board and performing error averaging and coordinate system normalization processing.

[0012] In a second aspect, the present application provides a ship berthing and unberthing route planning device based on computer vision, including: an acquisition unit configured to acquire berth mark image information and berth mark image information board attitude information of a target berth; an identification unit configured to identify berth position coordinate information of the target berth based on the berth mark image information; a generation unit configured to generate a planned sailing path according to the target berth position coordinate information and current position coordinate information of a target ship; and a display unit configured to superimpose and display the target berth position coordinate information and the planned sailing path in a target video for enhanced display, wherein the target video is acquired by a camera.

[0013] In a third aspect, the present application further provides an electronic device including a processor and a memory, wherein the memory stores computer program instructions, and the computer program instructions are used to execute a ship assisted berthing and unberthing method based on computer vision when executed by the processor.

[0014] In a fourth aspect, the present application further provides a storage medium, wherein program instructions are stored on the storage medium, and the program instructions are used to execute a ship assisted berthing and unberthing method based on computer vision when executed.

[0015] Therefore, the computer vision-based ship assisted berthing and unberthing method provided by the present application realizes automatic acquisition and identification of the target berth position coordinate information by acquiring the berth sign image information of the target berth, and generates a planned navigation path according to the target berth position coordinate information and the current position coordinate information of the target ship, thereby realizing the positioning of the target berth and the automation of the entire process of the target ship berthing and unberthing, reducing the manual participation process in the entire process of the target ship berthing and unberthing, thereby reducing the misoperation caused by relying on the past experience of the manual process to complete the target ship berthing and unberthing process, and further improving the success rate of the ship berthing and unberthing. The detection and guidance no longer depends on the positioning detection technology, and the target berth position coordinate information and the planned navigation path are superimposed and displayed in the target video for enhanced display, wherein the target video is acquired by a camera. The method breaks free from the restrictions of weather, distance and other objective factors on the ship berthing and unberthing process, and saves the labor cost and time cost of the ship berthing and unberthing process.

[0016] The computer vision-based ship assisted berthing and unberthing method of the present application, other advantages, objects and features of the present application will be embodied in part through the following description, and will be understood by those skilled in the art through research and practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0017] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of preferred embodiments and are not meant to limit the present application. Furthermore, the same reference numerals are used throughout the several drawings to represent similar parts. In the drawings:

[0018] Figure 1 A schematic flow chart of the computer vision-based ship assisted berthing and unberthing method according to one embodiment of the present application is shown;

[0019] Figure 2 A schematic block diagram of a berth sign device according to one embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of the information conversion relationship between the camera and the berth sign information board according to one embodiment of the present application is shown;

[0021] Figure 4 A schematic diagram of the processing process of the berth sign image information according to one embodiment of the present application is shown.

[0022] Figure 5 A schematic block diagram of the computer vision-based ship berthing and unberthing route planning device according to one embodiment of the present application is shown;

[0023] Figure 6 A schematic block diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0024] The terms "first", "second", "third", "fourth" and the like in the description and in the claims of the present application, if any, are used for distinguishing between similar objects talking about the embodiments and do not necessarily have a particular chronological, sequential or hierarchical order. It is to be understood that the data used in this way can be interchanged, so that whenever a particular embodiment is mentioned in this way it can be replaced by any other embodiment(s) provided the features of those embodiments are compatible. Furthermore, the terms "comprising", "having", "including", and "containing" and any variations thereof in the present specification are intended to cover a non-exclusive inclusion such that a process, method, system, article, or apparatus that comprises, has, includes, or contains a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, system, article, or apparatus. The following description of the embodiments of the present application with reference to the accompanying drawings is provided to understand various technical solutions and advantages of the present application, and the present application will be apparent to those skilled in the art from the description below, without any inventive effort. The embodiments described below are merely used to explain the present application, and should not be used to limit the present application. Obviously, the present application is not limited to the embodiments described below.

[0025] According to a first aspect of the present application, a computer vision-based ship assisted berthing and unberthing method is provided. Figure 1 A schematic flow chart of a computer vision-based ship assisted berthing and unberthing method 100 according to an embodiment of the present application is shown. As shown in Figure 1 The method 100 can include the following steps:

[0026] In step S110, berth sign image information and berth sign image information board attitude information of a target berth are acquired.

[0027] Exemplarily, the berth sign image information can be a two-dimensional code designed according to specific requirements. For example, the berth sign image information can be displayed in the form of a two-dimensional code on a berth sign image information board of the target berth. The berth sign image information can also be the geographic coordinates of the target berth marked in the form of text, and the specific form is not limited herein. Specifically, the berth sign image information board attitude information can include angle information, height information, etc.

[0028] It should be noted that the berth sign image information can also be used to estimate the distance information of the target ship from the target berth and the heading information of the target ship.

[0029] In step S120, berth position coordinate information of the target berth is identified based on the berth sign image information.

[0030] Exemplarily, the berth sign image information can be engraved on a berth sign image information board of the target berth in the form of a two-dimensional code or text. After the berth sign image information is scanned by the camera of the electronic device, the berth position coordinate information of the target berth can be displayed. Preferably, the camera of the electronic device can have sufficient resolution, infrared function or low-light function.

[0031] In step S130, a planned sailing path is generated according to the target berth position coordinate information and the current position coordinate information of the target ship.

[0032] Specifically, the current position coordinate information of the target ship can be directly measured by the electronic device on the target ship or derived by calculation, which is not specifically limited herein. It can be understood that all means that can achieve the current position coordinate information of the target ship at present or in the future are within the protection scope of the present application. Exemplarily, the heading direction can be guided based on the target berth position coordinate point and the current position coordinate point of the target ship identified in step S120, so as to realize the automatic berthing and unberthing of the target ship.

[0033] In step S140, the target berth position coordinate information and the planned sailing path are superimposed and displayed in a target video for enhanced display, wherein the target video is obtained by the camera.

[0034] Exemplarily, the camera is fixed on the target ship. Specifically, the target berth position coordinate information and the planned sailing path can be combined with the speed and heading information of the target ship, the wind direction and speed information of the current environment, and the flow direction and speed information of the water flow to enhance the Augmented Reality (AR) technology, so as to plan the unberthing path of the target ship.

[0035] The ship assisted berthing and unberthing method based on computer vision provided in the present application realizes automatic acquisition and identification of the target berth position coordinate information by acquiring the berth sign image information of the target berth, identifying the berth position coordinate information of the target berth based on the berth sign image information, and generating a planned navigation path according to the target berth position coordinate information and the current position coordinate information of the target ship, thereby realizing the positioning of the target berth and the automation of the whole process of the target ship berthing and unberthing, reducing the manual participation process in the whole process of the target ship berthing and unberthing, and thus reducing the misoperation caused by relying on the past experience of the manual operation to complete the process of the target ship berthing and unberthing, and further improving the success rate of the ship berthing and unberthing, and the detection guidance no longer depends on the positioning detection technology, and the target berth position coordinate information and the planned navigation path are superimposed and displayed in the target video for enhanced display, wherein the target video is acquired by a camera, thereby getting rid of the limitation of objective factors such as weather and distance on the process of the ship berthing and unberthing, and saving the labor cost and time cost in the process of the ship berthing and unberthing.

[0036] In some examples, the berth sign image information includes ArUco label code information, and the ArUco label code information is used to mark the position information of the target berth.

[0037] Specifically, the ArUco tag code can be composed of a wide black edge and an internal binary matrix, for example: a 4x4 marker is composed of 16 bits. The black boundary of the ArUco tag code is beneficial to quickly detect the image, and the internal matrix determines its ID. Since the ArUco tag code adopts binary coding, error detection and technical correction can be supported. The ArUco tag provides a dictionary class to describe the Marker dictionary, and records the encoding of different dictionaries through a file. For example, DICT_4*4_1000 is a predefined dictionary, which contains 1000 different Markers, and each Marker is composed of 16 bits. The dictionary pre-encoding format and the Marker style are one-to-one corresponding, and the Marker has directionality. The berth sign image information including the ArUco tag code information can be displayed on the berth sign image information board in the form of an electronic display screen, or alternatively, can be displayed on the berth sign image information board in the form of fluorescence, so as to facilitate the successful completion of the berthing and unberthing operation of the ship in low-visibility weather. The above method realizes the quick completion of the berth position coordinate information acquisition process of the target berth by using the ArUco tag code information to mark the position information of the target berth, and the use of the ArUco tag code information is beneficial to improve the fault tolerance, thereby realizing the saving of the time of the ship berthing and unberthing process, improving the success rate and accuracy of the ship berthing and unberthing process, and the use of the ArUco tag code information is helpful for the rapid detection and identification of electronic equipment. A single ArUco tag code information can provide enough information for obtaining the pose of the scanning device or measuring the position of the scanning device. Due to the use of binary coding, the ArUco tag code information algorithm is very powerful, can support error detection and technical correction, and is suitable for various complex environments, such as day, night, underwater, etc.

[0038] In some examples, the step S130 of generating a planned navigation path according to the target berth position coordinate information and the current position coordinate information of the target ship can include the following specific steps:

[0039] In step S131, camera coordinate information, ArUco tag code coordinate information and ship coordinate information are obtained.

[0040] Specifically, the camera coordinate information can be a camera coordinate system O c X c Y c Z c ; the ArUco tag code coordinate information can be an ArUco tag code coordinate system O w X w Yw Z w The ship coordinate information can be a ship coordinate system O s X s Y s Z s. The ArUco label code coordinate information can be used to represent the coordinate information of the berth sign image information board.

[0041] In step S132, the first conversion relationship between the camera coordinate information and the ArUco label code coordinate information is determined, wherein the first conversion relationship includes the angle relationship and / or the displacement relationship corresponding to the camera coordinate information and the ArUco label code coordinate information.

[0042] Specifically, the first conversion relationship between the camera coordinate system O c X c Y c Z c and the ArUco label code coordinate system O w X w Y w Z w can be determined. The first conversion relationship can be an angle relationship matrix Rwc and / or a displacement relationship Twc. The camera coordinate system O c X c Y c Z c and the ArUco label code coordinate system O w X w Y w Z w When there is a deflection angle θ around the Z axis, Rwc is Similarly, when there is a deflection angle φ around the X axis between the camera coordinate system O c X c Y c Z c and the ArUco label code coordinate system O w X w Y w Z w , Rwc is When there is a deflection angle ω around the Y axis between the camera coordinate system O c X c Y c Z c and the ArUco label code coordinate system O w X w Y w Z w , Rwc is When there is a deflection angle ω around the Y axis between the camera coordinate system O c X c Y cZ c When there is an offset h in the Y axis of the ArUco label code coordinate system, Twc can be (0, h, 0).

[0043] In step S133, a second conversion relationship between the ship coordinate information and the camera coordinate information is determined, wherein the second conversion relationship includes an angle relationship between the ship coordinate information and the camera coordinate information.

[0044] Specifically, the second conversion relationship between the ship coordinate system O s X s Y s Z s. and the camera coordinate system O c X c Y c Z c can be determined. The second conversion relationship can be an angle relationship matrix Rcs. The angle relationship matrix Rcs has the same value determination method as Rwc, which will not be described here.

[0045] In step S134, ship relative coordinate information is determined based on the first conversion relationship, the second conversion relationship, and the ship coordinate information, wherein the ship relative coordinate information is the corresponding position information of the ship in the ArUco label code coordinate system.

[0046] Specifically, the ship relative coordinate information can be the coordinates of any point on the target ship in the ship coordinate system O s X s Y s Z s. converted in the ArUco label code coordinate system O w X w Y w Z w , for example: the coordinates of a point P on the ship are (X, Y, Z), then based on (Rwc, Twc) in the first conversion relationship and (Rwc, Twc) in the second conversion relationship, the coordinates of P in the O w X w Y w Z w coordinate system are (X w , Y W , Z W ).

[0047] In step S135, a planned navigation path is generated based on the ship relative coordinate information.

[0048] Exemplarily, based on the ship relative coordinate information calculated in step S134, a directional navigation path can be generated for the target ship's approach and departure process.

[0049] The above method realizes the first conversion relationship between the ArUco label code coordinate system O c X c Y c Z c and the ship coordinate system O w X w Y w Z w , the second conversion relationship between the ship coordinate system O s X s Y s Z s. and the camera coordinate system O c X c Y c Z c , quickly and accurately determine the ship relative coordinate information of any point on the ship in the ArUco label code coordinate system O w X w Y w Z w , and further realize the accurate berthing and unberthing of the target ship on any side.

[0050] As Figure 2 shown, Figure 2 a schematic block diagram of a berth sign device 200 according to an embodiment of the present application is shown, the device 200 can include a berth sign image information board 210, a holder 220, a lifting device 230, a power supply 240, and a base 250. Wherein, the berth sign 210 can be used to display the berth sign image information; the holder 220 can be used to perform the steering operation on the berth sign 210, the lifting device 230 can be used to perform the lifting operation on the berth sign 210; the power supply 240 can be used to provide power for the berth sign device 200; and the base 250 is used to support the berth sign image information board 210, the holder 220, the lifting device 230 and the power supply 240.

[0051] In some examples, the method 100 can further include the following steps:

[0052] Step S150, obtaining the sailing direction and ship-shore distance of the target ship.

[0053] Specifically, the sailing direction of the target ship can be obtained by measurement of the target ship itself. The sailing direction of the target ship and the ship-shore distance information can also be obtained by scanning the berth marker image information through the camera. For example, the sailing direction of the target ship can be obtained based on the position coordinate information of the camera in the first scanning operation of the berth marker image information within a preset time and the position coordinate information of the camera in the last scanning operation of the berth marker image information within the preset time, and the ship-shore distance information can be obtained according to the position coordinate information of the camera in the last scanning operation of the berth marker image information.

[0054] In step S160, in the case that the ship-shore distance is less than a preset distance, the berth marker image information board is controlled to perform a turning operation based on the sailing direction, so that the angle information between the berth marker image information and the sailing direction is less than a preset angle, wherein the berth marker image information is displayed on the berth marker image information board.

[0055] For example, the ship-shore distance information can be the straight-line distance between the berth marker image and the camera. The preset distance can be the effective distance within which the camera can clearly scan the berth marker image information, for example, 50 meters. The preset angle can be set according to the ship size, the deflectable angle of the camera, or manually set according to the actual needs of the user, which is not limited here. For example, the preset angle can be 60 degrees. The above turning operation can be implemented based on the berth marker device 200 as shown in Figure 2 The above method makes the target ship scan the berth position coordinate information in the berth marker image information more quickly and conveniently by controlling the berth marker image information board to perform a turning operation based on the sailing direction of the target ship when the ship-shore distance is less than a preset distance, saves the operation steps of adjusting the ship body or the direction of the camera in the process of accurately scanning the berth marker image information, improves the scanning accuracy of the berth marker image information, and thus facilitates the target ship to more accurately and quickly complete the berthing and unberthing operations.

[0056] For example, Figure 3 FIG. 1 shows a schematic diagram of the information conversion relationship between the camera and the berth marker information board according to an embodiment of the present application. As shown in Figure 3 The camera arranged on the target ship can be used to scan the image information on the berth marker image information board fixed in the target berth. It should be noted that Figure 3 The number of berth marker information boards and the number of cameras are only examples and do not limit the number of cameras and berth marker information boards in the present method.

[0057] For example, Figure 4A schematic diagram of a process for processing berth sign image information is shown according to an embodiment of the present application. As shown, a camera scans image information on a berth sign image information board, and a data processing unit analyzes the berth sign image information to obtain position information and attitude information of a target ship relative to a target berth. In addition, the speed, heading, distance, and other information of the ship relative to the berth can be used to plan a route for the target ship to approach or leave the berth. Specifically, the route can be processed by augmented reality technology (AR), and the processed route can be displayed on a ship approach / leave berth auxiliary display module to assist the pilot in operation and decision-making. Figure 4

[0058] In some examples, the berth sign image information is obtained by a camera on the target ship. In the case of a fixed camera, step S160 of controlling the berth sign image information board to perform a turning operation based on the sailing direction can further include the following steps:

[0059] Step S161: Obtain the field of view angle range of the fixed camera.

[0060] For example, the field of view angle range of the fixed camera can be automatically obtained based on the brand information of the camera, for example, 90 degrees.

[0061] Step S162: Obtain the rotation angle and elevation information of the berth sign image information board coordinate system through a wireless data radio station.

[0062] Specifically, the berth sign image information board coordinate system is a coordinate system established with the geometric center of the berth sign image information board as the origin. It should be noted that this berth sign image information board coordinate system is a world coordinate system. The berth sign image information board is fixed in the target berth and is used to display relevant information of the target berth, including but not limited to the position coordinate information of the target berth. It can be understood that when the berth sign image is an ArUco label code, the berth sign image information board coordinate system and the ArUco label code coordinate system can be the same coordinate system. The wireless data radio station can be fixed on the target ship and the berth sign, and can be used for information transmission between the target ship and the target berth, especially for receiving relevant information of the berth sign image information board coordinate system of the target berth. Specifically, when the berth sign image is an ArUco label code, the rotation angle of the berth sign image information board coordinate system can be determined by obtaining the angle relationship matrix Rwc; the elevation information can be determined by Twc.

[0063] Step S163: Control the berth sign image information board to perform a rotation operation based on the field of view angle range of the fixed camera, the camera coordinate system, the rotation angle and elevation information of the berth sign image information board coordinate system, and the sailing direction.

[0064] ​Exemplarily, the camera coordinate system can be a camera coordinate system O c X c Y c Z c When the berth sign image is an ArUco label code, the berth sign image information board coordinate system can be an ArUco label code coordinate system O w X w Y w Z w The sailing direction is the heading direction of the target ship, which can be obtained by measuring the target ship itself.

[0065] The above method controls the berth sign image information board to perform a rotation operation by combining the field of view angle range of the fixed camera, the camera coordinate system, the rotation angle and the lifting information of the berth sign image information board coordinate system, and the sailing direction. In the case that the field of view angle range of the camera on the target ship is limited, the target berth and the target ship are measured in position and pose by combining the camera coordinate system and the berth sign image information board coordinate system, and the target ship sailing direction is combined to automatically control the berth sign image information board to perform a rotation operation, improve the efficiency and accuracy of the camera scanning the berth sign image information, and further improve the success rate and accuracy of the target ship completing the berthing and unberthing process.

[0066] In some examples, the berth sign image information is obtained by the camera on the target ship. In the case that the camera is a rotatable camera, step S160 of controlling the berth sign image information board to perform a turning operation based on the sailing direction can further include the following steps:

[0067] Step S164, obtaining the rotation angle of the rotatable camera.

[0068] Exemplarily, the rotation angle of the rotatable camera can be automatically obtained based on the camera's brand information, for example: 180 degrees.

[0069] Step S165, obtaining the rotation angle and lifting information of the berth sign image information board coordinate system through the wireless data radio station.

[0070] Exemplarily, the wireless data radio station can be fixed on the target ship and the berth sign, and can be used for information transmission between the target ship and the target berth, especially for receiving the relevant information of the berth sign image information board coordinate system of the target berth. Specifically, when the berth sign image is an ArUco label code, the rotation angle of the berth sign image information board coordinate system can be determined by obtaining the angle relationship matrix Rwc; the lifting height information can be determined by Twc.

[0071] Step S166, according to the rotation angle of the rotatable camera, the camera coordinate system, the rotation angle of the berth marker image information board coordinate system, the lifting information and the sailing direction, the berth marker image information is controlled to perform a rotation operation.

[0072] Exemplarily, the camera coordinate system can be a camera coordinate system O c X c Y c Z c When the berth marker image is an ArUco label code, the berth marker image information board coordinate system can be an ArUco label code coordinate system O w X w Y w Z w The sailing direction can be the heading direction of the target ship, which can be obtained by measuring the target ship itself.

[0073] The above method, by combining the rotation angle of the rotatable camera, the camera coordinate system, the rotation angle of the berth marker image information board coordinate system, the lifting information and the sailing direction, the berth marker image information board is controlled to perform a rotation operation, in the case that the rotation angle range of the rotatable camera on the target ship is limited, the target berth and the target ship are measured in pose by combining the camera coordinate system, the berth marker image information board coordinate system, and the sailing direction of the target ship is combined to automatically control the berth marker image information board to perform a rotation operation, improve the efficiency and accuracy of the rotatable camera scanning the berth marker image information, expand the effective area of the berth marker image that the rotatable camera can successfully scan, and further improve the success rate and accuracy of the target ship completing the berthing and unberthing process.

[0074] In some examples, the method 100 can further include the following steps:

[0075] Step S170, in the case that the target berth has at least two berth marker image information boards and multiple cameras distributed at different positions of the ship, a berth coordinate system is set with the midpoint of the berth line as the origin, and the arrangement positions of the multiple berth marker image information boards are measured and converted into coordinates of the berth coordinate system.

[0076] Exemplarily, it can be understood that the berth coordinate system is a world coordinate system. The arrangement positions of the multiple berth marker image information boards are measured and converted into coordinates of the berth coordinate system, which can be the horizontal (X-axis) distance parallel to the water surface direction, the vertical distance (Y-axis) in the longitudinal direction perpendicular to the water surface direction, and the vertical distance (Z-axis) perpendicular to the water surface direction of the multiple berth marker image information boards in the target berth relative to the midpoint of the berth line. After the position coordinates of each berth marker image information board relative to the midpoint of the berth line are converted and processed, the specific coordinates of each berth marker image information board in the final berth coordinate system are obtained.

[0077] Step S180, in the case of at least two berth marker image information boards in the target berth and multiple cameras distributed at different positions of the ship, a berth coordinate system is set with the midpoint of the berth line as the origin, and the arrangement positions of the multiple berth marker image information boards are measured and converted into coordinates of the berth coordinate system.

[0078] Step S190, the coordinate systems of different cameras relative to the same berth marker image information board are obtained and error averaging and coordinate system normalization processing are performed.

[0079] Exemplarily, the above method measures the coordinate systems of different cameras at different positions of the ship relative to the same berth marker image information board, and performs error averaging and coordinate system normalization processing, thereby improving the accuracy of the position and angle data of the camera coordinate system relative to the berth coordinate system, and placing multiple berth marker image information boards in the target berth can improve the scanning accuracy of the camera for the relevant information of the target berth. According to the second aspect of the present application, a computer vision-based ship berthing and unberthing route planning device is also proposed. Figure 5 A schematic block diagram of a computer vision-based ship berthing and unberthing route planning device 500 according to an embodiment of the present application is shown. As shown in the figure, Figure 5 The device 500 can include an acquisition unit 510, an identification unit 520, a generation unit 530, and a display unit 540.

[0080] The acquisition unit 510 is configured to acquire berth marker image information and berth marker image information board attitude information of a target berth.

[0081] The identification unit 520 is configured to identify berth position coordinate information of the target berth based on the berth marker image information.

[0082] The generation unit 530 is configured to generate a planned navigation path according to the target berth position coordinate information and current position coordinate information of a target ship.

[0083] The display unit 540 is configured to superimpose and display the target berth position coordinate information and the planned navigation path in a target video for enhanced display, wherein the target video is acquired by a camera.

[0084] According to a third aspect of the present application, an electronic device is also proposed. Figure 6 A schematic block diagram of an electronic device 600 according to an embodiment of the present application is shown. As shown in the figure, Figure 6 The electronic device 600 can include a processor 610 and a memory 620, wherein the memory 620 stores computer program instructions, and the computer program instructions are used to execute the above computer vision-based ship berthing and unberthing auxiliary method control method when executed by the processor 610.

[0085] According to a fourth aspect of the present application, there is also provided a storage medium having stored thereon program instructions which, when executed by a computer, cause the computer to carry out the computer vision based ship assisting berthing and unberthing method as described above. The storage medium may, for example, include a storage component of a tablet computer, a hard disk of a computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer readable storage medium can be any combination of one or more computer readable storage media.

[0086] The specific details and advantages of the computer vision based ship assisting berthing and unberthing method, the computer vision based ship berthing and unberthing route planning device, the electronic device and the storage medium can be understood by those skilled in the art through reading the above description of the computer vision based ship assisting berthing and unberthing method, and will not be described here in detail for the sake of brevity.

[0087] In several embodiments provided in the present application, it should be understood that the disclosed system, device and / or equipment can be implemented in other ways. For example, the above-described device embodiments are merely illustrative, for example, the division of units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e. they can be located in one place or distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0089] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0090] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the prior art that essentially contributes or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0091] The above, the above embodiments are only to illustrate the technical solutions of the present application, not to limit them; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A computer vision-based method for assisting ship berthing and unberthing, characterized in that, include: Acquire the berth marker image information and the attitude information of the berth marker image information board at the target berth; The berth location coordinates of the target berth are identified based on the berth marker image information; A planned navigation path is generated based on the target berth location coordinates and the target vessel's current location coordinates. The target berth location coordinates and the planned navigation path are overlaid on the target video to enhance the display, wherein the target video is acquired by a camera; The berth marker image information includes ArUco tag code information, which is used to mark the location information of the target berth. The step of generating a planned navigation path based on the target berth location coordinates and the target vessel's current location coordinates includes: Acquire camera coordinate information, ArUco tag code coordinate information, and ship coordinate information; A first transformation relationship between the camera coordinate information and the ArUco tag code coordinate information is determined, wherein the first transformation relationship includes the angle relationship and / or displacement relationship between the camera coordinate information and the ArUco tag code coordinate information; A second transformation relationship between the ship coordinate information and the camera coordinate information is determined, wherein the second transformation relationship includes the angular relationship between the ship coordinate information and the camera coordinate information; Based on the first transformation relationship, the second transformation relationship, and the ship coordinate information, the ship relative coordinate information is determined, wherein the ship relative coordinate information is the position information of the ship in the ArUco tag code coordinate system; The planned navigation path is generated based on the ship's relative coordinate information; Obtain the target vessel's sailing direction and distance from the shore; When the distance between the ship and the shore is less than a preset distance, the berth marker image information board is controlled to perform a turning operation based on the navigation direction, so that the angle between the berth marker image information and the navigation direction is less than a preset angle, wherein the berth marker image information is displayed on the berth marker image information board.

2. The computer vision-based ship berthing and unberthing method as described in claim 1, characterized in that, The berth marker image information was obtained through a camera on the target vessel. When the camera is a fixed camera, the step of performing a steering operation based on the navigation direction control berth marker image information board includes: Obtain the field of view angle range of the fixed camera; The rotation angle and lifting information of the coordinate system of the berth marker image information board are obtained through a wireless data transmission radio. The berth sign image information board is controlled to perform the steering operation based on the field of view of the fixed camera, the camera coordinate system, the rotation angle and lifting information of the berth sign image information board coordinate system, and the navigation direction.

3. The computer vision-based ship berthing and unberthing method as described in claim 2, characterized in that, The berth marker image information was obtained through a camera on the target vessel. When the camera is a rotatable camera, the steering operation based on the navigation direction control berth marker image information board includes: Obtain the rotation angle of the rotatable camera; The rotation angle and lifting information of the coordinate system of the berth marker image information board are obtained through a wireless data transmission radio. The steering operation is performed by controlling the berth sign image information based on the rotation angle of the rotatable camera, the camera coordinate system, the rotation angle and lifting information of the berth sign image information board coordinate system, and the navigation direction.

4. The computer vision-based ship berthing and unberthing method as described in claim 3, characterized in that, Also includes: In the case of at least two berth marker image information boards at the target berth and multiple cameras distributed at different locations on the vessel. A berth coordinate system is established with the midpoint of the berth line as the origin. The positions of multiple berth marker image information boards are measured and acquired, and then converted into coordinates of the berth coordinate system. Obtain the coordinate system of different cameras relative to the same berth sign information board, and perform error averaging and coordinate system normalization.

5. A computer vision-based ship berthing and unberthing auxiliary device, characterized in that, include: The acquisition unit is used to acquire the berth marker image information and the attitude information of the berth marker image information board of the target berth; The identification unit is used to identify the berth location coordinates of the target berth based on the berth marker image information; The generation unit is used to generate a planned navigation path based on the target berth location coordinate information and the target vessel's current location coordinate information; The display unit is used to overlay the target berth location coordinates and the planned navigation path onto the target video for enhanced display, wherein the target video is acquired by a camera; The berth marker image information includes ArUco tag code information, which is used to mark the location information of the target berth. The step of generating a planned navigation path based on the target berth location coordinates and the target vessel's current location coordinates includes: Acquire camera coordinate information, ArUco tag code coordinate information, and ship coordinate information; A first transformation relationship between the camera coordinate information and the ArUco tag code coordinate information is determined, wherein the first transformation relationship includes the angle relationship and / or displacement relationship between the camera coordinate information and the ArUco tag code coordinate information; A second transformation relationship between the ship coordinate information and the camera coordinate information is determined, wherein the second transformation relationship includes the angular relationship between the ship coordinate information and the camera coordinate information; Based on the first transformation relationship, the second transformation relationship, and the ship coordinate information, the ship relative coordinate information is determined, wherein the ship relative coordinate information is the position information of the ship in the ArUco tag code coordinate system; The planned navigation path is generated based on the ship's relative coordinate information; Obtain the target vessel's sailing direction and distance from the shore; When the distance between the ship and the shore is less than a preset distance, the berth marker image information board is controlled to perform a turning operation based on the navigation direction, so that the angle between the berth marker image information and the navigation direction is less than a preset angle, wherein the berth marker image information is displayed on the berth marker image information board.

6. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores computer program instructions, which are executed by the processor to perform the computer vision-based ship berthing and unberthing method as claimed in any one of claims 1 to 4.

7. A storage medium storing program instructions, which, when executed, perform the computer vision-based ship berthing and unberthing method as claimed in any one of claims 1 to 4.

Images