Method, system and equipment for identifying state of ship for berthing and unberthing operation and storage medium

Abstract

The embodiment of the invention provides a berthing and unberthing operation ship state identification method, system and device and a storage medium, and belongs to the technical field of ship control. According to the method, port image data are collected through a shore-based distance measuring module, the shore-based distance measuring module comprises a plurality of shore-based cameras arranged on a shore base, and ship identification is carried out on the port image data collected by the shore-based cameras through a target detection algorithm to obtain an image identification result; then determining a plurality of target cameras for shooting a target ship according to the image recognition result of each port image data, determining a target sensing network according to the plurality of target cameras, and carrying out ad hoc networking on shore-based cameras based on the target ship; the target ship is further subjected to binocular distance measurement through the target sensing network to obtain the ship position and the ship posture, then the ship state is displayed on the tug side terminal according to the ship position and the ship posture, the accurate ship state is provided for tugs in the mooring and unmooring operation, and the mooring and unmooring operation efficiency and safety are improved.

Description

Technical Field

[0001] The present application relates to the field of ship control technology, and in particular to a method, system, device and storage medium for identifying the state of a ship in berthing and unberthing operations. Background Art

[0002] As the tonnage and density of ships at ports and terminals continue to increase, ships with large tonnage and inertia have poor stability and operability when berthing and unberthing, resulting in low efficiency in berthing and unberthing operations and prone to safety accidents. Tugboats are special operation ships that provide professional water services such as towing, pushing, escorting, and rescue for large cargo ships. They are an important safety guarantee force for cargo ships entering and leaving ports and berthing and unberthing, and are an indispensable part of port production. Because the ships are relatively large, tugboat drivers are prone to visual blind spots, which makes it impossible for tugboats to efficiently and safely assist ships in berthing and unberthing operations. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose a method, system, device and storage medium for identifying the status of a ship in berthing and unberthing operations, aiming to provide accurate visual ship status information for tugboats in berthing and unberthing operations, thereby improving the efficiency and safety of berthing and unberthing operations.

[0004] To achieve the above-mentioned purpose, an embodiment of the present application provides a method for identifying the state of a berthing and unberthing ship, comprising the following steps:

[0005] The port image data is collected by a shore-based ranging module, wherein the shore-based ranging module includes a plurality of shore-based cameras arranged on the shore;

[0006] Using a target detection algorithm to perform ship recognition on the port image data collected by the plurality of shore-based cameras to obtain an image recognition result;

[0007] Determine a plurality of target cameras according to the image recognition results of the image data of each port, and determine a target perception network according to the plurality of target cameras, wherein the target camera is a shore-based camera for photographing the target ship;

[0008] Perform binocular ranging on the target ship through the target sensing network to obtain the ship position and ship attitude;

[0009] The ship status is displayed on the tugboat side terminal according to the ship position and the ship posture.

[0010] In some embodiments, the use of a target detection algorithm to perform ship recognition on the port image data collected by the plurality of shore-based cameras to obtain an image recognition result includes the following steps:

[0011] A ship detection model based on the Yolo target detection algorithm, wherein the ship detection model is used to identify the ship area in the image;

[0012] The port image data is input into a ship detection model based on the Yolo target detection algorithm to obtain an image recognition result in the port image data.

[0013] In some embodiments, the step of determining a plurality of target cameras according to the image recognition results of the image data of each port, and determining a target perception network according to the plurality of target cameras, comprises the following steps:

[0014] According to the image recognition results of each port image data, the port image data where the ship exists is selected as the first candidate image data;

[0015] Performing clarity detection and integrity detection on the ship area in the first candidate image data to obtain an availability score of the first candidate image data;

[0016] According to the availability scores of the first candidate image data, the first candidate image data are sorted in descending order, and a preset number of the first candidate image data ranked first are used as the second candidate image data;

[0017] A plurality of target cameras are determined according to the plurality of the second candidate image data.

[0018] In some embodiments, the step of determining a plurality of target cameras according to a plurality of the second candidate image data comprises the following steps:

[0019] Segmenting a plurality of ship regions in the second candidate image data according to an image recognition result of the second candidate image data to obtain a ship image data set of the second candidate image, wherein the ship image data set includes ship image data of the plurality of ship regions;

[0020] Selecting two ship image data from the ship image data sets of any two second candidate images respectively, and performing same ship detection based on the two ship image data, so as to divide the second candidate image data containing the same ship into the same image data set, and marking corresponding ship features for the image data set;

[0021] A corresponding image data set is determined according to the ship features of the target ship, and a camera that photographs the second candidate image data in the image data set is determined as the target camera.

[0022] In some embodiments, the method for identifying the state of a berthing and unberthing vessel further comprises the following steps:

[0023] Determine the initial ship trajectory through the target ship video captured by the target perception network;

[0024] Inputting the initial ship trajectory into a bidirectional long short-term memory network ship trajectory completion model based on graph attention to obtain a target ship trajectory;

[0025] The ship status is displayed at the tugboat side terminal according to the target ship trajectory.

[0026] In some embodiments, performing binocular ranging on the target ship through the target sensing network to obtain the ship position and ship attitude includes the following steps:

[0027] Performing binocular ranging on a first target point of the target ship through the target sensing network to obtain the position of the ship, wherein the first target point is located in the middle area of ​​the target ship;

[0028] Performing binocular ranging on a second target point and a third target point of the target ship through the target perception network to obtain a second target position and a third target position, wherein the second target point is located in a front end area of ​​the target ship, and the third target point is located in a rear end area of ​​the target ship;

[0029] The ship attitude is determined according to the second target position and the third target position.

[0030] In some embodiments, performing binocular ranging on the first target point of the target ship through the target perception network to obtain the ship position includes the following steps:

[0031] Select two target cameras from the target perception network to shoot the first target point to obtain image data of the two target points;

[0032] The position of the ship is determined according to the distance between the two selected target cameras, the focal length of the cameras and the shooting parallax of the image data of the two target points.

[0033] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application provides a system for identifying the state of a berthing and unberthing vessel, comprising:

[0034] The first module is used to collect port image data through a shore-based ranging module, wherein the shore-based ranging module includes a plurality of shore-based cameras arranged on the shore;

[0035] The second module is used to use a target detection algorithm to perform ship recognition on the port image data collected by the plurality of shore-based cameras to obtain an image recognition result;

[0036] The third module is used to determine a plurality of target cameras according to the image recognition results of the image data of each port, and determine a target perception network according to the plurality of target cameras, wherein the target camera is a shore-based camera for photographing the target ship;

[0037] The fourth module is used to perform binocular ranging on the target ship through the target perception network to obtain the ship position and ship attitude;

[0038] The fifth module is used to display the ship status at the tugboat side terminal according to the ship position and the ship posture.

[0039] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program implements the method described in the above embodiment when executed by the processor.

[0040] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium used for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method described in the above embodiment.

[0041] The method, system, device and storage medium for identifying the state of a ship in berthing and unberthing operations proposed in this application collect port image data through a shore-based ranging module. The shore-based ranging module includes multiple shore-based cameras arranged on the shore, and uses a target detection algorithm to perform ship recognition on the port image data collected by the multiple shore-based cameras to obtain image recognition results. Then, according to the image recognition results of each port image data, multiple target cameras for shooting target ships are determined, and a target perception network is determined based on the multiple target cameras. The shore-based cameras are self-organized based on the target ship to facilitate the subsequent acquisition of various visual information of the target ship. Further, binocular ranging is performed on the target ship through the target perception network to obtain the accurate ship position and ship attitude, and then the ship status is displayed on the tugboat side terminal according to the ship position and ship attitude, which can provide accurate visual ship status information for the tugboat in berthing and unberthing operations, and improve the efficiency and safety of berthing and unberthing operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a method for identifying the state of a berthing and unberthing ship provided in an embodiment of the present application;

[0043] Figure 2 It is a schematic diagram of a berthing and unberthing operation scenario provided in an embodiment of the present application;

[0044] Figure 3 It is a schematic diagram of the binocular ranging principle provided in an embodiment of the present application;

[0045] Figure 4 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] It should be noted that, although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0049] The embodiments of the present application provide a method, system, device and storage medium for identifying the status of a ship in berthing and unberthing operations, aiming to provide accurate visual ship status information for tugboats in berthing and unberthing operations, thereby improving the efficiency and safety of berthing and unberthing operations.

[0050] The method, system, device and storage medium for identifying the state of a ship in berthing and unberthing operations provided in the embodiments of the present application are specifically described through the following embodiments. First, the method for identifying the state of a ship in berthing and unberthing operations in the embodiments of the present application is described.

[0051] The method for identifying the state of a ship in berthing and unberthing operations provided in an embodiment of the present application relates to the field of ship control technology. The method for identifying the state of a ship in berthing and unberthing operations provided in an embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the method for identifying the state of a ship in berthing and unberthing operations, etc., but is not limited to the above forms.

[0052] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0053] Figure 1 is an optional flow chart of a method for identifying the state of a berthing and unberthing ship provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0054] Step S101, collecting port image data through a shore-based distance measurement module, wherein the shore-based distance measurement module includes a plurality of shore-based cameras arranged on the shore;

[0055] Step S102, using a target detection algorithm to perform ship recognition on the port image data collected by multiple shore-based cameras to obtain an image recognition result;

[0056] Step S103, determining a plurality of target cameras according to the image recognition results of the image data of each port, and determining a target perception network according to the plurality of target cameras, wherein the target camera is a shore-based camera for photographing the target ship;

[0057] Step S104, performing binocular ranging on the target ship through the target perception network to obtain the ship's position and attitude;

[0058] Step S105, displaying the ship status at the tugboat side terminal according to the ship position and ship attitude.

[0059] According to some embodiments of this application, please refer to Figure 2 This embodiment provides a port berthing and unberthing operation scenario, specifically:

[0060] Data measurement is performed on the shore, and the shore-based data measurement includes a shore-based ranging module and a shore-based communication module. Data processing is performed on the ship, and the ship-side data processing may include a ship-borne communication module and a central processing unit module. Enhanced display modules can be optionally set up on the ship, shore and tugboat ends. During berthing and unberthing operations, the shore-based ranging module transmits the collected information to the shore-based communication module, and the shore-based communication module transmits it to the central processing unit module on the ship to calculate the position of the target ship and give tugboat maneuvering suggestions.

[0061] The shore-based ranging module can be distributed on the sea side of the dock wall in an array manner and electrically connected to the shore-based communication module. The shore-based communication module collects the measurement information of the target ship by the self-organized target perception of the shore-based ranging module and sends the information to the ship-borne communication module. The ship-borne communication module is electrically connected to the central processing unit module deployed on the ship side. Among them, the shore-based ranging module can monitor the distance between the ship and the dock, the berthing and unberthing speed, and the berthing and unberthing angles in real time.

[0062] The central processing unit module deployed on the ship side includes a position calculation module, a position prediction module, a maneuvering decision module and a maneuvering suggestion module. After receiving information such as the distance of the target ship from the dock, the berthing and unberthing speed, and the berthing and unberthing angle, the position calculation module can calculate the attitude and position of the ship in this berthing and unberthing operation in real time and generate a historical trajectory. The position prediction module can predict the future state of the berthing and unberthing operation of the large ship through the built-in BP neural network algorithm based on the real-time attitude and position and historical trajectory of the ship in this berthing and unberthing operation calculated by the position calculation module, combined with the current port area’s wind speed, wind direction, tide level, flow velocity and flow direction and the ship’s hydrodynamic model, so as to timely discover dangerous berthing and unberthing operation behaviors. The maneuvering decision module can continuously deduce and test appropriate tugboat maneuvering suggestions based on the position calculation module and the position prediction module through the built-in adaptive learning neural network, so as to give a reasonable maneuvering process and risk response measures. The maneuvering suggestion module can convert the maneuvering process calculated by the maneuvering decision module into a simple and easy-to-understand maneuvering suggestion that conforms to the driver’s cognitive habits, and output it to the enhanced display module on the tugboat side. The maneuvering suggestions output by the maneuvering suggestion module received in the enhanced display module can be output to the tugboat driver and the shore-based dispatching command center in different forms.

[0063] For the tugboat driver, in the process of assisting the ship to berth and unberth according to the maneuvering suggestions, it is necessary to accurately obtain the real-time status of the ship. Therefore, the embodiment of the present application monitors the target ship through the camera array of the shore-based ranging module, and can process the image data at any end of the shore, ship or tugboat to determine the ship status information such as the posture and position of the target ship, and display it at the tugboat end, thereby realizing efficient and safe berthing and unberthing operations.

[0064] In step S101 of some embodiments, surveillance cameras are deployed at various locations in the port to form a shore-based ranging module to obtain port image data in real time. By deploying multiple cameras in the port, the port environment is fully monitored, thereby performing multi-angle monitoring of berthing and unberthing ships.

[0065] In step S102 of some embodiments, the target detection algorithm is a key technology in the field of computer vision, which is used to find and locate the target object of interest in the image or video. The target detection algorithm can adopt a target detection algorithm based on deep learning or a traditional target detection algorithm with manually designed features and classifiers, and can identify the ships in the image by learning the feature representation of the ships in the port image. In this embodiment, all ships in the port image data collected by the shore-based camera are identified to obtain an image recognition result, which can be an area where ships exist in the port image.

[0066] In some embodiments, step S102 may include but is not limited to steps S201 to S202:

[0067] Step S201, a ship detection model based on the Yolo target detection algorithm, wherein the ship detection model is used to identify the ship area in the image;

[0068] Step S202: input the port image data into a ship detection model based on the Yolo target detection algorithm to obtain an image recognition result in the port image data.

[0069] In this embodiment, in the task of detecting ship targets in port areas, the target detection algorithm is required to identify the positions of all ships in the image as accurately as possible. Compared with the ship detection task under the ocean background, the ship target detection in port areas has the characteristics of complex background, many obstructing objects and many interference features. The traditional ship target detection algorithm performs poorly in this task. The embodiment of the present application can adopt a target detection algorithm based on deep learning, which can realize accurate identification of ships by automatically learning the port features in the image. It has strong learning and anti-interference capabilities, better robustness, and can identify targets more accurately in complex scenes, which is more suitable for port ship detection tasks.

[0070] Specifically, the ship target detection algorithm based on deep learning can be divided into two categories: two-stage algorithm and one-stage algorithm. The two-stage algorithm needs to generate several candidate frames according to the input image first, and then complete the detection through the convolutional neural network, so its detection speed is relatively slow, for example, the R-CNN series of algorithms (including R-CNN, Fast R-CNN, Faster R-CNN, etc.). The one-stage algorithm directly treats the detection as a regression problem and does not need to generate candidate frames. The algorithm has a faster detection speed, for example, the YOLO series of algorithms (including YOLOv1, YOLOv2, YOLOv3, YOLOv4, YOLOv5) and SSD (Single Shot MultiBox Detector). The YOLO (You Only Look Once) series of algorithms has fast detection speed and high accuracy. In this embodiment, the Yolo target detection algorithm can be used to construct a ship detection model, and then the ship detection model is used to realize the recognition of ships in the port image data.

[0071] In step S103 of some embodiments, the port image data captured by different cameras at the same time have different perspectives and contents. Some cameras may be able to capture a target ship that a tugboat needs assistance, while some cameras may not. Therefore, this embodiment needs to match the same ship based on the image recognition results of each port image data, so as to determine the camera that can capture the same target ship as the target camera, and multiple target cameras share a common target perception network.

[0072] In some embodiments, step S103 may include but is not limited to steps S301 to S304:

[0073] Step S301, selecting port image data with a ship area as first candidate image data according to the image recognition results of each port image data;

[0074] Step S302, performing a clarity detection and a completeness detection on the ship area in the first candidate image data to obtain an availability score of the first candidate image data;

[0075] Step S303, sorting the first candidate image data in descending order according to the availability score of each first candidate image data, and taking a preset number of first candidate image data ranked first as second candidate image data;

[0076] Step S304: determining a plurality of target cameras according to the plurality of second candidate image data.

[0077] In this embodiment, due to the harsh port environment, the camera may be contaminated, resulting in the inability to capture clear images. In addition, environmental factors such as temperature, humidity, and seawater salinity may also affect camera performance and video effects. Therefore, in this embodiment, cameras that can capture clear, unobstructed images of the ship can be screened out from all cameras, and subsequent data analysis can be performed based on the screened cameras to obtain more accurate ship status information.

[0078] Specifically, first, according to the image recognition results of each port image data, the port image data with the ship area is selected as the first candidate image data, which is equivalent to selecting the camera that can shoot the ship from all cameras. Secondly, the clarity detection and integrity detection of the ship area in the first candidate image data are performed. The clarity of the image is related to the gradient change of the edge pixels of the object. The clarity detection can use the edge detection operator to detect the edge, and then calculate the gradient amplitude of the edge, and count these amplitudes to obtain a metric representing the clarity of the image. The larger the sum or average of the gradient amplitude, the clearer the image; the integrity of the ship area of ​​the image can characterize whether the ship is blocked from shooting. The integrity detection can use the edge detection operator to detect the edge of the ship, and then calculate the closure of each edge curve. The higher the overall closure, the higher the integrity of the ship area. After calculating the clarity and integrity of the ship area in the first candidate image data respectively, the clarity and integrity are weighted to obtain the availability score of the first candidate image data. Finally, the first candidate image data are sorted in descending order according to the availability score, and a preset number of first candidate image data ranked first are used as second candidate image data for subsequent analysis, so as to screen out several cameras with clear shooting and good field of view of the ship.

[0079] In some embodiments, step S304 may include but is not limited to steps S401 to S403:

[0080] Step S401, segmenting a plurality of ship regions in the second candidate image data according to the image recognition result of the second candidate image data to obtain a ship image data set of the second candidate image, wherein the ship image data set includes ship image data of the plurality of ship regions;

[0081] Step S402, selecting two ship image data from the ship image data sets of any two second candidate images, respectively, and performing identical ship detection based on the two ship image data, so as to divide the second candidate image data containing the identical ship into the same image data set, and marking the corresponding ship features for the image data set;

[0082] Step S403: determining a corresponding image data set according to the ship features of the target ship, and determining a camera that captures the second candidate image data in the image data set as the target camera.

[0083] In this embodiment, each second candidate image data may contain different ships. This embodiment needs to further segment the second candidate image data into multiple ship areas according to the image recognition results of the second candidate image data, and then match the ship areas of any two second candidate image data, and then divide the second candidate image data that can capture the same ship into a group, so as to realize mutual recognition of ship information between different cameras, and then realize self-organizing network detection of perception information between different cameras.

[0084] Specifically, the same ship detection method across cameras can be implemented based on the Siamese network. The Siamese network algorithm is used to identify whether the ships photographed by cameras at different horizontal lines and angles are the same ship. The Siamese network is a two-branch neural network model that can be used to judge the similarity of input sample pairs (i.e., ship image data). The Siamese network consists of two sub-networks with the same structure and shared weights, and the sub-networks are used to extract features of input sample pairs. The Siamese network can receive two input samples X1 and X2 at the same time, and then use two weight-shared feature extraction networks to extract the features G(X1) and G(X2) of X1 and X2 respectively, and then use the similarity calculation formula to calculate the feature distance between G(X1) and G(X2), thereby obtaining their feature similarity. The calculation formula is as follows:

[0085] E w (X1,X2)=f(G(X1),G(X2));

[0086] Among them, f is the similarity calculation function, and the Euclidean distance is generally selected. w Between 0 and 1, when the features of X1 and X2 are closer, E w The larger the result, the smaller the w The smaller. Compared with traditional deep learning algorithms, the twin network can better learn the difference between similar sample pairs and non-similar sample pairs through two weight-shared feature extraction networks, reduce the loss value between similar samples during training, and amplify the loss value between non-similar samples. Therefore, it can obtain stronger generalization ability when there are only a small number of samples in the training data set and reduce the occurrence of overfitting. By judging similar features, it can be determined whether the input sample pairs are the same.

[0087] Furthermore, by performing the above process on the ship regions in any two image data, the second candidate image data containing the same ship can be divided into the same image data set, and the corresponding ship features (such as ship model) are marked for the image data set. The corresponding image data set is determined according to the ship features of the target ship that the tugboat is assisting, and the camera that took the second candidate image data in the image data set is determined as the target camera.

[0088] In some embodiments, the method for identifying the state of a berthing and unberthing vessel in the embodiment of the present application may also include, but is not limited to, steps S501 to S503:

[0089] Step S501, determining an initial ship trajectory through a target ship video captured by a target perception network;

[0090] Step S502, inputting the initial ship trajectory into the bidirectional long short-term memory network ship trajectory completion model based on graph attention to obtain the target ship trajectory;

[0091] Step S503: Display the ship status at the tugboat terminal according to the target ship trajectory.

[0092] In this embodiment, after the twin network solves the cross-camera recognition problem of the ship, the target camera can be further used to display the ship status according to the ship trajectory and the target ship trajectory on the tugboat side terminal, so that the tugboat driver can operate better. When the target camera tracks the ship, there may be a situation where the trajectory is missing. This embodiment uses a bidirectional LSTM ship trajectory completion model based on graph attention to complete the ship trajectory data and obtain a complete target ship trajectory. This embodiment introduces a graph attention mechanism (Graph Attention) based on the original Seq2Seq model. The model integrates time features and spatial interaction features between ships to improve the completion accuracy of the ship trajectory. In view of the particularity of the trajectory completion problem, this embodiment can adopt a dual-stream mechanism, and the two missing trajectories before and after are respectively input into the encoder for encoding, merged and input into the decoder, and then output as a complete trajectory. When it is judged that the Euclidean distance between the two outputs is less than 0.05 or the maximum number of completed frames is reached (for example, set to 200, which is about 12s trajectory), the completion is stopped. In view of the time series characteristics of ship navigation, the LSTM module is first used in the encoder to extract the time series characteristics of the trajectory sequence, focusing on the motion mode of the target ship itself. The preliminary encoding result is recorded as:

[0093]

[0094] in, For The embedding coding result, W m Represents the LSTM weight.

[0095] The graph attention model is used in the encoder to deal with the interactive behaviors of the target ship during navigation, such as avoiding, overtaking, meeting, and following. The model regards all ships confirmed by the twin network in the field of view as nodes and assigns importance, aggregating neighbor information. The preliminary temporal encoding result is input into the graph attention model, and the output is:

[0096]

[0097] in, is the attention coefficient of the node pair (i, j) at time t, W is the dimension mapping matrix, represents the set of neighboring nodes of node i, and σ is the LeaklyReLU activation function. To aggregate temporal and spatial features, the LSTM module is used again. At time point t, node i has the following characteristics: Output after graph attention and LSTM encoding again In the last layer, the two are spliced ​​into the input state layer to complete the global trajectory of the target ship.

[0098] In step S104 of some embodiments, the automatically constructed target perception network is used to perceive the target ship with high precision, thereby identifying the ship status, which includes but is not limited to the ship position, ship attitude and ship trajectory.

[0099] In some embodiments, step S104 may include but is not limited to steps S601 to S602:

[0100] Step S601, performing binocular ranging on a first target point of a target ship through a target sensing network to obtain the ship position, wherein the ship position is in the middle area of ​​the target ship;

[0101] Step S602, performing binocular ranging on a second target point and a third target point of the target ship through a target perception network to obtain a second target position and a third target position, wherein the second target point is located in a front region of the target ship, and the third target point is located in a rear region of the target ship;

[0102] Step S603, determining the ship posture according to the second target position and the third target position.

[0103] In this embodiment, the ship can be divided into three sections, namely the front area, the middle area and the rear area. The first target point is selected in the middle area to represent the target ship position, and the second target point and the third target point are selected in the front area and the rear area. The relative position relationship between the second target point and the third target point represents the ship posture. Two target cameras are randomly selected from the target perception network. The position of the ship can be obtained by measuring the position of the first target point based on the binocular ranging principle; the position of the second target point and the third target point is measured based on the binocular ranging principle, and the ship posture can be determined by performing phase position analysis.

[0104] In some embodiments, step S601 may include but is not limited to steps S701 to S702:

[0105] Step S701, selecting two target cameras from the target perception network to shoot a first target point to obtain image data of two target points;

[0106] Step S702, determining the position of the ship according to the distance between the two selected target cameras, the focal length of the cameras and the shooting parallax of the image data of the two target points.

[0107] In this embodiment, the first target point is taken as an example to illustrate the binocular ranging principle of this embodiment. The measurement methods of the second and third target points are the same as those of the first target point, and will not be repeated here. Figure 3 , the principle of binocular ranging is as follows:

[0108] The imaging planes of the two target cameras are parallel to each other, the main optical axes of the two cameras are parallel, the distance between the optical centers is B, and the focal lengths of the cameras are both f, c left is the center principal point of camera 1, c right is the central principal point of camera 2, and the projection coordinates of the first target point M on the left and right imaging planes are M left (x lelf ,y left ) and M right (x right ,y right ), y left =y right =Z, then from the trigonometric relationship we get:

[0109]

[0110] M left M right =B-(x lelf -x right );

[0111]

[0112] From the above relationship, we can know that in binocular stereo vision, the distance Z from the first target point to the camera, the focal length f of the camera, the base distance B between the two cameras, and the parallax x lelf -x right The focal length and baseline are two known parameters, and only the parallax is an unknown quantity. The parallax can be obtained by calculating the parallax of all pixels in the left and right imaging pictures. Combining the above parameters can realize the accurate measurement of the ship's position.

[0113] In step S105 of some embodiments, the real-time ship status is displayed on the operating tugboat, so that the tugboat driver can easily view the berthing and unberthing status of the ship from different perspectives at any time. The ship status can be displayed in the form of a map, that is, the ship position, ship posture and trajectory information are displayed in the port area map.

[0114] The present application also provides a system for identifying the state of a berthing and unberthing vessel, including:

[0115] The first module is used to collect port image data through a shore-based ranging module, wherein the shore-based ranging module includes a plurality of shore-based cameras arranged on the shore;

[0116] The second module is used to use the target detection algorithm to perform ship recognition on the port image data collected by multiple shore-based cameras to obtain image recognition results;

[0117] The third module is used to determine multiple target cameras according to the image recognition results of the image data of each port, and determine the target perception network according to the multiple target cameras, wherein the target camera is a shore-based camera that photographs the target ship;

[0118] The fourth module is used to perform binocular ranging of the target ship through the target perception network to obtain the ship's position and attitude;

[0119] The fifth module is used to display the ship status at the tugboat side terminal according to the ship position and ship attitude.

[0120] It can be understood that the contents of the above-mentioned embodiments of the method for identifying the state of a ship for berthing and unberthing operations are applicable to the embodiments of the present system, the functions specifically implemented by the embodiments of the present system are the same as those in the above-mentioned embodiments of the method for identifying the state of a ship for berthing and unberthing operations, and the beneficial effects achieved are also the same as the beneficial effects achieved by the above-mentioned embodiments of the method for identifying the state of a ship for berthing and unberthing operations.

[0121] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned method for identifying the state of a berthing and unberthing ship. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0122] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0123] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0124] The memory 402 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the state identification method of the berthing and unberthing operation ship in the embodiment of this application;

[0125] Input / output interface 403, used to implement information input and output;

[0126] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0127] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0128] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0129] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned method for identifying the status of a berthing and unberthing ship.

[0130] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0132] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0133] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0134] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0135] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0136] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0137] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0138] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0141] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for identifying the state of a ship in berthing and unberthing operations, characterized in that: The following steps are involved: The port image data is collected by a shore-based ranging module, wherein the shore-based ranging module includes a plurality of shore-based cameras arranged on the shore; Using a target detection algorithm to perform ship recognition on the port image data collected by the plurality of shore-based cameras to obtain an image recognition result; Determine a plurality of target cameras according to the image recognition results of the image data of each port, and determine a target perception network according to the plurality of target cameras, wherein the target camera is a shore-based camera for photographing the target ship; Perform binocular ranging on the target ship through the target sensing network to obtain the ship's position and attitude; The ship status is displayed on the tugboat side terminal according to the ship position and the ship posture.

2. The method according to claim 1, characterized in that The method of using a target detection algorithm to perform ship recognition on the port image data collected by the plurality of shore-based cameras to obtain an image recognition result includes the following steps: A ship detection model based on the Yolo target detection algorithm, wherein the ship detection model is used to identify the ship area in the image; The port image data is input into a ship detection model based on the Yolo target detection algorithm to obtain an image recognition result in the port image data.

3. The method for identifying the state of a berthing and unberthing vessel according to claim 1, characterized in that: Determining a plurality of target cameras according to the image recognition results of the image data of each port, and determining a target perception network according to the plurality of target cameras, comprises the following steps: According to the image recognition results of each port image data, the port image data where the ship exists is selected as the first candidate image data; Performing clarity detection and integrity detection on the ship area in the first candidate image data to obtain an availability score of the first candidate image data; According to the availability scores of the first candidate image data, the first candidate image data are sorted in descending order, and a preset number of the first candidate image data ranked first are used as the second candidate image data; A plurality of target cameras are determined according to the plurality of the second candidate image data.

4. The method for identifying the state of a ship in berthing and unberthing operations according to claim 3, characterized in that: The step of determining a plurality of target cameras according to a plurality of the second candidate image data comprises the following steps: Segmenting a plurality of ship regions in the second candidate image data according to an image recognition result of the second candidate image data to obtain a ship image data set of the second candidate image, wherein the ship image data set includes ship image data of the plurality of ship regions; Selecting two ship image data from the ship image data sets of any two second candidate images respectively, and performing same ship detection based on the two ship image data, so as to divide the second candidate image data containing the same ship into the same image data set, and marking corresponding ship features for the image data set; A corresponding image data set is determined according to the ship features of the target ship, and a camera that photographs the second candidate image data in the image data set is determined as the target camera.

5. The method for identifying the state of a berthing and unberthing vessel according to claim 1, characterized in that: The method for identifying the state of a berthing and unberthing ship further comprises the following steps: Determine the initial ship trajectory through the target ship video captured by the target perception network; Inputting the initial ship trajectory into a bidirectional long short-term memory network ship trajectory completion model based on graph attention to obtain a target ship trajectory; The ship status is displayed at the tugboat side terminal according to the target ship trajectory.

6. The method for identifying the state of a berthing and unberthing vessel according to claim 1, characterized in that: The method of performing binocular ranging on the target ship through the target sensing network to obtain the ship position and ship attitude includes the following steps: Performing binocular ranging on a first target point of the target ship through the target sensing network to obtain the position of the ship, wherein the first target point is located in the middle area of ​​the target ship; Performing binocular ranging on a second target point and a third target point of the target ship through the target perception network to obtain a second target position and a third target position, wherein the second target point is located in a front end area of ​​the target ship, and the third target point is located in a rear end area of ​​the target ship; The ship attitude is determined according to the second target position and the third target position.

7. The method for identifying the state of a berthing and unberthing vessel according to claim 6, characterized in that: The method of performing binocular ranging on the first target point of the target ship through the target perception network to obtain the ship position includes the following steps: Select two target cameras from the target perception network to shoot the first target point to obtain image data of the two target points; The position of the ship is determined according to the distance between the two selected target cameras, the focal length of the cameras and the shooting parallax of the image data of the two target points.

8. A ship status identification system for berthing and unberthing operations, characterized in that: include: The first module is used to collect port image data through a shore-based ranging module, wherein the shore-based ranging module includes a plurality of shore-based cameras arranged on the shore; The second module is used to use a target detection algorithm to perform ship recognition on the port image data collected by the plurality of shore-based cameras to obtain an image recognition result; The third module is used to determine a plurality of target cameras according to the image recognition results of the image data of each port, and determine a target perception network according to the plurality of target cameras, wherein the target camera is a shore-based camera for photographing the target ship; The fourth module is used to perform binocular ranging on the target ship through the target perception network to obtain the ship position and ship attitude; The fifth module is used to display the ship status at the tugboat side terminal according to the ship position and the ship posture.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 7.

Images