Ship following method, system and equipment based on binocular vision and storage medium

Abstract

The embodiment of the invention provides a ship following method, system and equipment based on binocular vision and a storage medium, and belongs to the technical field of ship autonomous control. According to the method, a binocular camera collects target image data, a target detection model is used to carry out target tracking ship identification on the target image data of the binocular camera, and a target area in the target image data is determined; performing parallax estimation on target areas in the two target image data by adopting a semi-global matching algorithm to obtain a sub-pixel parallax, determining a relative distance between the target tracking ship and the ship according to the sub-pixel parallax, and performing motion state prediction on the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship; and finally, determining a following control instruction of the ship according to the first situation information pair so as to follow the target tracking ship. According to the invention, the robustness and environmental adaptability of autonomous following control of the ship can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of autonomous control of ships, and particularly relates to a ship following method and system based on binocular vision, a device and a storage medium. BACKGROUND

[0002] The existing ship formation following control technology excessively relies on multi-sensor data fusion, resulting in high system hardware cost, complex structure and single-point failure risk. Meanwhile, the cross-modal processing of multi-source heterogeneous data leads to real-time degradation and feature loss in the information abstraction process, which is difficult to meet the rapid and accurate perception demand of the tracked target ship motion situation under dynamic sea conditions, resulting in poor ship formation coordination.

[0003] With the rapid development of the camera industry, binocular pure vision technology has unique advantages in real-time perception of target relative distance due to its bionic stereo perception mechanism. However, the existing binocular ranging method has the problem of insufficient perception accuracy in complex motion scenes. For example, the baseline tilt error caused by the roll of the ship will lead to inaccurate target distance obtained by binocular ranging, thereby affecting the robustness and environmental adaptability of ship autonomous following control. SUMMARY

[0004] The main purpose of the embodiments of the present application is to provide a ship following method, system, device and storage medium based on binocular vision, aiming to improve the robustness and environmental adaptability of ship autonomous following control.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application provides a ship following method based on binocular vision, comprising the following steps:

[0006] acquiring target image data by a binocular camera;

[0007] using a target detection model to perform target tracking ship identification on the target image data of the binocular camera, and determining a target region in the target image data;

[0008] using a semi-global matching algorithm to perform disparity estimation on the target region in the two target image data, to obtain sub-pixel disparity;

[0009] determining the relative distance between the target tracking ship and the ship according to the sub-pixel disparity;

[0010] performing motion state prediction on the target tracking ship according to the relative distance, to obtain first situation information of the target tracking ship;

[0011] determining a following control instruction of the ship according to the first situation information, to follow the target tracking ship.

[0012] In some embodiments, the target image data collected by the binocular camera includes the following steps:

[0013] The initial image data collected by the binocular camera is subjected to monocular semantic segmentation processing for target tracking ships to obtain a target contour of the binocular camera;

[0014] A stereo matching process is performed according to the target contour of the binocular camera to obtain an initial value of a disparity map, and a target distance estimation value is determined according to the initial value of the disparity map;

[0015] The baseline distance of the binocular camera is adjusted according to the range interval in which the target distance estimation value is located;

[0016] The target image data is collected by the binocular camera after the baseline distance is adjusted.

[0017] In some embodiments, the target detection model is used to identify the target tracking ships in the target image data collected by the binocular camera to determine the target region in the target image data, including the following steps:

[0018] The target image data is input into a target detection model based on a Yolov5s network to obtain a detection frame output by the target detection model, and the detection frame is used to determine the target region in the target image data;

[0019] The target detection model includes a backbone network, a connection network and a detection network connected in sequence; the backbone network includes a lightweight channel attention module, and the lightweight channel attention module is used to learn the importance weight of each feature channel; the connection network adopts a spatial channel dual-path attention mechanism for feature fusion.

[0020] In some embodiments, the semi-global matching algorithm is used to estimate the disparity of the target region in the two target image data to obtain sub-pixel disparity, including the following steps:

[0021] The disparity estimation values of the target feature points and the adjacent feature points in the target region in the image data collected by the binocular camera are determined respectively;

[0022] The matching cost of the target feature points and the adjacent feature points is determined according to the disparity estimation values of the target feature points and the adjacent feature points by a matching cost function;

[0023] The disparity error is determined according to the matching cost of the target feature points and the adjacent feature points;

[0024] The disparity estimation value of the target feature points is optimized according to the disparity error to obtain the sub-pixel disparity of the target feature points.

[0025] In some embodiments, the motion state prediction of the target tracking ship according to the relative distance comprises the following steps:

[0026] determining the position of the target tracking ship in a dynamic coordinate system according to the relative distance and the relative orientation, wherein the dynamic coordinate system takes the center of the binocular camera as the origin;

[0027] determining the motion state vector of the target tracking ship at different times according to the dynamic coordinate system;

[0028] adopting a Kalman particle filtering hybrid algorithm to estimate the motion state of the motion state vector at the next time to obtain the first situation information of the target tracking ship.

[0029] In some embodiments, the motion state prediction of the target tracking ship according to the relative distance further comprises the following steps:

[0030] In the case of observing the target ship heading from the ship automatic identification system, correcting the relative velocity parameter in the motion state vector according to the target ship heading.

[0031] In some embodiments, the determination of the following control instruction of the ship according to the first situation information comprises the following steps:

[0032] determining a ship state vector according to the first situation information, the second situation information of the ship, and the control amount of the power component of the ship at the last time;

[0033] adopting a Markov decision model to make a control increment decision of the power component of the ship according to the ship state vector to obtain the following control instruction of the ship.

[0034] To achieve the above-mentioned purpose, another aspect of the embodiment of the present application proposes a ship following system based on binocular vision, comprising:

[0035] The first module is configured to collect target image data through a binocular camera;

[0036] The second module is configured to identify the target tracking ship in the target image data of the binocular camera respectively by using a target detection model to determine the target region in the target image data;

[0037] The third module is configured to estimate the disparity of the target region in the two target image data by using a semi-global matching algorithm to obtain sub-pixel disparity;

[0038] The fourth module is configured to determine the relative distance between the target tracking ship and the ship according to the sub-pixel disparity.

[0039] a fifth module, performing motion state prediction on the target tracking ship according to the relative distance, to obtain first situation information of the target tracking ship;

[0040] a sixth module, determining a following control instruction of the ship according to the first situation information, to follow the target tracking ship.

[0041] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory, and the program is executed by the processor to realize the method in the above embodiment.

[0042] To achieve the above object, another aspect of the embodiment of the present application provides a storage medium, which is a computer readable storage medium, for computer readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the method in the above embodiment.

[0043] The ship following method, system, device and storage medium based on binocular vision provided in the present application collect target image data through a binocular camera, use a target detection model to respectively identify target tracking ships for the target image data of the binocular camera, determine target regions in the target image data, then use a semi-global matching algorithm to perform disparity estimation on the target regions in the two target image data to obtain sub-pixel disparity, then determine the relative distance between the target tracking ship and the ship according to the sub-pixel disparity, perform motion state prediction on the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship, and finally determine a following control instruction of the ship according to the first situation information to follow the target tracking ship. The semi-global matching algorithm is used to realize sub-pixel level dynamic measurement of the distance between the two ships, which can reduce the disparity estimation error caused by lens shaking due to complex maritime environment, improve the accuracy of dynamic relative distance measurement, and thus improve the robustness and environmental adaptability of the ship autonomous following control. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 is a flowchart of the ship following method based on binocular vision provided in the embodiment of the present application;

[0045] Figure 2 is a hardware architecture schematic diagram of the ship following method provided in the embodiment of the present application;

[0046] Figure 3 is a dynamic coordinate system schematic diagram provided in the embodiment of the present application;

[0047] Figure 4 Fig. 1 is a schematic diagram of a hardware structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0049] It should be noted that although the functional modules are divided in the system and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0051] First, the terms involved in the present application are analyzed:

[0052] Yolov5s is a version of YOLO, a YOLO model (You Only Look Once: Unified, Real-Time Object Detection, a target detection system based on a single neural network) is a deep learning algorithm in computer vision technology that can be used for image recognition. The YOLO model converts the target detection problem into a Regression regression problem, that is, given an input image, directly regress the bounding box (bounding box) and its classification category on multiple positions of the image. The YOLO model includes but is not limited to Yolov5s and other versions, and different versions of the model differ in the weight, network structure or region sampling method used.

[0053] At present, ship following navigation for target perception depends on multi-source heterogeneous data fusion technology (mostly AIS, GPS, camera, laser radar and other data). Although this technology can effectively improve the perception accuracy and reduce the interference caused by environmental disturbance, it still has the following limitations: 1. High hardware cost, multi-sensor integration leads to system complexity and cost multiplication; 2. High failure rate, system reliability is subject to single-point sensor failure - data-level fusion is highly sensitive to the integrity of the original data, when any sensor fails, it will cause verification error; 3. Real-time bottleneck is prominent: feature-level and decision-level fusion need to perform cross-modal feature alignment or preliminary decision-making, this step will lose a lot of information in the process of information abstraction and has high delay, it is difficult to make timely decisions in emergency scenarios.

[0054] Compared with the above, with the rapid development of the camera industry, binocular pure vision technology has unique advantages with bionic stereo perception mechanism, by determining the target parallax through stereo matching, the target can be measured. However, the current binocular ranging method has the problems of limited range of view, cumulative baseline tilt error caused by ship roll and parallax lag effect caused by relative motion in complex motion scene, which restricts the robustness and environmental adaptability of ship autonomous following control.

[0055] Based on this, the embodiment of the application provides a ship following method, system, device and storage medium based on binocular vision, aiming to improve the robustness and environmental adaptability of ship autonomous following control.

[0056] The ship following method, system, device and storage medium based on binocular vision provided by the embodiment of the application are specifically explained by the following embodiments, first, the ship following method based on binocular vision in the embodiment of the application is described.

[0057] The ship following method based on binocular vision provided in the embodiments of the present application relates to the technical field of autonomous ship control. The ship following method based on binocular vision provided in the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in a terminal or a server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as a separate physical server, can be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application program for implementing the ship following method based on binocular vision, but is not limited to the above forms. In an example, the terminal or the server for implementing the ship following method based on binocular vision of the embodiments of the present application can be a terminal or a server located in a formation ship.

[0058] In an application architecture embodiment of a ship following method, please refer to Figure 2 The architecture includes a perception layer, a decision layer, and a control layer. The perception layer is mainly composed of a perception unit, the decision layer is mainly composed of a data processing unit, and the control layer is mainly composed of a control execution unit. The perception unit is mainly composed of an industrial-grade binocular camera array with adjustable baseline distance and an IMU (Inertial Measurement Unit). The binocular camera array is equipped with a wide-angle anti-glare lens, which can scan the longitudinal environment of the ship in real time and obtain high-definition images of the target ship. The IMU inertial measurement unit is fixed to the ship body and is responsible for monitoring the motion situation of the ship in real time, such as ship heading, speed, acceleration, and turning rate. Further, the perception layer can also include an automatic temperature control system, which is used to make the sensor output stable images and inertial navigation data at an appropriate working temperature.

[0059] The data processing unit is mainly composed of a shipborne server and a switch. The shipborne server is responsible for processing and analyzing perception data, evaluating the situation of the ship, determining driving decisions, and issuing control instructions. The switch is responsible for information exchange between various sensor data and the server.

[0060] The control execution unit is mainly composed of a ship propulsion controller and a rudder control module, which is responsible for executing the control instructions issued by the shipborne server.

[0061] Figure 1 is an optional flowchart of the ship following method based on binocular vision provided in the embodiments of the present application, Figure 1 The method in the flowchart can include but is not limited to steps S101 to S106.

[0062] Step S101, collecting target image data through a binocular camera;

[0063] Step S102, performing target tracking and ship identification on the target image data of the binocular camera respectively by using a target detection model to determine a target region in the target image data;

[0064] Step S103, performing disparity estimation on the target regions in the two target image data by using a semi-global matching algorithm to obtain sub-pixel disparity;

[0065] Step S104, determining the relative distance between the target tracking ship and the own ship according to the sub-pixel disparity;

[0066] Step S105, predicting the motion state of the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship;

[0067] Step S106, determining the following control instruction of the own ship according to the first situation information to follow the target tracking ship.

[0068] In step S101 of some embodiments, the target image data refers to the image containing the target tracking ship collected by the binocular camera on the own ship, and the binocular camera collects the target image data at the same time. Since the magnification of different parts of the lens is different and the installation position is not parallel to the imaging plane, radial distortion and tangential distortion will occur, which are independent of each other and will seriously affect distance estimation. Therefore, the image data collected by the camera needs to be adjusted to eliminate the error caused by the camera array and the lens shooting through distortion correction and epipolar rectification. The correction formula is as follows:

[0069]

[0070] Where k1, k2, k3 are radial distortion coefficients; p1, p2 are tangential distortion coefficients; is the coordinate of the image feature point before correction; is the coordinate of the image feature point after correction.

[0071] According to some embodiments of the present application, step S101 can include but is not limited to the following steps:

[0072] Step S201, performing monocular semantic segmentation processing on the initial image data collected by the binocular camera for the target tracking ship to obtain the target contour of the binocular camera;

[0073] Step S202, performing stereo matching processing according to the target contour of the binocular camera to obtain a disparity map initial value, and determining a target distance estimation value according to the disparity map initial value;

[0074] Step S203, adjusting the baseline distance of the binocular camera according to the range interval where the target distance estimate value is located;

[0075] Step S204, collecting target image data by the binocular camera after the baseline distance adjustment.

[0076] In the embodiment, the embodiment considers that the binocular camera with fixed baseline will cause the limited range of sight adaptation in the complex motion scene, and thus cannot accurately identify the target distance. Based on this, the embodiment of the application constructs a binocular camera with a baseline distance, and drives the double camera by a servo motor to dynamically adjust the baseline distance of the camera to adapt to different distance target tracking ships. Specifically, for the initial image data collected by the binocular camera, the initial image data collected by the binocular camera is subjected to monocular semantic segmentation processing for target tracking ships to obtain the target contour of the binocular camera. The monocular semantic segmentation processing refers to region segmentation of the image data collected by the monocular camera, and determining the classification label (such as sky, ship, sea surface, etc.) of each segmentation region, and determining the region classified as a ship as the target contour. After the target contour is preliminarily identified, the stereo matching processing is performed on the target contour of the binocular camera to obtain the initial value of the disparity map, and the target distance estimate value is determined according to the initial value of the disparity map. When the target is in the medium and far distance interval, the control system moves the binocular camera in opposite directions through the slide rail to expand the binocular baseline length, and uses the long baseline to improve the depth estimation resolution; when the target approaches to the near distance interval, the control system moves the binocular camera in the same direction through the slide rail to contract the binocular baseline length, and avoids the field of view mismatch. The target image data is collected by the binocular camera after the baseline adjustment, and the accurate ranging is realized through the target image data subsequently. It can be understood that the far distance interval and the near distance interval can be set according to actual needs, and the embodiment of the application does not make specific limitation.

[0077] In step S102 of some embodiments, the target detection model can be a convolutional neural network model, for example, a YOLO series model. The trained target detection model is used to identify ships and output the recognition detection frame. Therefore, the target image data of the binocular camera can be identified by the target tracking ship by using the target detection model, and the target region in the target image data is determined;

[0078] According to some embodiments of the application, step S102 can include but is not limited to the following steps:

[0079] Step S301, inputting the target image data into the target detection model based on the Yolov5s network to obtain the detection frame output by the target detection model, and the detection frame is used to determine the target region in the target image data.

[0080] In this embodiment, the target detection model can adopt a Yolov5s network, which includes a backbone network, a connection network (Neck) and a detection network (Head) connected in turn. The backbone network is used to extract multi-scale and multi-level feature maps from the input image; the connection network is used to connect the Backbone and the Head, responsible for receiving the multi-scale feature maps extracted by the backbone network, and effectively aggregating the features of different levels and different scales through feature fusion techniques (such as upsampling, downsampling, connection / splicing); the detection network is used to receive the multi-scale feature maps optimized by the Neck, and directly predict the bounding box position (x, y, w, h), confidence (probability of containing objects) and class probability of the target object at each scale. In order to improve the adaptability of Yolov5s to complex sea surface environment and improve the accuracy of target recognition, this embodiment improves the YoloV5s network as the basic framework, embeds a lightweight ECA-Net channel interaction enhancement ship feature in the backbone network, the lightweight ECA-Net channel refers to an efficient channel attention module (Efficient Channel Attention Module), which can dynamically learn the importance weight of each feature channel, adaptively enhance useful channel features and suppress useless channel features, thereby improving network performance with almost no increase in computational overhead. The spatial-channel dual-channel attention mechanism (SCMA) is introduced in the PANet (path aggregation network) part of the Neck, and the spatial mask is used to focus on the ship contour area. According to the ship data features, the K-means++ optimization anchor box size distribution is adopted, and the multi-scale dynamic anchor box mechanism is used to realize the adaptive matching of near-range small lifeboats and far-range large cargo ships. At the same time, a dynamic Focal Loss loss function is used in the training process to alleviate the class imbalance problem, and an edge perception loss function is used to strengthen the ship waterline positioning accuracy.

[0081] In step S103 of some embodiments, after determining the target regions in the two target image data of the binocular camera, a semi-global matching algorithm is used to estimate the disparity of the target regions in the two target image data to obtain sub-pixel disparity. The semi-global matching algorithm (Semi-Global Matching, SGM) is to approximate the global energy function optimization by multi-path cost aggregation, which significantly improves the quality of the disparity map while ensuring real-time. Sub-pixel disparity refers to the horizontal position difference of a feature point in the smallest unit (pixel) of the imaging surface.

[0082] According to some embodiments of the present application, step S103 can include but is not limited to the following steps:

[0083] Step S401, respectively determine the disparity estimation value of the target feature point and the adjacent feature point in the target region in the image data collected by the binocular camera;

[0084] In step S402, the matching cost of the target feature point and the adjacent feature point is determined according to the disparity estimation value of the target feature point and the adjacent feature point by matching the cost function.

[0085] In step S403, the disparity error is determined according to the matching cost of the target feature point and the adjacent feature point.

[0086] In step S404, the disparity estimation value of the target feature point is optimized according to the disparity error, and the sub-pixel disparity of the target feature point is obtained.

[0087] In the embodiment, considering that the maritime scene is mostly a wide space, and the ship target and the ship are both in a moving state, which causes the camera view distance to be unstable. Based on this, the improved semi-global matching algorithm (SGM) can be used for disparity estimation in the embodiment of the application, a dense disparity map is constructed, and then sub-pixel level dynamic measurement of the distance between the two ships is realized. The optimization process of the sub-pixel disparity estimation is as follows:

[0088]

[0089] d sub =d+δ; (3)

[0090] In formula (2) and formula (3), the disparity error is further reduced by calculating (d-1, d, d+1) three times, and then the sub-pixel disparity d sub is obtained. In the above formula, d is the disparity estimation value, that is, the horizontal pixel difference of the target feature point in the images collected by the left and right cameras, which is represented as d=u l -u r ; d-1 and d+1 respectively represent the adjacent feature point disparity estimation values smaller or larger than the current disparity by one pixel; δ is the sub-pixel offset (i.e., the disparity error), and C(x) is the matching cost function. The matching cost function needs to be a quadratic function in the interval [d-1, d+1], which is represented as follows:

[0091] C(x)=a(x-d)2+b(x-d)+c; (4)

[0092] Wherein, a, b, and c are undetermined coefficients, and x is the disparity variable.

[0093] In step S104 of some embodiments, the sub-pixel disparity of each pixel point (i.e., feature point) of the target region forms a disparity map, the corresponding relative distance is determined according to the sub-pixel disparity, and thus a depth map is obtained. The specific process of calculating the relative distance according to the sub-pixel disparity is as follows:

[0094] In the basic binocular model On this basis, the sub-pixel parallax depth is converted according to the maritime scene, and then the relative distance of the target ship is calculated. As shown in equation (5),

[0095]

[0096] In equation (5), f is the focal length of the camera, B is the baseline distance, k1*sinθ is the baseline inclination correction caused by the roll of the ship, θ is the inclination angle of the binocular camera baseline, which can be regarded as the roll angle of the ship, k2*v rel is the relative speed v rel of the target ship caused by the parallax lag effect, wherein k1 and k2 are correction parameters that need to be optimized in combination with theoretical models and experiments, and Z is the relative distance between the two ships.

[0097] In step S105 of some embodiments, the motion state of the target tracking ship is predicted according to the relative distance, and the first situation information of the target tracking ship is obtained, wherein the first situation information includes but is not limited to the relative position, the relative speed, the target ship heading, and the relative acceleration at the next moment, etc. In an example, the relative speed, the relative acceleration, and the target ship heading can be calculated by using the corresponding calculation formula in combination with the relative distance measured at different times, and the motion state parameters at the next moment can be predicted by using a time series prediction algorithm such as a convolutional neural network, to obtain the first situation information.

[0098] According to some embodiments of the present application, step S105 can include but is not limited to the following steps:

[0099] Step S501, determining the position of the target tracking ship in the dynamic coordinate system according to the relative distance and the relative bearing, wherein the dynamic coordinate system takes the center of the binocular camera as the origin;

[0100] Step S502, determining the motion state vector of the target tracking ship at different times according to the dynamic coordinate system;

[0101] Step S503, using a Kalman particle filter hybrid algorithm to estimate the motion state of the motion state vector at the next moment, to obtain the first situation information of the target tracking ship.

[0102] According to some embodiments of the present application, step S105 can further include but is not limited to the following steps:

[0103] Step S601, in the case of observing the target ship heading from the ship automatic identification system, correcting the relative speed parameter in the motion state vector according to the target ship heading.

[0104] In the embodiment, after obtaining the relative distance between the two ships, the heading of the target ship and the relative speed of the two ships are also determined to have the basic conditions for following the target ship. The Kalman particle filtering hybrid algorithm is used to construct a multi-modal ship motion estimator to achieve high-precision calculation of the heading and relative speed of the target ship.

[0105] Reference Figure 3 A dynamic coordinate system is established with the binocular center, wherein the x-axis points to the bow of the ship. The relative position of the target ship and the ship is calculated by formula (6) as follows:

[0106]

[0107] wherein R(θ) is the roll rotation matrix, and φ is the azimuth angle of the x-axis pointing to the target ship, which can be calculated by formula (7).

[0108]

[0109] In formula (7), c x is the distance from the binocular center to the camera.

[0110] After the dynamic coordinate system is established, the Kalman and particle filtering are used to predict the first situation information of the target tracking ship, and the prediction equation is shown in formula (8) and formula (9).

[0111] x k =F·x k-1 +B·u k-1 +w k ; (8)

[0112] z k =h(x k )+v k =[Z,Δψ AIS ] T +v k ; (9)

[0113]

[0114] In formula (8), x k and x k-1 represent the motion state vectors of the target tracking ship at the current time and the previous time, which can be represented as [X, Y, v rel , ψ, a rel ] T , B is a control input matrix, u k-1 is a control input vector at the previous time, w k is process noise; X and Y are relative positions; ψ is the heading of the target ship; a relis the relative acceleration; F is the transition matrix, which can be expressed by equation (10), where T is the sampling period. In equation (9), z k is the observation vector, and the observation range is If the AIS information of the target tracking ship can be observed, Δψ AIS (target ship heading error), according to Δψ AIS The heading of the target ship in the motion state vector can be corrected, and the predicted relative velocity v k is the observation noise.

[0115] In this embodiment, the speed of the target tracking ship relative to the ground coordinate system, i.e., the actual speed of the target tracking ship is expressed as follows:

[0116]

[0117] The heading of the target ship ψ is extracted from the motion state vector of the target tracking ship, and the actual function of the target tracking ship needs to be decoupled by equation (11), where v ego is the speed of the own ship, v rel is the relative speed of the two ships.

[0118] In step S106 of some embodiments, the following control instruction of the own ship is determined according to the first situation information, so as to follow the target tracking ship. Specifically, the first situation information is taken as the expected state of the own ship, and the following control instruction of the own ship is calculated by using a control algorithm such as PID control, model predictive control (MPC), and neural network algorithm of Markov decision, so as to realize following the target tracking ship.

[0119] According to some embodiments of the present application, step S106 can include but is not limited to the following steps:

[0120] Step S701, determining the ship state vector according to the first situation information, the second situation information of the own ship, and the control amount of the power component of the own ship at the last time;

[0121] Step S702, using a Markov decision model to make control increment decision of the power component of the own ship according to the ship state vector, to obtain the following control instruction of the own ship.

[0122] In this embodiment, the ship is controlled by using a reinforcement learning algorithm on the basis of determining the first situation information. Specifically, the control problem of the ship is converted into a Markov decision process (MDP), and a high-altitude ship state vector is constructed in combination with binocular pure vision camera and IMU inertial navigation data, as follows:

[0123] s t = [X, Y, ψ rel , u, v, r, δ t-1 , Nt-1 F env ]; (12)

[0124] a t =[Δδ,ΔN]; (13)

[0125] In formula (12), ψ rel is the relative heading of the target tracking ship, which can be obtained by ψ rel = ψ - ψ ego , wherein ψ ego is the heading of the ship; u, v, r are the longitudinal, lateral and turning bow speeds of the ship; δ t-1 is the rudder angle at the last time, N t-1 is the propeller speed at the last time, and the rudder angle and the propeller speed are both referred to as power component control quantities; F env is the environmental disturbance quantity. The ship movement instruction is indicated by formula (13), and Δδ, ΔN respectively represent the increments of the rudder angle and the propeller speed. The design reward function is shown in formula (14):

[0126] r t = w1·e -α·||Δx,Δy|| + w2·(-N 2 ) + w3·(-Δδ 2 ) + w4·CollisionPenalty; (14)

[0127] In formula (14), α is an exponential decay term to encourage the reduction of the position deviation; -N 2 suppresses excessive propulsion; -Δδ 2 reduces the frequent action of the rudder; CollisionPenalty is the collision penalty; and w1, w2, w3, w4 are weight assignments.

[0128] In addition, before the above-mentioned Markov decision model is cited, the model needs to be trained, which can be carried out through a simulation environment such as Unity3D / Unreal Engine, and a four-degree-of-freedom MMG model is used as the ship movement model for mechanism simulation. After a large amount of training is completed, further actual deployment and transfer learning are carried out, so that the ship-mounted server can control and predict the rudder angle and the propeller speed of the ship based on binocular pure vision data and IMU inertial navigation data, and the prediction results are sent to the ship propulsion controller and the rudder angle control module to complete the ship tracking.

[0129] According to some embodiments of the present application, the baseline distance adjustable binocular vision architecture and the sub-pixel level disparity-motion joint estimation model are cooperated to greatly reduce the cost of target ship detection while significantly improving the comprehensive performance of ship autonomous tracking control.

[0130] Specifically, when detecting a target ship, the binocular pure vision camera transmits the image collected by the binocular camera to the shipborne server. By adjusting the baseline distance cx of the binocular camera, the ability to dynamically adjust the predicted ship distance at the physical level while identifying and tracking the target ship is realized, which greatly improves the distance prediction range compared with the traditional triangle function-based ship distance prediction method. In addition, by using the improved YOLOv5s network embedded with the space-channel dual-channel attention mechanism (SCMA) and cooperating with the K-means++ dynamic anchor optimization, the problem of false detection and missed detection caused by target size difference can be effectively overcome.

[0131] Under the dynamic baseline distance adjustment mechanism, the shipborne server optimizes the target ship image at the sub-pixel level disparity by improving the SGM algorithm, and further reduces the depth measurement error by using a cubic cost quadratic function interpolation. At the same time, considering the errors caused by the baseline tilt and disparity lag due to the relative motion of the target ship and the ship body sway, the algorithm can still ensure the accuracy of the distance measurement between the two ships in complex sea conditions.

[0132] The shipborne coordinate system and state vector of the ship are complementary, and the Kalman particle filtering hybrid algorithm is further used to predict the motion of the target ship state (heading and speed). In addition, in the absence of AIS type, the target ship state can be automatically evaluated, and in the case of observing the target ship AIS signal, the target ship state can be corrected using the AIS signal.

[0133] According to the state information of the target ship, a Markov decision model is constructed based on the reinforcement learning control strategy, multi-dimensional state vector fusion visual perception and inertial navigation data are used to dynamically adjust the rudder angle and propeller speed increment of the ship, so as to realize the longitudinal following control of the ship.

[0134] The embodiments of the present application greatly reduce the data fusion difficulty through algorithm and hardware cooperative optimization, significantly shorten the system response delay while theoretically guaranteeing the prediction accuracy. In addition, the dependence on expensive sensors such as laser radar and millimeter wave radar is reduced, which greatly reduces the hardware cost, and provides an all-weather autonomous following navigation solution for inland ship formation with high precision, strong robustness and economy.

[0135] The embodiments of the present application also propose a ship following system based on binocular vision, comprising:

[0136] The first module is used for collecting target image data by a binocular camera;

[0137] The second module is used for identifying and tracking the target ship in the target image data of the binocular camera by using a target detection model, and determining the target region in the target image data.

[0138] The third module is configured to perform disparity estimation on the target regions in the two target image data by using a semi-global matching algorithm to obtain sub-pixel disparity;

[0139] The fourth module is configured to determine the relative distance between the target tracking ship and the own ship according to the sub-pixel disparity;

[0140] The fifth module is configured to perform motion state prediction on the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship;

[0141] The sixth module is configured to determine a following control instruction of the own ship according to the first situation information to follow the target tracking ship.

[0142] It can be understood that the content in the above ship following method embodiment based on binocular vision is applicable to the present system embodiment, the present system embodiment specifically implements the same function as the above ship following method embodiment based on binocular vision, and achieves the same beneficial effects as the above ship following method embodiment based on binocular vision.

[0143] The present application also provides an electronic device, which comprises a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above ship following method based on binocular vision. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0144] Please refer to Figure 4 , Figure 4 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0145] The processor 901 can be implemented in a general-purpose CPU (Central Processing Unit), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the present application;

[0146] The memory 902 can be implemented in the form of a Read Only Memory (ROM), a static storage device, a dynamic storage device, or a Random Access Memory (RAM), etc. The memory 902 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 902 and are called and executed by the processor 901 to implement the ship following method based on binocular vision of the embodiments of the present application;

[0147] The input / output interface 903 is configured to realize information input and output.

[0148] The communication interface 904 is configured to realize the communication interaction between the device and other devices, and the communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0149] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0150] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0151] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium, and is configured to store computer readable information. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the ship following method based on binocular vision.

[0152] The memory is a non-transitory computer readable storage medium, and can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0153] The embodiments described in the specification are for more clearly and specifically illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0154] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0155] The system embodiments described above are merely illustrative, and the modules described as separate components can or can not be physically separated, that is, can be located in one place, or can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0156] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0157] The terms "include" and "have" and any variations thereof in the specification are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or modules does not have to be limited to those steps or modules clearly listed, but can include other steps or modules not clearly listed or inherent to these processes, methods, products or devices.

[0158] In several embodiments provided in the present application, it should be understood that the disclosed system and method can be implemented in other ways. For example, the system embodiments described above are merely illustrative, for example, the division of the above modules is only a logical functional division, and actual implementation can have another division manner, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, indirect coupling or communication connection between the system or modules, which can be electrical, mechanical or other forms.

[0159] The modules described as separate components above can or can not be physically separated, and the components shown as modules can or can not be physical modules, that is, can be located in one place, or can be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0160] In addition, each function module in each embodiment of the present application can be integrated in one processing module, or each module can be physically present alone, or two or more modules can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module.

[0161] When the integrated module is realized in the form of a software function module and sold or used as an independent product, the integrated module can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0162] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A ship following method based on binocular vision, characterized in that: The following steps are involved: Collect target image data through binocular cameras; Using the target detection model, the target image data of the binocular camera are respectively subjected to target tracking and ship identification to determine the target area in the target image data; Using a semi-global matching algorithm to perform disparity estimation on the target area in the two target image data to obtain sub-pixel disparity; determining a relative distance between the target tracking vessel and the own vessel based on the sub-pixel parallax; Predicting the motion state of the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship; A following control instruction of the own ship is determined based on the first situation information to follow the target tracking ship.

2. The ship following method based on binocular vision according to claim 1, characterized in that: The method of collecting target image data by using a binocular camera includes the following steps: Performing monocular semantic segmentation processing on the initial image data collected by the binocular camera for the target tracking ship to obtain the target outline of the binocular camera; Performing stereo matching processing on the target outline of the binocular camera to obtain an initial value of a disparity map, and determining an estimated value of the target distance based on the initial value of the disparity map; Adjusting the baseline distance of the binocular camera according to the range of the target distance estimation value; The binocular camera collects target image data according to the adjusted baseline distance.

3. The ship following method based on binocular vision according to claim 1, characterized in that: The method of using the target detection model to perform target tracking and ship identification on the target image data of the binocular camera to determine the target area in the target image data includes the following steps: Inputting the target image data into a target detection model based on the Yolov5s network to obtain a detection frame output by the target detection model, wherein the detection frame is used to determine the target area in the target image data; Among them, the target detection model includes a backbone network, a connection network and a detection network connected in sequence; the backbone network includes a lightweight channel attention module, and the lightweight channel attention module is used to learn the importance weight of each feature channel; the connection network adopts a spatial channel dual-path attention mechanism for feature fusion.

4. The ship following method based on binocular vision according to claim 1, characterized in that: The method of using a semi-global matching algorithm to perform disparity estimation on the target area in the two target image data to obtain sub-pixel disparity includes the following steps: Determining respectively the disparity estimation values ​​of the target feature points and adjacent feature points in the target area in the image data collected by the binocular camera; Determining the matching costs of the target feature point and the adjacent feature point respectively according to the disparity estimation values ​​of the target feature point and the adjacent feature point by using a matching cost function; Determining a disparity error based on a matching cost between the target feature point and the adjacent feature points; The disparity estimation value of the target feature point is optimized according to the disparity error to obtain the sub-pixel disparity of the target feature point.

5. The ship following method based on binocular vision according to claim 1, characterized in that: The step of predicting the motion state of the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship comprises the following steps: Determining the position of the target tracking vessel in a dynamic coordinate system according to the relative distance and relative orientation, wherein the dynamic coordinate system takes the center of the binocular camera as an origin; Determining the motion state vectors of the target tracking vessel at different times according to the dynamic coordinate system; A Kalman particle filter hybrid algorithm is used to estimate the motion state of the motion state vector at the next moment, and the first situation information of the target tracking ship is obtained.

6. The ship following method based on binocular vision according to claim 5, characterized in that: The step of predicting the motion state of the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship further includes the following steps: When the target ship heading is observed from the automatic identification system of a ship, the relative speed parameter in the motion state vector is corrected according to the target ship heading.

7. The ship following method based on binocular vision according to claim 1, characterized in that: Determining the following control instruction of the own ship according to the first situation information comprises the following steps: Determine a ship state vector based on the first situation information, the second situation information of the own ship, and the control amount of the power component of the own ship at the last moment; A Markov decision model is adopted to make control increment decisions of the power components of the ship according to the ship state vector, and a following control instruction of the ship is obtained.

8. A ship following system based on binocular vision, characterized in that: include: The first module is used to collect target image data through a binocular camera; The second module is used to use the target detection model to perform target tracking and ship recognition on the target image data of the binocular camera, and determine the target area in the target image data; The third module is used to use a semi-global matching algorithm to perform disparity estimation on the target area in the two target image data to obtain sub-pixel disparity; A fourth module is configured to determine a relative distance between the target tracking vessel and the own vessel based on the sub-pixel parallax; A fifth module predicts a motion state of the target tracking ship according to the relative distance to obtain first situation information of the target tracking ship; The sixth module determines the following control instruction of the own ship according to the first situation information to follow the target tracking ship.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. A storage medium, which is a computer-readable storage medium and is used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of any one of claims 1 to 7.

Images