Barge queue identification method and system, electronic equipment and medium

Abstract

The invention discloses a barge queue identification method and system, electronic equipment and a medium, and relates to the technical field of image identification. The method comprises the following steps: inputting a picture of a tail into a tail ship target detection model, and calculating an offset angle difference between adjacent ships in the tail; determining whether the identification result of the target barge queue is normal or not according to the offset angle difference of the tail and a tail threshold value, if not, inputting the head picture into a head ship target detection model, and calculating the offset angle difference between adjacent ships in the head; according to the offset angle difference of the head and a head threshold value, determining whether the identification result of the target barge queue is scattered or not, if not, inputting the picture of the middle part into a middle part ship target detection model, and calculating the offset angle difference between adjacent ships in the middle part; and determining an identification result according to the middle offset angle difference and a middle threshold. The accuracy of the recognition result of the barge queue can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly to a method, system, electronic device and medium for tugboat fleet recognition. Background Art

[0002] In the shipping industry, due to human operation errors or external environmental factors, the formation of a tugboat fleet may be affected, causing the ships to lose their stable positions and distances. This scattered formation increases the risk of collision between ships, which may lead to damage to the hull structure and equipment, affecting the normal navigation of the tugboat fleet. In severe cases, it may cause the hull to capsize, the cargo to spill, and even result in casualties. To adjust the fleet when it becomes scattered, it is crucial to detect anomalies in the ship formation in a timely manner. Previously, manual supervision was generally used, where crew members observed the formation of the ship fleet at regular intervals to issue early warnings. However, this method has significant limitations. The crew members need to have certain experience, and there may be negligence and misjudgment in manual operation. Based on this situation, installing cameras on tugboats and using machine vision for ship fleet recognition has great advantages. Among them, how to identify the barges at the far end of the fleet and judge the overall formation is the key to tugboat fleet recognition.

[0003] Currently, in the field of image recognition, deep learning has excellent adaptability and accuracy. However, when using a single camera to capture images for recognition on a tugboat fleet, it is difficult to accurately identify the target ships at a relatively long distance in the fleet, resulting in the inability to comprehensively evaluate the entire tugboat fleet and a relatively low accuracy of the tugboat fleet recognition result. Summary of the Invention

[0004] The purpose of the present invention is to provide a method, system, electronic device and medium for tugboat fleet recognition, which can comprehensively evaluate the entire tugboat fleet and improve the accuracy of the tugboat fleet recognition result.

[0005] To achieve the above purpose, the present invention provides the following solutions:

[0006] A method for tugboat fleet recognition includes:

[0007] Obtaining pictures of the head, middle and tail of a target tugboat fleet at the current moment;

[0008] Inputting the picture of the tail of the target tugboat fleet obtained at the current moment into a tail ship target detection model to obtain the coordinates of the center points of the target boxes of all ships at the tail of the target tugboat fleet;

[0009] Calculate the offset angle difference between adjacent ships in the tail of the target tow barge queue according to the center point coordinates of the target boxes of all ships in the tail of the target tow barge queue;

[0010] Determine whether the recognition result of the target tow barge queue is normal according to the offset angle difference between adjacent ships in the tail of the target tow barge queue and the tail threshold, and obtain the first judgment result;

[0011] If the first judgment result is negative, input the picture of the head of the target tow barge queue obtained at the current moment into the head ship target detection model to obtain the center point coordinates of the target boxes of all ships in the head of the target tow barge queue;

[0012] Calculate the offset angle difference between adjacent ships in the head of the target tow barge queue according to the center point coordinates of the target boxes of all ships in the head of the target tow barge queue;

[0013] Determine whether the recognition result of the target tow barge queue is scattered according to the offset angle difference between adjacent ships in the head of the target tow barge queue and the head threshold, and obtain the second judgment result;

[0014] If the second judgment result is negative, input the picture of the middle part of the target tow barge queue obtained at the current moment into the middle ship target detection model to obtain the center point coordinates of the target boxes of all ships in the middle of the target tow barge queue;

[0015] Calculate the offset angle difference between adjacent ships in the middle of the target tow barge queue according to the center point coordinates of the target boxes of all ships in the middle of the target tow barge queue;

[0016] Determine the recognition result of the target tow barge queue according to the offset angle difference between adjacent ships in the middle of the target tow barge queue and the middle threshold; the recognition result is normal or scattered.

[0017] Optionally, the determination process of the tail ship target detection model, the head ship target detection model or the middle ship target detection model is as follows:

[0018] Obtain a picture set; the picture set includes multiple pictures of the head, middle or tail of a training tow barge queue with marked ship detection boxes;

[0019] Divide the picture set into a training set, a validation set and a test set according to a set ratio;

[0020] Preset anchor boxes of multiple sizes, and input each picture in the training set into the ship target detection model to obtain the feature probabilities of each anchor box corresponding to each picture in the training set; the ship target detection model includes a convolutional neural network and a fully connected module connected in sequence, and the fully connected module includes multiple fully connected layers connected in sequence;

[0021] Obtain the optimal predicted anchor box corresponding to each picture in the training set according to the feature probabilities of each anchor box corresponding to each picture in the training set;

[0022] Obtain the loss function according to the optimal predicted anchor box corresponding to each picture in the training set and the ship detection box marked in the training set;

[0023] Train the ship target detection model with the goal of minimizing the loss function to obtain the trained ship target detection model;

[0024] Use the test set and the validation set to test and validate the trained ship target detection model to obtain the detection model; when the picture set includes multiple training barge queue head pictures marked with ship detection boxes, the detection model is the head ship target detection model; when the picture set includes multiple training barge queue middle pictures marked with ship detection boxes, the detection model is the middle ship target detection model; when the picture set includes multiple training barge queue tail pictures marked with ship detection boxes, the detection model is the tail ship target detection model.

[0025] Optionally, the determination process of the tail threshold, the head threshold or the middle threshold is as follows:

[0026] According to the optimal predicted anchor box corresponding to the target picture in the training set, calculate the offset angle of each ship in the target picture; the target picture is any picture of the head, middle or tail of a training barge queue with a normal barge queue recognition result;

[0027] Calculate the offset angle difference between adjacent ships in the target picture according to the offset angles of each ship in the target picture;

[0028] Determine the maximum offset angle difference as the threshold; when the target picture is any training barge queue head picture with a normal barge queue recognition result, the threshold is the head threshold; when the target picture is any training barge queue middle picture with a normal barge queue recognition result, the threshold is the middle threshold; when the target picture is any training barge queue tail picture with a normal barge queue recognition result, the threshold is the tail threshold.

[0029] Optionally, calculating the offset angle difference between adjacent ships is specifically:

[0030] According to the formula Calculate the offset angle θ of the i-th ship (i-1,i) ;

[0031] According to the formula Calculate the offset angle θ of the (i + 1)-th ship(i,i+1) ;

[0032] According to the formula d i = θ (i,i+1) - θ (i-1,i) Calculate the offset angle difference d between the i-th ship and the (i + 1)-th ship i , where y i represents the ordinate of the i-th ship, y i-1 represents the ordinate of the (i - 1)-th ship, y i+1 represents the ordinate of the (i + 1)-th ship, x i represents the abscissa of the i-th ship, x i-1 represents the abscissa of the (i - 1)-th ship, x i+1 represents the abscissa of the (i + 1)-th ship.

[0033] Optionally, determine whether the recognition result of the target tugboat queue is normal according to the offset angle differences between adjacent ships in the tail of the target tugboat queue and the tail threshold, specifically including:

[0034] If the offset angle differences between adjacent ships in the tail of the target tugboat queue are all less than the tail threshold, the recognition result of the target tugboat queue is normal.

[0035] Optionally, determine whether the recognition result of the target tugboat queue is scattered according to the offset angle differences between adjacent ships in the head of the target tugboat queue and the head threshold, specifically including:

[0036] Judge whether at least a first quantity of the offset angle differences between adjacent ships in the head of the target tugboat queue is less than the head threshold, and obtain a third judgment result; the first quantity is the first set percentage of the total quantity of the offset angle differences corresponding to the head of the target tugboat queue;

[0037] If the third judgment result is negative, the recognition result of the target tugboat queue is scattered.

[0038] Optionally, determine the recognition result of the target tugboat queue according to the offset angle differences between adjacent ships in the middle of the target tugboat queue and the middle threshold, specifically including:

[0039] Judge whether at least a second quantity of the offset angle differences between adjacent ships in the middle of the target tugboat queue is less than the middle threshold, and obtain a fourth judgment result; the second quantity is the first set percentage of the total quantity of the offset angle differences corresponding to the middle of the target tugboat queue;

[0040] If the fourth judgment result is negative, the recognition result of the target tugboat queue is scattered;

[0041] If the fourth judgment result is yes, then determine that the ratios of the offset angle differences greater than the middle threshold among the offset angle differences between adjacent ships in the middle of the target tug-barge queue to the middle threshold are all less than the second set percentage of the middle threshold, and obtain a fifth judgment result;

[0042] If the fifth judgment result is yes, then determine that the recognition result of the target tug-barge queue is normal;

[0043] If the fifth judgment result is no, then determine that the recognition result of the target tug-barge queue is scattered.

[0044] A tug-barge queue recognition system, comprising:

[0045] An acquisition module, configured to acquire pictures of the head, middle, and tail of a target tug-barge queue at the current moment;

[0046] A tail ship processing module, configured to input the picture of the tail of the target tug-barge queue acquired at the current moment into a tail ship target detection model to obtain the target box center point coordinates of all ships at the tail of the target tug-barge queue;

[0047] A tail ship offset angle difference calculation module, configured to calculate the offset angle differences between adjacent ships in the tail of the target tug-barge queue according to the target box center point coordinates of all ships at the tail of the target tug-barge queue;

[0048] A first judgment module, configured to determine whether the recognition result of the target tug-barge queue is normal according to the offset angle differences between adjacent ships in the tail of the target tug-barge queue and a tail threshold, and obtain a first judgment result;

[0049] A head ship processing module, configured to, if the first judgment result is no, input the picture of the head of the target tug-barge queue acquired at the current moment into a head ship target detection model to obtain the target box center point coordinates of all ships at the head of the target tug-barge queue;

[0050] A head ship offset angle difference calculation module, configured to calculate the offset angle differences between adjacent ships in the head of the target tug-barge queue according to the target box center point coordinates of all ships at the head of the target tug-barge queue;

[0051] A second judgment module, configured to determine whether the recognition result of the target tug-barge queue is scattered according to the offset angle differences between adjacent ships in the head of the target tug-barge queue and a head threshold, and obtain a second judgment result;

[0052] The middle ship processing module is configured to, if the second judgment result is negative, input the picture in the middle of the target tow barge queue obtained at the current moment into the middle ship target detection model to obtain the coordinates of the center points of the target frames of all ships in the middle of the target tow barge queue;

[0053] The middle ship offset angle difference calculation module is configured to calculate the offset angle differences between adjacent ships in the middle of the target tow barge queue according to the coordinates of the center points of the target frames of all ships in the middle of the target tow barge queue;

[0054] The final judgment module is configured to determine the recognition result of the target tow barge queue according to the offset angle differences between adjacent ships in the middle of the target tow barge queue and the middle threshold; the recognition result is normal or scattered.

[0055] An electronic device includes:

[0056] A memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the tow barge queue recognition method according to the above.

[0057] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the tow barge queue recognition method according to the above is implemented.

[0058] According to the specific embodiments provided by the present invention, the following technical effects are disclosed by the present invention:

[0059] In the process of recognizing the tow barge queue, the present invention uses the images of the head, tail and middle of the tow barge queue, can recognize all ships in the tow barge queue, can comprehensively evaluate the entire tow barge queue, and improves the accuracy of the recognition result of the tow barge queue. Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the following described drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0061] Figure 1 It is a flowchart of the tow barge queue recognition method provided by the embodiment of the present invention;

[0062] Figure 2 It is a basic flowchart of the tow barge queue recognition method provided by the embodiment of the present invention;

[0063] Figure 3This is the framework flowchart of the tug-barge fleet identification method provided by the embodiments of the present invention. Detailed implementation manners

[0064] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0065] To make the above objects, features, and advantages of the present invention more obvious and understandable, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific implementation manners.

[0066] In order to solve the problem of identifying whether the fleet queue is scattered during the navigation of the tug-barge fleet, the present invention proposes a tug-barge fleet identification method based on multi-camera cooperation. Through the image target detection algorithm, the ships in the pictures taken by 3 cameras are identified, and the slope judgment method is used to judge whether the ships are in a queue, and then the results identified by the 3 cameras are fused to comprehensively judge the queue situation of the tug-barge fleet. The present invention can also be extended to other fields, which are used in scenarios where computer vision technology is required and it is difficult for a single camera to capture all targets, and helps to realize the automation of queue identification in other fields. As Figure 1 shown, the tug-barge fleet identification method includes:

[0067] Step 101: Obtain pictures of the head, middle, and tail of the target tug-barge fleet at the current moment.

[0068] Step 102: Input the picture of the tail of the target tug-barge fleet obtained at the current moment into the tail ship target detection model to obtain the center point coordinates of the target boxes of all ships at the tail of the target tug-barge fleet.

[0069] Step 103: Calculate the offset angle difference between adjacent ships at the tail of the target tug-barge fleet according to the center point coordinates of the target boxes of all ships at the tail of the target tug-barge fleet.

[0070] Step 104: Determine whether the identification result of the target tug-barge fleet is normal according to the offset angle difference between adjacent ships at the tail of the target tug-barge fleet and the tail threshold, and obtain the first judgment result.

[0071] Step 105: If the first judgment result is no, input the picture of the head of the target tug-barge fleet obtained at the current moment into the head ship target detection model to obtain the center point coordinates of the target boxes of all ships at the head of the target tug-barge fleet.

[0072] Step 106: Calculate the offset angle differences between adjacent vessels in the head of the target tow barge queue based on the center point coordinates of the target boxes of all vessels in the head of the target tow barge queue.

[0073] Step 107: Determine whether the recognition result of the target tow barge queue is scattered based on the offset angle differences between adjacent vessels in the head of the target tow barge queue and the head threshold, and obtain a second judgment result.

[0074] Step 108: If the second judgment result is negative, input the picture of the middle part of the target tow barge queue obtained at the current moment into the middle vessel target detection model to obtain the center point coordinates of the target boxes of all vessels in the middle part of the target tow barge queue.

[0075] Step 109: Calculate the offset angle differences between adjacent vessels in the middle part of the target tow barge queue based on the center point coordinates of the target boxes of all vessels in the middle part of the target tow barge queue.

[0076] Step 110: Determine the recognition result of the target tow barge queue based on the offset angle differences between adjacent vessels in the middle part of the target tow barge queue and the middle threshold; the recognition result is normal or scattered.

[0077] In practical applications, the determination process of the tail vessel target detection model, the head vessel target detection model or the middle vessel target detection model is as follows:

[0078] Obtain a picture set; the picture set includes multiple pictures of the head, middle or tail of the training tow barge queue with marked ship detection frames.

[0079] Divide the picture set into a training set, a validation set and a test set according to a set ratio.

[0080] Preset anchor boxes of multiple sizes, and input each picture in the training set into the vessel target detection model to obtain the feature probabilities of each anchor box corresponding to each picture in the training set; the vessel target detection model includes a convolutional neural network and a fully connected module connected in sequence, and the fully connected module includes multiple fully connected layers connected in sequence.

[0081] Obtain the best predicted anchor box corresponding to each picture in the training set according to the feature probabilities of each anchor box corresponding to each picture in the training set. The one with the largest feature probability is the best predicted anchor box.

[0082] Obtain a loss function according to the best predicted anchor box corresponding to each picture in the training set and the ship detection frame marked in the training set.

[0083] Train the vessel target detection model with the goal of minimizing the loss function to obtain the trained vessel target detection model.

[0084] The trained ship target detection model is tested and verified using a test set and a validation set to obtain a detection model; when the picture set includes multiple training barge queue head pictures with ship detection frames marked, the detection model is a head ship target detection model; when the picture set includes multiple training barge queue middle pictures with ship detection frames marked, the detection model is a middle ship target detection model; when the picture set includes multiple training barge queue tail pictures with ship detection frames marked, the detection model is a tail ship target detection model.

[0085] In practical applications, the determination process of the tail threshold, the head threshold, or the middle threshold is as follows:

[0086] According to the best predicted anchor box corresponding to the target picture in the training set, calculate the offset angle of each ship in the target picture; the target picture is any picture of the head, middle, or tail of a training barge queue with a normal recognition result.

[0087] Calculate the offset angle difference between adjacent ships in the target picture based on the offset angles of each ship in the target picture.

[0088] Determine the maximum offset angle difference as the threshold; when the target picture is any picture of the head of a training barge queue with a normal recognition result, the threshold is the head threshold; when the target picture is any picture of the middle of a training barge queue with a normal recognition result, the threshold is the middle threshold; when the target picture is any picture of the tail of a training barge queue with a normal recognition result, the threshold is the tail threshold.

[0089] In practical applications, calculating the offset angle difference between adjacent ships is specifically as follows:

[0090] According to the formula Calculate the offset angle θ of the i-th ship (i-1,i) ;

[0091] According to the formula Calculate the offset angle θ of the (i + 1)-th ship (i,i+1) ;

[0092] According to the formula d i = θ (i,i+1) - θ (i-1,i) Calculate the offset angle difference d between the i-th ship and the (i + 1)-th ship i , where y i represents the ordinate of the i-th ship, y i-1 represents the ordinate of the (i - 1)-th ship, y i+1 represents the ordinate of the (i + 1)-th ship, xi represents the abscissa of the i-th ship, x i-1 represents the abscissa of the (i - 1)-th ship, x i+1 represents the abscissa of the (i + 1)-th ship.

[0093] In practical applications, determining whether the recognition result of the target tug - barge queue is normal according to the offset angle difference between adjacent ships in the tail of the target tug - barge queue and the tail threshold, specifically includes:

[0094] If the offset angle difference between adjacent ships in the tail of the target tug - barge queue is less than the tail threshold, then the recognition result of the target tug - barge queue is normal.

[0095] In practical applications, determining whether the recognition result of the target tug - barge queue is scattered according to the offset angle difference between adjacent ships in the head of the target tug - barge queue and the head threshold, specifically includes:

[0096] Judging whether at least a first quantity of the offset angle differences between adjacent ships in the head of the target tug - barge queue is less than the head threshold, to obtain a third judgment result; the first quantity is the first set percentage of the total quantity of the offset angle differences corresponding to the head of the target tug - barge queue;

[0097] If the third judgment result is no, then the recognition result of the target tug - barge queue is scattered.

[0098] In practical applications, determining the recognition result of the target tug - barge queue according to the offset angle difference between adjacent ships in the middle of the target tug - barge queue and the middle threshold, specifically includes:

[0099] Judging whether at least a second quantity of the offset angle differences between adjacent ships in the middle of the target tug - barge queue is less than the middle threshold, to obtain a fourth judgment result; the second quantity is the first set percentage of the total quantity of the offset angle differences corresponding to the middle of the target tug - barge queue.

[0100] If the fourth judgment result is no, then the recognition result of the target tug - barge queue is scattered.

[0101] If the fourth judgment result is yes, then judging whether the ratios of the offset angle differences greater than the middle threshold to the middle threshold among the offset angle differences between adjacent ships in the middle of the target tug - barge queue are all less than the second set percentage of the middle threshold, to obtain a fifth judgment result.

[0102] If the fifth judgment result is yes, then determining that the recognition result of the target tug - barge queue is normal.

[0103] If the result of the fifth judgment is negative, determine that the recognition result of the target tug-barge queue is scattered.

[0104] The present invention provides a more specific embodiment to introduce in detail the tug-barge queue recognition algorithm provided in the above embodiment. Through the image recognition algorithm and cooperation strategy in the embodiments of the present invention, scattered ships can be recognized in more detail, making the recognition of the tug-barge queue more accurate. The embodiments of the present invention are based on deep learning image recognition technology. To adapt to the queue recognition of tug-barges, a method of multi-camera cooperative recognition is used. Cameras are installed at the head, middle, and tail of the tug-barge fleet. During the fleet's travel, through the camera network module, a set of pictures is sent to the server every 10 seconds. Each set of pictures consists of 3 pictures taken by 3 cameras. Convolutional neural networks are used to extract features from the pictures. Preset anchor boxes are placed on the images, and the pictures are scanned by the sliding window method. A classifier is used to predict whether each anchor box contains a ship. After obtaining the detection box of the ship target in the picture, its center point is taken as the anchor point for queue recognition. The offset angle between every 3 consecutive anchor points is calculated from start to end. If the offset angle difference is less than the threshold, it is judged to be in a queue. The head camera detects whether the first half of the fleet is in a queue, the middle camera detects whether the second half of the fleet is in a queue, and the tail camera makes a rough estimate of the queue of the entire fleet. This embodiment combines the image features of different perspectives of 3 cameras on the tug-barge fleet, avoiding the limitation that it is difficult for a single camera image to capture long-distance features, making a comprehensive evaluation of the entire tug-barge queue, and preferably improving the accuracy of tug-barge queue detection. As Figure 3 shown, the specific steps are as follows:

[0105] (1) Establish a tug-barge queue dataset:

[0106] (1.1) Data collection: Through 3 cameras deployed at the head, middle, and tail of the tug-barge queue, during the fleet's navigation, a set of pictures is sent to the server through the network module every 10 seconds. On the server, the pictures received by each camera are placed in different folders.

[0107] (1.2) Data annotation: Detect and label the pictures collected in each folder on the server, that is, manually frame the ship part on each picture. In this embodiment, 1000 pictures are collected on each camera.

[0108] (1.3) Data segmentation: The images in each folder in (1.2) are respectively established into a training set, a validation set, and a test set in a ratio of 7:1:2.

[0109] (2) Train the ship target detection model:

[0110] (2.1) Data augmentation: Randomly crop, scale, and perform contrast transformation on the training set images and other data augmentation methods to make the training set more diverse.

[0111] (2.2) Feature extraction: Use a convolutional neural network (a convolutional neural network is a general deep learning model that uses convolutional operations to capture local features of the input data and performs feature extraction and dimensionality reduction through multiple convolutional layers and pooling layers) to extract features from the images, and obtain the feature values of each pixel in the image through the fully connected layer (associate the feature map output by the pooling layer with the class labels through the fully connected layer to achieve final classification or prediction).

[0112] (2.3) Calculate anchor boxes:

[0113] (2.3.1) Set preset anchor boxes of multiple sizes, and sequentially map the pixel feature values to a single preset anchor box through the fully connected layer to obtain the feature probability p of each anchor box. Specifically, for example, a preset anchor box of 32×32 is used, and this anchor box slides on the image in turn. Every time it slides one step, the 32×32 pixel features at the current position on the image are mapped to the feature probability p through the fully connected layer. In a sliding window manner, from left to right and from top to bottom, the entire image is detected to obtain multiple candidate target boxes. In this embodiment, preset anchor boxes of three sizes, 32×32, 64×64, and 128×128, are respectively set, and they slide at a step distance of 1 pixel, and the anchor boxes with p greater than 0.6 are selected as candidate target boxes.

[0114] (2.3.2) Obtain the predicted anchor boxes: Use the non-maximum suppression algorithm, set a threshold, and process multiple candidate target boxes to obtain the best predicted anchor boxes. Among them, non-maximum suppression is a commonly used object detection and edge detection algorithm, which judges whether it is a real object by comparing the scores or confidences in the detection results. The specific approach is to select the candidate target box with the highest score in a local area, and then filter out other candidate target boxes whose overlap with this target is higher than the threshold according to a certain threshold. In this way, the most representative target boxes can be retained and redundant detection results can be removed. In this embodiment, the threshold of non-maximum suppression is set to 0.25, that is, in the local area, other target boxes whose overlap with the candidate target box with the highest score is greater than 0.25 are removed.

[0115] (2.5) Model training: Calculate the loss L through the feature probability p of the predicted anchor boxes and the classification result y of the real anchor boxes (that is, there is no ship or there is a ship in the anchor box, obtained according to the annotation), and train the ship target detection model. Because the ship target detection model only judges whether there is a ship in the anchor box, each anchor box part can be used as a binary classification task, and its loss function uses binary cross-entropy:

[0116] L = -(y log(p) + (1 - y) log(1 - p)).

[0117] (3) Determine the queue judgment threshold:

[0118] (3.1) Obtain the center point of the ship target: Use the best predicted anchor box coordinates corresponding to any one of the images obtained in 2.3, and take its center point as the judgment point.

[0119] (3.2) Calculate the offset angle: Obtain the abscissa x of the judgment point of each ship in (3.1) i and the ordinate y i , and re - sort them in descending order according to the ordinate y i . The offset angle between adjacent ships is:

[0120]

[0121] (3.3) Calculate the thresholds of each camera: Calculate the offset angle θ between every two consecutive ships in the normal class of the barge queue in the training sets of the three cameras at the head, middle, and tail (i,i+1) , and then calculate the difference d in the offset angles of every three adjacent ships i :

[0122] d i = θ (i,i+1) - θ (i-1,i)

[0123] Respectively take the maximum value of the difference in offset angles in the normal queue as the threshold τ of each camera j :

[0124]

[0125] where j = (1, 2, 3) respectively represents the three positions of the camera at the head, middle, and tail, and n is the number of ships recognized by each camera. In this embodiment, the finally obtained judgment threshold τ1 of the camera head is 15°, the judgment threshold τ2 of the middle part is 17°, and the judgment threshold τ3 of the tail is 12°.

[0126] (4) Use the object detection of multiple cameras combined with the threshold for the barge fleet:

[0127] In the previous steps, the recognition algorithm for the barge queue based on multi - camera cooperation is basically completed, and the overall recognition architecture is as Figure 2 shown:

[0128] After obtaining the trained ship target detection models for the three cameras in the tug-barge fleet ((obtained in (2))) and the queue judgment thresholds ((obtained in (3))) respectively, every time the server receives a group of pictures from the three cameras at the head, middle, and tail, it inputs them into their respective ship target detection models to obtain the ship target coordinates (i.e., the center point coordinates of the target box for each ship in the picture), and then calculates the difference in offset angles.

[0129] Use the respective judgment thresholds to judge the obtained difference in offset angles. If the difference in offset angles of all detected target ships in the image of the tail camera is less than the threshold, then judge the cameras at the head and middle, otherwise directly judge that the queue is scattered.

[0130] When judging the cameras at the head and middle, if the number of differences in offset angles less than the threshold in these two cameras is greater than or equal to 90% of the total number of differences in offset angles corresponding to the cameras, and the ratio of the part of the difference in offset angles corresponding to the middle camera that is greater than the middle threshold to the middle threshold is less than 30% of the middle threshold, then judge it as a normal formation, otherwise judge that the queue is scattered.

[0131] The present invention has high precision, strong robustness, good scalability, and has broad popularization value. With the development of the shipping industry, the sailing frequency of tug-barge fleets is getting higher and higher, the towing force of tugboats is getting stronger, and the length of their barges is also increasing continuously. Accordingly, the risks brought by scattered tug-barge queues are gradually increasing, and the limitations of previous manual observation or traditional image recognition are also becoming greater and greater. The present invention uses deep learning technology to implement a tug-barge queue recognition algorithm based on multi-camera collaboration. Through the deep learning image target detection algorithm, ships in the queue can be detected more accurately. At the same time, through multiple cameras, by fusing image features in multiple directions and angles, it can accurately judge whether the tug-barge queue is scattered. The present invention has a certain popularization value in other related scenario fields and can adapt to situations where there are far targets and it is difficult to accurately detect in image processing projects.

[0132] The present invention provides a tug-barge queue recognition system for the above method embodiments, including:

[0133] An acquisition module, configured to acquire pictures of the head, middle, and tail of the target tug-barge queue at the current moment.

[0134] A tail ship processing module, configured to input the picture of the tail of the target tug-barge queue acquired at the current moment into the tail ship target detection model to obtain the center point coordinates of the target boxes of all ships at the tail of the target tug-barge queue.

[0135] The tail ship offset angle difference calculation module is used to calculate the offset angle difference between adjacent ships in the tail of the target tug-barge queue according to the target box center point coordinates of all ships in the tail of the target tug-barge queue.

[0136] The first judgment module is used to determine whether the recognition result of the target tug-barge queue is normal according to the offset angle difference between adjacent ships in the tail of the target tug-barge queue and the tail threshold, and obtain the first judgment result.

[0137] The head ship processing module is used to, if the first judgment result is negative, input the picture of the head of the target tug-barge queue obtained at the current moment into the head ship target detection model, and obtain the target box center point coordinates of all ships in the head of the target tug-barge queue.

[0138] The head ship offset angle difference calculation module is used to calculate the offset angle difference between adjacent ships in the head of the target tug-barge queue according to the target box center point coordinates of all ships in the head of the target tug-barge queue.

[0139] The second judgment module is used to determine whether the recognition result of the target tug-barge queue is scattered according to the offset angle difference between adjacent ships in the head of the target tug-barge queue and the head threshold, and obtain the second judgment result.

[0140] The middle ship processing module is used to, if the second judgment result is negative, input the picture of the middle of the target tug-barge queue obtained at the current moment into the middle ship target detection model, and obtain the target box center point coordinates of all ships in the middle of the target tug-barge queue.

[0141] The middle ship offset angle difference calculation module is used to calculate the offset angle difference between adjacent ships in the middle of the target tug-barge queue according to the target box center point coordinates of all ships in the middle of the target tug-barge queue.

[0142] The final judgment module is used to determine the recognition result of the target tug-barge queue according to the offset angle difference between adjacent ships in the middle of the target tug-barge queue and the middle threshold; the recognition result is normal or scattered.

[0143] An embodiment of the present invention also provides an electronic device, including:

[0144] A memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the tug-barge queue recognition method described in the above method embodiment.

[0145] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the tugboat queue recognition method described in the above method embodiment is implemented.

[0146] The advantages of the present invention compared with the prior art are as follows:

[0147] (1) In the aspect of identifying tugboat queues, the present invention can reduce a large amount of labor costs and risks. By using the method of computer vision to identify target ships, it has higher accuracy and better generalization ability compared with traditional image processing methods.

[0148] (2) The present invention focuses on solving the problem that it is difficult to identify distant target ships in a shipping fleet. By using multiple cameras, it processes the problem of ship queue recognition from multiple angles, providing a solution for information judgment at a long distance in a fixed scenario.

[0149] (3) The present invention uses a variety of empirical thresholds for fusion judgment, and uses multi-camera collaboration to obtain correct results as much as possible, showing excellent accuracy and robustness in the field of multi-camera detection of ship queues.

[0150] In this specification, each embodiment is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. For the same or similar parts among the embodiments, reference can be made to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and reference can be made to the description of the method part for related parts.

[0151] In this article, specific examples are used to illustrate the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for identifying a tow barge queue, characterized in that, Including: Obtain pictures of the head, middle, and tail of the target tow barge queue at the current moment; Input the picture of the tail of the target tow barge queue obtained at the current moment into the tail ship target detection model to obtain the center point coordinates of the target bounding boxes of all ships at the tail of the target tow barge queue; Calculate the offset angle differences between adjacent ships in the tail of the target tow barge queue according to the center point coordinates of the target bounding boxes of all ships at the tail of the target tow barge queue; Determine whether the recognition result of the target tow barge queue is normal according to the offset angle differences between adjacent ships in the tail of the target tow barge queue and the tail threshold, and obtain the first judgment result; If the first judgment result is negative, input the picture of the head of the target tow barge queue obtained at the current moment into the head ship target detection model to obtain the center point coordinates of the target bounding boxes of all ships at the head of the target tow barge queue; Calculate the offset angle differences between adjacent ships in the head of the target tow barge queue according to the center point coordinates of the target bounding boxes of all ships at the head of the target tow barge queue; Determine whether the recognition result of the target tow barge queue is scattered according to the offset angle differences between adjacent ships in the head of the target tow barge queue and the head threshold, and obtain the second judgment result; If the second judgment result is negative, input the picture of the middle of the target tow barge queue obtained at the current moment into the middle ship target detection model to obtain the center point coordinates of the target bounding boxes of all ships in the middle of the target tow barge queue; Calculate the offset angle differences between adjacent ships in the middle of the target tow barge queue according to the center point coordinates of the target bounding boxes of all ships in the middle of the target tow barge queue; Determine the recognition result of the target tow barge queue according to the offset angle differences between adjacent ships in the middle of the target tow barge queue and the middle threshold; the recognition result is normal or scattered.

2. The method for identifying a tow barge queue according to claim 1, wherein The determination process of the tail ship target detection model, the head ship target detection model, or the middle ship target detection model is as follows: Obtain a picture set; the picture set includes multiple pictures of the head, middle, or tail of the training tow barge queue with marked ship detection frames; Divide the picture set into a training set, a validation set, and a test set according to a set ratio; Preset anchor boxes of multiple sizes, and input each picture in the training set into the ship target detection model to obtain the feature probabilities of each anchor box corresponding to each picture in the training set; the ship target detection model includes a convolutional neural network and a fully connected module connected in sequence, and the fully connected module includes multiple fully connected layers connected in sequence; Obtain the best predicted anchor box corresponding to each picture in the training set according to the feature probabilities of each anchor box corresponding to each picture in the training set; Obtain a loss function according to the best predicted anchor box corresponding to each picture in the training set and the ship detection frames marked in the training set; Train the ship target detection model with the goal of minimizing the loss function to obtain the trained ship target detection model; The trained ship target detection model is tested and verified using a test set and a validation set to obtain a detection model; when the picture set includes multiple training barge queue head pictures with marked ship detection frames, the detection model is a head ship target detection model; when the picture set includes multiple training barge queue middle pictures with marked ship detection frames, the detection model is a middle ship target detection model; when the picture set includes multiple training barge queue tail pictures with marked ship detection frames, the detection model is a tail ship target detection model.

3. The method for identifying a tow barge queue according to claim 2, wherein The process of determining the tail threshold, the head threshold, or the middle threshold is as follows: According to the best predicted anchor box corresponding to the target picture in the training set, calculate the offset angle of each ship in the target picture; the target picture is any picture of the head, middle, or tail of a training barge queue with a normal barge queue recognition result. Calculate the offset angle difference between adjacent ships in the target picture based on the offset angles of each ship in the target picture. Determine the maximum offset angle difference as the threshold. When the target picture is any training barge queue head picture with a normal barge queue recognition result, the threshold is the head threshold; when the target picture is any training barge queue middle picture with a normal barge queue recognition result, the threshold is the middle threshold; when the target picture is any training barge queue tail picture with a normal barge queue recognition result, the threshold is the tail threshold.

4. The barge train identification method according to claim 1, wherein Calculating the offset angle difference between adjacent ships specifically includes: According to the formula calculate the offset angle θ of the i-th ship (i-1,i) ; According to the formula calculate the offset angle θ of the (i + 1)-th ship (i,i+1) ; According to the formula d i = θ (i,i+1) - θ (i-1,i) Calculate the offset angle difference d between the i-th ship and the (i + 1)-th ship i , where y i represents the ordinate of the i-th ship, y i-1 represents the ordinate of the (i - 1)-th ship, y i+1 represents the ordinate of the (i + 1)-th ship, x i represents the abscissa of the i-th ship, x i-1 represents the abscissa of the (i - 1)-th ship, x i+1 represents the abscissa of the (i + 1)-th ship.

5. The method for identifying a tow barge queue according to claim 1, wherein Determine whether the recognition result of the target barge queue is normal according to the offset angle difference between adjacent ships in the target barge queue tail and the tail threshold, specifically including: If the offset angle difference between adjacent ships in the target barge queue tail is less than the tail threshold, the recognition result of the target barge queue is normal.

6. The method for identifying a tow barge queue according to claim 1, wherein Determine whether the recognition result of the target barge queue is scattered according to the offset angle difference between adjacent ships in the target barge queue head and the head threshold, specifically including: Judge whether at least a first quantity of the offset angle differences between adjacent ships in the target barge queue head is less than the head threshold to obtain a third judgment result; the first quantity is the first set percentage of the total quantity of the offset angle differences corresponding to the target barge queue head. If the third judgment result is no, the recognition result of the target barge queue is scattered.

7. The method for identifying a tow barge queue according to claim 1, characterized in that Determine the recognition result of the target barge queue according to the offset angle difference between adjacent ships in the target barge queue middle and the middle threshold, specifically including: Judge whether at least a second quantity of the offset angle differences between adjacent ships in the target barge queue middle is less than the middle threshold to obtain a fourth judgment result; the second quantity is the first set percentage of the total quantity of the offset angle differences corresponding to the target barge queue middle. If the fourth judgment result is no, the recognition result of the target barge queue is scattered; If the fourth judgment result is yes, then determine that the ratios of the offset angle differences greater than the middle threshold among the offset angle differences between adjacent ships in the middle of the target barge tow queue to the middle threshold are all less than the second set percentage of the middle threshold, and obtain a fifth judgment result; If the fifth judgment result is yes, then determine that the recognition result of the target barge tow queue is normal; If the fifth judgment result is no, then determine that the recognition result of the target barge tow queue is scattered.

8. A barge train recognition system, characterized in that, Including: An acquisition module, configured to acquire pictures of the head, middle, and tail of the target barge tow queue at the current moment; A tail ship processing module, configured to input the picture of the tail of the target barge tow queue acquired at the current moment into a tail ship target detection model to obtain the target box center point coordinates of all ships at the tail of the target barge tow queue; A tail ship offset angle difference calculation module, configured to calculate the offset angle differences between adjacent ships in the tail of the target barge tow queue according to the target box center point coordinates of all ships at the tail of the target barge tow queue; A first judgment module, configured to determine whether the recognition result of the target barge tow queue is normal according to the offset angle differences between adjacent ships in the tail of the target barge tow queue and a tail threshold, and obtain a first judgment result; A head ship processing module, configured to, if the first judgment result is no, input the picture of the head of the target barge tow queue acquired at the current moment into a head ship target detection model to obtain the target box center point coordinates of all ships at the head of the target barge tow queue; A head ship offset angle difference calculation module, configured to calculate the offset angle differences between adjacent ships in the head of the target barge tow queue according to the target box center point coordinates of all ships at the head of the target barge tow queue; A second judgment module, configured to determine whether the recognition result of the target barge tow queue is scattered according to the offset angle differences between adjacent ships in the head of the target barge tow queue and a head threshold, and obtain a second judgment result; A middle ship processing module, configured to, if the second judgment result is no, input the picture of the middle of the target barge tow queue acquired at the current moment into a middle ship target detection model to obtain the target box center point coordinates of all ships in the middle of the target barge tow queue; A middle ship offset angle difference calculation module, configured to calculate the offset angle differences between adjacent ships in the middle of the target barge tow queue according to the target box center point coordinates of all ships in the middle of the target barge tow queue; A final judgment module, configured to determine the recognition result of the target barge tow queue according to the offset angle differences between adjacent ships in the middle of the target barge tow queue and a middle threshold; the recognition result is normal or scattered.

9. An electronic device, characterized in that, Including: A memory and a processor, the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the barge tow queue recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by the processor, it implements the barge tow queue recognition method according to any one of claims 1 to 7.

Images