A method and device for detecting inland river vessel targets based on video surveillance images

Abstract

A method and device for detecting inland river ship targets based on video surveillance images, relating to the technical field of computer vision detection. The invention adopts a semantic aggregation module to aggregate deep features rich in semantic information with shallow features rich in position information, thereby improving the model's ability to classify and locate ship targets. On this basis, by adding a larger-scale target detection layer, the model's detection accuracy for small targets such as fishing boats is guaranteed. By using DIoU to design a loss function, the disadvantage of IoU that the gradient cannot be returned when there is no overlap between the predicted border and the real border is overcome, thereby further improving the model's detection performance for ship targets. Its computing speed can meet the requirements of real-time detection, and has broad application prospects in the fields of maritime safety and maritime management.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision detection, and in particular to a method and device for detecting inland river vessel targets based on video surveillance images. Background Art

[0002] At present, ship target detection technology has become a research hotspot in the field of computer vision. Video surveillance images can be obtained from continuous video sequences and contain rich detailed texture information. Unlike images with a single ocean and sky background, inland river video surveillance ship images are easily affected by changes in imaging size, imaging perspective and distance, and their backgrounds are often complex backgrounds such as buildings, trees and grass. Therefore, ship target detection in inland river video surveillance images is much more difficult.

[0003] There are two main types of ship target detection methods at present, one is the traditional ship target detection method, and the other is the ship target detection method based on deep learning. The traditional ship target detection method includes two stages: candidate region extraction and target recognition. The target recognition stage is mainly achieved by combining artificial feature extraction and machine learning methods such as support vector machines. The generalization ability of the traditional ship target detection method is poor, and the detection accuracy is not high under complex background conditions. At present, most scholars are committed to studying ship target detection methods based on deep learning. For example, the detection accuracy of the Faster R-CNN method is high, but the detection speed does not meet the real-time requirements, and in the selection of candidate regions, it relies on manually designed anchor frames like the SSD method. The ship targets in the actual scene are of different sizes, and the problem of slow convergence of the bounding box regression will occur during the training process. The SSD method finds anchor frames on feature maps of multiple scales and uses features at different positions for regression. The accuracy of ship target detection is better than that of the YOLOv1 method, but it ignores the effective information of the lower-level feature maps, so the detection effect of small targets is still relatively poor. The YOLOv2 method adds a batch normalization layer after the convolution layer to speed up the network training speed, and uses multi-scale training to improve the accuracy of target detection. The YOLOv3 method uses the Darknet-53 network for feature extraction and uses multi-scale features for prediction, which can improve the accuracy of small target detection, but the detection speed is slower than that of the YOLOv2 method. The YOLOv2 method also has shortcomings. On the one hand, it only considers the extraction of shallow features and fusion into deep features, but ignores the fusion of deep features that are beneficial to the detection of small target ships into shallow features. On the other hand, in the detection network part, the YOLOv2 method only uses a 13×13 pixel feature map (when the input video surveillance image size is 416×416 pixels) for prediction. The receptive field size is limited, and there will be missed detection or false detection when processing small targets such as fishing boats, resulting in low average accuracy for the detection of small targets such as fishing boats. Summary of the invention

[0004] The purpose of the present invention is to provide an inland river vessel target detection method based on video surveillance images, so as to solve the problem that the existing detection methods are difficult to meet the real-time detection requirements due to insufficient detection accuracy or detection speed.

[0005] In order to achieve the above object, the present invention adopts the following technical solution: a method for detecting inland river vessel targets based on video surveillance images, comprising the following steps:

[0006] S1. Obtain ship images and corresponding XML annotation files from an existing video surveillance image dataset as a training set, and obtain ship images and corresponding XML annotation files from a video surveillance image dataset to be tested as a test set;

[0007] S2. Use the K-means clustering method to cluster the real annotation boxes of the training set to obtain the optimal width, height and number of initial anchor boxes;

[0008] S3, after preprocessing and data enhancement, the video surveillance images of the training set are input into the Darknet-19 network for feature extraction, and the semantic aggregation module is used to fuse the deep features containing semantic information with the shallow features containing location information. At the same time, the feature fusion module is used to fuse the shallow features into the deep features through the pass-through layer, and a multi-scale target detection layer is added;

[0009] S4. Use DIoU to design the loss function, adopt a multi-scale input training strategy to train the network model, update the weight parameters through continuous iterative training until the loss function converges, and save the optimal training model;

[0010] S5. Input the preprocessed test set ship images into the Darknet-19 network for feature extraction, use the optimal training model to detect the ship images to be detected, obtain the predicted bounding box of the ship target, filter out the predicted bounding box below the threshold, use the non-maximum suppression algorithm to post-process the prediction results, and obtain the target category and bounding box position parameters.

[0011] The size of the ship image obtained in step S1 is 1920×1080 pixels, and the XML annotation file of each ship image contains the name, path, resolution of the original image, category information and coordinate information of all borders of the corresponding image.

[0012] In step S2, the distance metric formula is used:

[0013] d(box,centroid)=1-IoU(box,centroid);

[0014] Among them, box is the true bounding box of the ship target, centroid is the central anchor box of the cluster, IoU(box, centroid) is the intersection over union ratio between the true bounding box of the ship target and the central anchor box of the cluster, and d is the distance between the bounding box of the ship target and the central anchor box of the cluster;

[0015] And combine the following formula to calculate the maximum average of the IoU values ​​of the real bounding boxes of all ship targets and the central anchor boxes of the cluster to get the average IoU:

[0016]

[0017] Among them, q represents the total number of ship targets, k represents the number of clusters, and q k Represents the number of ship targets in the kth cluster center;

[0018] The k value is determined through multiple clustering experiments and comprehensive consideration of the recall rate and complexity of the detection model, thereby obtaining the optimal width, height and number of the initial anchor boxes.

[0019] Furthermore, step S3 and step S5 also include a step of preprocessing the video surveillance image. When preprocessing the video surveillance image, the training set image is scaled to a fixed size of N×N pixels every 10 epochs, N∈[320,608], with a step size of 32, and the test set image is downsampled to 480×480 pixels.

[0020] In addition, step S3 also includes a step of performing data enhancement on the preprocessed video surveillance images. After preprocessing, the video surveillance images of the training set are randomly flipped, hue changed, saturation changed, and exposure changed to achieve data enhancement.

[0021] The Darknet-19 network consists of 19 convolutional layers and 5 maximum pooling layers. The convolution kernel sizes are 3×3 and 1×1. A 1×1 convolution kernel is used between the 3×3 convolution kernels to compress features and deepen the network depth. After batch normalization, a leaky ReLU activation function is used to avoid the gradient disappearance phenomenon in the deep network.

[0022] In step S3, the semantic aggregation module first performs 1×1 convolution and 2x upsampling on the deep features and then aggregates them with the shallow features; the feature fusion module first reduces the dimension of the shallow features through 1×1 convolution, and uses the reorg layer to deform them, and then fuses them with the deep features; the multi-scale object detection layer is a 26×26 object detection layer composed of a semantic aggregation module and a feature fusion module added to the 13×13 object detection layer.

[0023] In step S4, DIoU (Distance Intersection over Union) is used to design the loss function, specifically:

[0024]

[0025] Among them, L confidence Represents the confidence error of the background, L coord represents the coordinate error between the anchor box and the predicted box, L groundtruth It represents the sum of the coordinate error, confidence error and classification error of the predicted bounding box that matches each true bounding box; W is the width of the feature map, H is the height of the feature map, A is the number of anchor boxes corresponding to each grid, i, j, k represent the row and column where the center of the current ship target is located and the category to which the current ship target belongs respectively; Indicates that there is no ship target in the current grid, λ noobj Indicates the weight coefficient of no ship target, l′=1 MaxDIoU＜Thresh When the maximum DIoU between the predicted border and the true border is less than the set threshold, the predicted border is marked as background; prior represents the weight coefficient of the anchor box, represents the coordinates of the anchor box of the kth class, represents the coordinates of the predicted box of the kth class, r represents the position of the anchor box and the predicted box, l = 1 t＜12800 represents the coordinate error between the anchor box and the predicted box of the first 12800 batches, so that the predicted box can quickly learn the shape of the anchor box in the early stage of network training; coord is the weight coefficient of coordinate error, truth r represents the coordinates of the real bounding box, represents the coordinates of the predicted bounding box of the kth class, λ obj Indicates the weight coefficient of the ship target, λ class represents the weight coefficient of the category, c represents the category to which the current ship target belongs, C T Indicates the total number of categories, truth c Indicates the true category of the ship target, Indicates the category of the ship target to which the predicted bounding box belongs, Represents the DIoU between the real bounding box and the predicted bounding box.

[0026] In step S4, during the process of training the network model with the multi-scale input training strategy, after every 10 batches, the size of the input video surveillance image is randomly selected from {320, 352, 384, 416, 448, 480, 512, 544, 576, 608}, and the corresponding size of the detection output feature map is {10, 11, 12, 13, 14, 15, 16, 17, 18, 19}; the stochastic gradient descent optimization algorithm is used to solve the minimum value of the loss function to obtain the optimal training model.

[0027] Finally, the present invention also relates to an inland river vessel target detection device based on video surveillance images, which includes a processor, a graphics card and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor and the graphics card, the inland river vessel target detection method as described above is implemented.

[0028] Compared with the prior art, the present invention uses a semantic aggregation module to aggregate deep features rich in semantic information with shallow features rich in location information, which can improve the model's ability to classify and locate ship targets. On this basis, combined with the added multi-scale target detection layer and using DIoU to design the loss function, the model's detection accuracy for small targets such as fishing boats is guaranteed, and the disadvantage of IoU that the gradient cannot be returned when there is no overlap between the predicted border and the real border is overcome, and the average accuracy of ship target detection is further improved, so that the present invention has both high detection speed and detection accuracy, and can meet the requirements of real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Flow chart of the inland river vessel target detection method based on video surveillance images in Example 1;

[0030] Figure 2 It is a specific parameter diagram of the network model in Example 1;

[0031] Figure 3 This is the structural diagram of the multi-scale target detection layer;

[0032] Figure 4 It is the visual detection result of the existing YOLOv2 method and the method involved in the embodiment on some samples; wherein, Figure 4 (a) Figure 4 (c) Figure 4 (e) Figure 4 (g) is the detection result of the existing YOLOv2 method. Figure 4 (b) Figure 4 (d) Figure 4 (f) Figure 4 (h) is the detection result of the method involved in the embodiment. DETAILED DESCRIPTION

[0033] In order to facilitate those skilled in the art to better understand the improvements of the present invention relative to the prior art, the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] Example 1

[0035] This embodiment 1 relates to an inland river vessel target detection method based on video surveillance images, hereinafter referred to as "this method", and its implementation process is as follows: Figure 1 shown.

[0036] 1. Obtain ship images and XML annotation files from an existing video surveillance image dataset and divide them into a training set and a test set. It should be noted that when the method is actually applied, the test set should be obtained from the video surveillance image dataset to be tested. Since this embodiment is only used to illustrate and verify the method, the ship images and XML annotation files obtained from the existing video surveillance image dataset can be used as the test set.

[0037] Video surveillance images are randomly selected from the video surveillance image dataset, and the selected video images are divided into a training set and a test set in a ratio of 4:1. The training set includes 5600 video surveillance images with a size of 1920×1080 pixels and 5600 XML files corresponding to the video surveillance images, and the test set includes 1400 video surveillance images and 1400 XML files corresponding to them. The XML file of each video surveillance image contains the name, path, resolution of the original image, category information and coordinate information of all borders of the corresponding image. The above-obtained dataset contains 6 categories, namely: bulk cargo carrier, container ship, fishing boat, general cargo ship, ore carrier and passenger ship.

[0038] 2. Use the K-means clustering method to cluster the real annotated bounding boxes of the training set to obtain the optimal width, height and number of initial anchor boxes.

[0039] When using the K-means clustering method to cluster the real annotation bounding boxes of the training set, the formula for the distance metric used is:

[0040] d(box,centroid)=1-IoU(box,centroid);

[0041] Among them, box is the true bounding box of the ship target, centroid is the central anchor box of the cluster, IoU (box, centroid) is the intersection over union (IoU) of the true bounding box of the ship target and the central anchor box of the cluster, and d is the distance between the bounding box of the ship target and the central anchor box of the cluster. Using this distance metric can achieve the goal of obtaining a high IoU value through a preset anchor box, and the IoU value is independent of the size of the preset anchor box. The maximum average of the IoU values ​​of the true bounding boxes of all ship targets and the central anchor boxes of the cluster is calculated to obtain the average IoU, and its objective function is shown as follows:

[0042]

[0043] Among them, q represents the total number of ship targets, k represents the number of clusters, and q k Represents the number of ship targets in the kth cluster center. After multiple clustering experiments, the average IoU values ​​shown in Table 1 are obtained when the cluster number k takes different values. When k < 5, the average IoU value increases relatively quickly, and when k > 5, the average IoU value becomes relatively stable. Considering the recall rate and complexity of the detection model, taking the initial anchor box when k = 5 can not only speed up the convergence of the loss function during training, but also reduce the error between the real box and the predicted box. When k = 5, the initial anchor boxes are: (0.982, 0.457), (2.085, 0.831), (3.683, 1.396), (6.371, 1.998), (8.849, 3.298), the first item in the brackets represents the width of the anchor box, and the second item represents the height of the anchor box.

[0044] Table 1 Average IoU when the number of clusters k takes different values

[0045]

[0046] 3. After preprocessing and data enhancement, the video surveillance images of the training set are input into the Darknet-19 network for feature extraction. The semantic aggregation module is used to fuse the deep features rich in semantic information with the shallow features rich in location information. At the same time, the feature fusion module is used to fuse the shallow features into the deep features through the pass-through layer, and a multi-scale target detection layer is added. The specific method is:

[0047] The training set images are preprocessed into N×N pixels every 10 epochs, where N∈[320,608] and the step size is 32. The preprocessed video surveillance images are then subjected to data enhancement by random flipping, hue change, saturation and exposure change. The probability of random flipping is 1 / 2, the hue change range is -0.1 to 0.1, and the saturation and exposure change size is 1 to 1.5 times.

[0048] The Darknet-19 network consists of 19 convolutional layers and 5 maximum pooling layers. The convolution kernel sizes are 3×3 and 1×1. A 1×1 convolution kernel is used between the 3×3 convolution kernels to compress features and deepen the network depth, so that the number of convolution filters after each maximum pooling layer becomes twice the original number. After batch normalization, the leaky ReLU activation function is used to avoid the gradient disappearance phenomenon in the deep network. The expression of the leaky ReLU function is as follows:

[0049]

[0050] Where x represents the input value, i represents different channels, and a i is a fixed value, ranging from (1, +∞). In this method, a i Take 10.

[0051] Figure 2 The figure shows the parameters of the YOLOv2 network model used in this method. In the YOLOv2 method, the last convolution layer of the Darknet-19 network is first removed, and three convolution layers are added, where the convolution kernel size is 3×3 and the number of convolution kernels is 1024; after the added second convolution layer (serial number 24), there is a route layer, which directly connects the feature map of size 26×26×512 output by the 16th layer to the front of the 26th layer. The 26th layer is a convolution layer with 64 convolution kernels and a convolution kernel size of 1×1. The 27th layer is a reorg layer, which deforms the 26×26×64 feature map output by the 26th layer into a 13×13×256 feature map to utilize the fine-grained features of the model. The 28th layer is the route layer, which concatenates the output feature maps of the 24th and 27th layers to obtain a 13×13×1280 feature map. Finally, a convolution layer with 1024 convolution kernels and a convolution kernel size of 3×3 is used to fuse cross-channel information to obtain a 13×13×1024 feature map. The 30th layer is a convolution layer with 55 convolution kernels and a convolution kernel size of 1×1. The 55 here can be expressed as 5×(6+5), where the 5 outside the brackets is the number of anchor boxes predicted for each pixel grid, the 6 in the brackets is the number of ship categories in the video surveillance image dataset, and the 5 in the brackets represents the 4 coordinate values ​​(center point coordinates, height and width) and confidence scores of each border.

[0052] The semantic aggregation module first performs 1×1 convolution and 2x upsampling on the deep features of the 24th layer (size: 13×13×1024) and then aggregates them with the shallow features of the 16th layer (size: 26×26×512). In this method, two feature fusion modules are used. Feature fusion module 1 first reduces the dimension of the shallow features of the 16th layer by convolution with 64 convolution kernels and 1×1 convolution kernel size, and uses the reorg layer to deform it to obtain a 13×13×256 feature map, and then fuses the feature map with the deep features of the 24th layer (size: 13×13×1024). Feature fusion module 2 deforms the shallow features of the 10th layer using the reorg layer to obtain a 26×26×1024 feature map, and then fuses the feature map with the shallow features of the 16th layer (size: 26×26×512).

[0053] The structure diagram of the multi-scale target detection layer in this method is as follows Figure 3 As shown in FIG. 13 , the composition of the object detection layer of size 13×13 is consistent with the process of the feature fusion module 1. The object detection layer of size 26×26 is composed of the semantic aggregation module and the feature fusion module 2. Figure 3 Convs in the above means that a convolution layer with 1024 convolution kernels and a convolution kernel size of 3×3 is first used to perform cross-channel information fusion, and then a 1×1 convolution is used to reduce the dimension of the feature map.

[0054] 4. Use DIoU (Distance Intersection over Union) to design the loss function, adopt a multi-scale input training strategy to train the network model, update the weight parameters through continuous iterative training until the loss function converges and save the optimal training model.

[0055] First, the DIoU method is used to directly minimize the normalized distance between the center point of the predicted bounding box and the true bounding box, which can be expressed as:

[0056]

[0057] Among them, b p is the predicted bounding box B p The center point, b gt is the real border B gt The center point of , ρ(·) is the Euclidean distance, c d Represents the diagonal distance of the minimum enclosing area that contains both the predicted box and the true box. DIoU is more in line with the mechanism of target box regression.

[0058] The loss function designed using DIoU (Distance Intersection over Union) can be expressed as:

[0059]

[0060] Among them, L confidence Represents the confidence error of the background, L coord represents the coordinate error between the anchor box and the predicted box, L groundtruth It represents the sum of the coordinate error, confidence error and classification error of the predicted bounding box that matches each true bounding box. W is the width of the feature map, H is the height of the feature map, A is the number of anchor boxes corresponding to each grid, i, j, k represent the row and column where the center of the current ship target is located and the category to which the current ship target belongs respectively; Indicates that there is no ship target in the current grid, λ noobj Indicates the weight coefficient of no ship target, l′=1 MaxDIoU＜Thresh When the maximum DIoU between the predicted border and the true border is less than the set threshold, the predicted border is marked as background. prior represents the weight coefficient of the anchor box, represents the coordinates of the anchor box of the kth class, represents the coordinates of the predicted box of the kth class, r represents the position of the anchor box and the predicted box, l = 1 t＜12800 Indicates the coordinate error between the anchor box and the predicted box of the first 12800 batches, so that the predicted box can quickly learn the shape of the anchor box in the early stage of network training. coord is the weight coefficient of coordinate error, truth r represents the coordinates of the real bounding box, represents the coordinates of the predicted bounding box of the kth class, λ obj Indicates the weight coefficient of the ship target, λ class represents the weight coefficient of the category, c represents the category to which the current ship target belongs, C T Indicates the total number of categories, truth c Indicates the true category of the ship target, Indicates the category of the ship target to which the predicted bounding box belongs. Represents the DIoU between the real bounding box and the predicted bounding box.

[0061] When the network model is trained using the multi-scale input training strategy, the size of the input video surveillance image is randomly selected from {320, 352, 384, 416, 448, 480, 512, 544, 576, 608} after every 10 batches during the training process, and the corresponding detection output feature map size is {10, 11, 12, 13, 14, 15, 16, 17, 18, 19}. For the same trained model, the test set video surveillance images of different sizes can be tested.

[0062] The above-mentioned method of updating the weight parameters through continuous iterative training until the loss function converges and saving the optimal training model is as follows: the loss function is optimized by using a stochastic gradient descent optimization algorithm until the loss function converges to obtain the optimized optimal training model. The parameters of this embodiment are set as follows: the size of the input training set video surveillance image is 416×416 pixels, the number of batch samples is 64, the maximum number of iterations is 17500 times, the momentum is 0.9, the weight decay coefficient is 0.0005, the initial learning rate is 0.0001, and the learning rate decay step boundaries are: "400, 700, 900, 1000, 15000", and the learning rates corresponding to the step boundaries are: "0.0001, 0.0005, 0.0005, 0.001, 0.0001". λ prior is 1, λ coord is 1, λ obj is 5, λ noobj is 1, is 0, Thresh is 0.6. The confidence threshold is 0.1, the DIoU threshold is 0.5, and N t Set to 0.45.

[0063] 5. The preprocessed test set ship images are input into the Darknet-19 network for feature extraction, and the ship images to be detected are detected using the optimal training model to obtain the predicted bounding boxes of the ship targets, and the predicted bounding boxes below the threshold are filtered out. The prediction results are post-processed using the non-maximum suppression algorithm to obtain the target category and bounding box position parameters.

[0064] The non-maximum suppression process is: sort all predicted bounding boxes according to the confidence score, select the predicted bounding box with the highest score, and then traverse the remaining predicted bounding boxes. If the IoU between a predicted bounding box and the predicted bounding box with the highest confidence score is greater than the set threshold N t , delete this predicted border, and then select a predicted border with the highest score from the remaining unprocessed predicted borders, repeat the above process, and the predicted border that is finally retained is the optimal predicted border. The formula of the attenuation function of non-maximum suppression is as follows:

[0065]

[0066] Among them, b i To predict the bounding box, is the predicted bounding box with the highest current confidence score, s i is the target score of the predicted bounding box, N t is the IoU threshold.

[0067] The average accuracy of the 1400 video surveillance images in the test set of this embodiment is 89.30%, the frame rate per second (FPS) is 28, the average accuracy for bulk cargo carrier is 0.8920, the average accuracy for container ship is 0.9091, the average accuracy for fishing boat is 0.8694, the average accuracy for general cargo ship is 0.8991, the average accuracy for ore carrier is 0.9024, and the average accuracy for passenger ship is 0.8863. The average accuracy of the traditional YOLOv2 method is 85.15%, the average accuracy for bulk cargo carrier is 0.8664, the average accuracy for container ship is 0.9085, the average accuracy for fishing boat is 0.7681, the average accuracy for general cargo ship is 0.8681, the average accuracy for ore carrier is 0.8979, and the average accuracy for passenger ship is 0.7998. In contrast, the method of the present invention not only has a high average accuracy, but also improves the average accuracy of fishing boats by 13.19%, which is the ship type with the largest average accuracy improvement among the six ship types.

[0068] The visual detection results of the existing YOLOv2 method and the method of the present invention on some samples are as follows: Figure 4 As shown, Figure 4 (a) Figure 4 (c) Figure 4 (e) Figure 4 (g) is the detection result of the existing YOLOv2 method. Figure 4 (b) Figure 4 (d) Figure 4 (f) Figure 4 (h) is the detection result of the method of the present invention. Figure 4 It can be seen that this method can avoid detecting similar backgrounds as ship targets, and can obtain better detection results and more accurate positioning when there are occlusions between ship targets and small targets such as fishing boats.

[0069] In summary, this method improves the model's ability to classify and locate ship targets by using a semantic aggregation module to aggregate deep features rich in semantic information with shallow features rich in location information. On this basis, by adding a larger-scale target detection layer, the model's detection accuracy for small targets such as fishing boats is guaranteed. By using DIoU to design the loss function, the disadvantage of IoU that the gradient cannot be returned when there is no overlap between the predicted bounding box and the true bounding box is overcome, further improving the model's detection performance for ship targets. Its computing speed can meet the requirements of real-time detection and can be applied in the fields of maritime safety and maritime management.

[0070] Example 2

[0071] This embodiment 2 relates to an inland vessel target detection device based on video surveillance images, which includes a processor, a graphics card and a memory. A computer program is stored on the memory. When the computer program is executed by the processor and the graphics card, the inland vessel target detection method based on video surveillance images provided in embodiment 1 is implemented.

[0072] Specifically, the processor can be an Inter(R)Core(TM) i9-7980XE@2.6GHz processor, 32GB memory, and a graphics card of NVIDIA TITAN Xp Pascal, and software programming is performed on the Pytorch framework using python3.7.

[0073] The above-mentioned inland river vessel target detection device based on video surveillance images is used to implement the inland river vessel target detection method based on video surveillance images in Example 1. Therefore, the technical effects possessed by the inland river vessel target detection method based on video surveillance images are also possessed by the inland river vessel target detection device based on video surveillance images, which will not be repeated here.

[0074] The above embodiments are preferred implementation schemes of the present invention. In addition, the present invention may also be implemented in other ways. Any obvious replacement without departing from the concept of the present technical solution is within the protection scope of the present invention.

[0075] In order to make it easier for ordinary technicians in the field to understand the improvements of the present invention over the prior art, some drawings and descriptions of the present invention have been simplified, and for the sake of clarity, some other elements are omitted in this application document. Ordinary technicians in the field should realize that these omitted elements may also constitute the content of the present invention.

Claims

1. A method for detecting inland river vessel targets based on video surveillance images, characterized in that: The following steps are involved: S1. Obtain ship images and corresponding XML annotation files from an existing video surveillance image dataset as a training set, and obtain ship images and corresponding XML annotation files from a video surveillance image dataset to be tested as a test set; S2. Use the K-means clustering method to cluster the real annotation boxes of the training set to obtain the optimal width, height and number of initial anchor boxes; S3, after preprocessing and data enhancement, the video surveillance images of the training set are input into the Darknet-19 network for feature extraction, and the semantic aggregation module is used to fuse the deep features containing semantic information with the shallow features containing location information. At the same time, the feature fusion module is used to fuse the shallow features into the deep features through the pass-through layer, and a multi-scale target detection layer is added; The semantic aggregation module first performs 1×1 convolution and 2-fold upsampling on the deep features and then aggregates them with the shallow features; The feature fusion module first reduces the dimension of the shallow features through 1×1 convolution, and uses the reorg layer to deform them, and then fuses them with the deep features; the multi-scale object detection layer is a 26×26 object detection layer composed of a semantic aggregation module and a feature fusion module added to the 13×13 object detection layer; S4. Use DIoU to design the loss function, adopt a multi-scale input training strategy to train the network model, update the weight parameters through continuous iterative training until the loss function converges, and save the optimal training model; S5. Input the preprocessed test set ship images into the Darknet-19 network for feature extraction, use the optimal training model to detect the ship images to be detected, obtain the predicted bounding box of the ship target, filter out the predicted bounding box below the threshold, use the non-maximum suppression algorithm to post-process the prediction results, and obtain the target category and bounding box position parameters.

2. The inland river vessel target detection method according to claim 1, characterized in that: The size of the ship image obtained in step S1 is 1920×1080 pixels. The XML annotation file of each ship image contains the name, path, resolution of the original image, category information and coordinate information of all borders of the corresponding image.

3. The inland river vessel target detection method according to claim 1, characterized in that: In step S2, the distance metric formula is used: d(box,centroid)=1-IoU(box,centroid); Among them, box is the true bounding box of the ship target, centroid is the central anchor box of the cluster, IoU(box, centroid) is the intersection over union ratio between the true bounding box of the ship target and the central anchor box of the cluster, and d is the distance between the bounding box of the ship target and the central anchor box of the cluster; And combine the following formula to calculate the maximum average of the IoU values ​​of the real bounding boxes of all ship targets and the central anchor boxes of the cluster to get the average IoU: Among them, q represents the total number of ship targets, k represents the number of clusters, and q k Represents the number of ship targets in the kth cluster center; The k value is determined through multiple clustering experiments and comprehensive consideration of the recall rate and complexity of the detection model, thereby obtaining the optimal width, height and number of the initial anchor boxes.

4. The inland river vessel target detection method according to claim 2, characterized in that: Step S3 and step S5 also include the step of preprocessing the video surveillance images. When preprocessing the video surveillance images, the training set images are scaled to a fixed size of N×N pixels every 10 epochs, N∈[320,608], with a step size of 32, and the test set images are downsampled to 480×480 pixels.

5. The inland river vessel target detection method according to claim 1, characterized in that: Step S3 also includes a step of performing data enhancement on the preprocessed video surveillance images. After preprocessing, the video surveillance images of the training set are randomly flipped, hue changed, saturation changed, and exposure changed to achieve data enhancement.

6. The inland river vessel target detection method according to claim 4, characterized in that: The Darknet-19 network consists of 19 convolutional layers and 5 maximum pooling layers. The convolution kernel sizes are 3×3 and 1×1. A 1×1 convolution kernel is used between the 3×3 convolution kernels to compress features and deepen the network depth. After batch normalization, a leaky ReLU activation function is used to avoid the gradient disappearance phenomenon in the deep network.

7. The inland river vessel target detection method according to claim 1, characterized in that: In step S4, DIoU (Distance Intersection over Union) is used to design the loss function, specifically: Among them, L confidence Represents the confidence error of the background, L coord represents the coordinate error between the anchor box and the predicted box, L groundtruth It represents the sum of the coordinate error, confidence error and classification error of the predicted bounding box that matches each true bounding box; W is the width of the feature map, H is the height of the feature map, A is the number of anchor boxes corresponding to each grid, i, j, k represent the row and column where the center of the current ship target is located and the category to which the current ship target belongs respectively; Indicates that there is no ship target in the current grid, λ noobj Indicates the weight coefficient of no ship target, l′=1 MaxDIoU<Thresh When the maximum DIoU between the predicted border and the true border is less than the set threshold, the predicted border is marked as background; prior represents the weight coefficient of the anchor box, represents the coordinates of the anchor box of the kth class, represents the coordinates of the predicted box of the kth class, r represents the position of the anchor box and the predicted box, l = 1 t＜12800 represents the coordinate error between the anchor box and the predicted box of the first 12800 batches, so that the predicted box can quickly learn the shape of the anchor box in the early stage of network training; coord is the weight coefficient of coordinate error, truth r represents the coordinates of the real bounding box, represents the coordinates of the predicted bounding box of the kth class, λ obj Indicates the weight coefficient of the ship target, λ class represents the weight coefficient of the category, c represents the category to which the current ship target belongs, C T Indicates the total number of categories, truth c Indicates the true category of the ship target, Indicates the category of the ship target to which the predicted bounding box belongs, Represents the DIoU between the real bounding box and the predicted bounding box.

8. The inland river vessel target detection method according to claim 1, characterized in that: In step S4, in the process of training the network model with the multi-scale input training strategy, after every 10 batches, the size of the input video surveillance image is randomly selected from {320, 352, 384, 416, 448, 480, 512, 544, 576, 608}, and the corresponding size of the detection output feature map is {10, 11, 12, 13, 14, 15, 16, 17, 18, 19}; the stochastic gradient descent optimization algorithm is used to solve the minimum value of the loss function to obtain the optimal training model.

9. An inland river vessel target detection device based on video surveillance images, characterized in that: It includes a processor, a graphics card and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor and the graphics card, the inland vessel target detection method as described in any one of claims 1 to 8 is implemented.

Images