An Infrared Shoreline Segmentation and Target Detection Fusion Method for Unmanned Surface Vessels in Dark Conditions

Abstract

The infrared shoreline segmentation and target detection fusion method for unmanned surface vessels under dark conditions provided by the present invention first establishes an infrared shoreline segmentation and target detection dataset, then establishes a DeeplabV3 Plus model and a YOLOv5L model, and uses the infrared shoreline segmentation and target detection dataset to train the two models to obtain training weights. Then, the DeeplabV3 Plus model and the YOLOv5L model under the training weights are evaluated and predicted, the hyperparameters are adjusted, and a Pytorch framework network model for cascading the DeeplabV3 Plus model and the YOLOV5m model at the decision level is established. Finally, the weight file of the Pytorch framework network model is converted into the weight file of the TensorRT framework network model and migrated to the edge computing platform on the unmanned surface vessel to realize the recognition of water targets and the feasible region based on the edge computing platform.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence, and particularly to a method for fusing infrared shoreline segmentation and target detection of an unmanned surface vehicle under dark conditions. Background Art

[0002] Traditional unmanned surface vehicle water surface perception technologies mainly rely on sensors such as millimeter-wave radars, lidars (LiDAR), inertial measurement units, and GPSs carried on the unmanned surface vehicle. In recent years, perception technologies based on computer vision have developed rapidly. Optical images contain richer detailed information on target areas. Therefore, perception technologies based on vision are more conducive to effectively identifying water surface targets. Currently, there is little research on shoreline segmentation based on infrared thermal imaging in the industry, and there are still huge challenges for unmanned surface vehicles to navigate at night. Therefore, it is particularly important to study the technology for identifying water surface targets, perceiving and positioning water area environments of unmanned surface vehicles based on infrared thermal imaging visual images. Summary of the Invention

[0003] In order to achieve effective identification of water surface targets and navigable areas by an unmanned surface vehicle under dark conditions, the present invention provides a method for fusing infrared shoreline segmentation and target detection of an unmanned surface vehicle under dark conditions.

[0004] To solve the above technical problems, the present invention adopts the following technical method: A method for fusing infrared shoreline segmentation and target detection of an unmanned surface vehicle under dark conditions, including:

[0005] Step S1, dataset establishment: The unmanned surface vehicle is equipped with multiple infrared thermal imagers, and the infrared thermal imagers are used to take low-altitude photos in a water area scene, simulating the navigation perspective of the unmanned surface vehicle, taking videos of the water surface, processing the videos to obtain original images, using annotation tools to annotate the original image data to obtain an infrared shoreline segmentation dataset and an infrared target detection dataset, and dividing both of these datasets into a training set, a validation set, and a test set according to a predetermined ratio;

[0006] Step S2, Model Establishment: The DeeplabV3 Plus model is adopted as the network model for shoreline segmentation, and the YOLOV5m model is adopted as the network model for object detection; set the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model, and use the YOLOV5m model weights based on the voc2012 dataset as the pre-training weights for transfer learning. Subsequently, use the training set of the infrared shoreline segmentation dataset to repeatedly train and validate the DeeplabV3 Plus model, and use the training set of the infrared object detection dataset to repeatedly train and validate the YOLOV5m model to obtain the training weights based on the infrared shoreline segmentation dataset and the infrared object detection dataset; then use the test set and validation set of the infrared shoreline segmentation dataset to evaluate and predict the DeeplabV3 Plus model under this training weight, and use the test set and validation set of the infrared object detection dataset to evaluate and predict the YOLOV5m model under this training weight. Continuously adjust the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model according to the evaluation and test results, and establish a Pytorch framework network model that cascades the DeeplabV3 Plus model and the YOLOV5m model with optimized hyperparameters at the decision level; convert the weight file of the Pytorch framework network model into the weight file of the TensorRT framework network model and transfer it to the edge computing platform on the unmanned boat;

[0007] Step S3, Model Application: Transmit the infrared shoreline segmentation data and infrared object detection data obtained by processing the water area scene video captured in real time by the infrared thermal imager carried by the unmanned boat through the method involved in Step S1 to the edge computing platform, and process them by the TensorRT framework network model to obtain the recognition results of water targets and the feasible region.

[0008] Further, in step S1, dataset establishment: An infrared thermal imager of the DJI M300 drone is mounted on the unmanned boat. The infrared thermal imager takes low-altitude photos in the water area scene, simulating the navigation perspective of the unmanned boat, and takes videos of the water surface. Then, the videos are frame-extracted, duplicate-removed, and screened to obtain original images. Then, the annotation tool "Labelimg" is used to perform rectangular box annotation on the water targets, and the annotation tool "Labelme" is used to perform polygon box annotation on the feasible region to obtain the infrared shoreline segmentation dataset and the infrared target detection dataset. The infrared shoreline segmentation dataset includes three categories: background, water, and obstacle; the infrared target detection dataset includes five categories: boat, person_shore, person_boat, swimming, and dolphin1. The infrared shoreline segmentation dataset and the infrared target detection dataset are both divided into three sub-datasets: training set, validation set, and test set, according to the ratio of 8:1:1. The number of samples in each category is the same in each sub-dataset.

[0009] Further, when the DeeplabV3 Plus model is used as the network model for shoreline segmentation:

[0010] The resnet is used as the backbone feature extraction network. The encoder main part decomposes the standard convolution into depthwise convolution and pointwise convolution. The depthwise convolution uses spatial convolution independently for each channel, and the pointwise convolution is used to combine the output of the depthwise convolution. In the encoder, for the initially effective feature layer compressed four times, parallel atrous convolutions are used to extract features with different rates respectively, then concatenated and merged, and then 1×1 convolution is used to compress the features to obtain the feature map. In the decoder, the initially effective feature layer compressed twice is used to adjust the number of channels by 1×1 convolution, and then stacked with the upsampled result of the effective feature after atrous convolution output by the encoder. After the stacking is completed, two depthwise separable convolutions are performed to obtain the final effective feature layer, and then a 1×1 convolution is used to adjust the channels to Num_Classes, and finally resize is used for upsampling so that the width and height of the final output layer are the same as those of the input image.

[0011] Furthermore, when the YOLOV5m model is used as the network model for target detection:

[0012] The darknet-53 is used as the backbone feature extraction network. The input image is subjected to feature extraction by darknet53. A total of three feature layers are extracted in the feature utilization part. The three feature layers are subjected to 5 times of convolution processing. After the processing, part of them is used to output the prediction results corresponding to the feature layer, and part of them is used to perform deconvolution umSampling2d and then combined with other feature layers.

[0013] Preferably, the darknet-53 is composed of the darknetConv2D and the residual network residual module. The residual convolution in the darknet53 first performs a convolution with a kernel size of 3*3 and a stride of 2, and then saves this convolution layer. Then, it performs a convolution with a kernel size of 1*1 and a convolution with a kernel size of 3*3, and adds this result to the layer as the final result. Then, a large number of residual skip connections are used for five times of downsampling with a stride of 2 and a kernel size of 3. The feature dimensions are 64, 128, 256, 512, and 1024 respectively. The average pooling layer and the fully connected layer are not used. L2 regularization is performed during each convolution. After the convolution, BatchNormalization normalization and the LeakyReLU activation function are performed. The LeakyReLU activation function is:

[0014]

[0015] In the feature utilization part of the YOLOV5m model, multiple feature layers are extracted for object detection. A total of three feature layers are extracted. The three feature layers are located at different positions of the backbone part darknet53, namely the middle layer, the middle and lower layer, and the bottom layer. The shapes of the three feature layers are (52, 52, 256), (26, 26, 512), and (13, 13, 1024) respectively. The three feature layers are subjected to 5 times of convolution processing. After the processing, part of them is used to output the prediction results corresponding to the feature layer, and part of them is used to perform deconvolution UmSampling2d and then combined with other feature layers.

[0016] Furthermore, the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model include the size of the image samples in the input dataset, the batch size, the number of iterations, the learning rate, and the number of classes.

[0017] Still further, in step S2, after repeatedly training and validating the DeeplabV3 Plus model and the YOLOV5m model, the cross-entropy loss function and the Adam loss function optimizer are used to continuously optimize the hyperparameters of the two models, and the training weights based on the infrared shoreline segmentation dataset and the infrared object detection dataset are obtained.

[0018] Preferably, in the step S2, when evaluating and predicting the DeeplabV3 Plus model and the YOLOV5m model under the training weights:

[0019] 1) Evaluation: Screen the obtained training weights, and select the weights with the lowest total loss and val loss as the weights of the DeeplabV3 Plus model and the YOLOV5m model; respectively use the test sets of the infrared shoreline segmentation dataset and the infrared target detection dataset to test the DeeplabV3 Plus model and the YOLOV5m model, obtain the mIoU of the DeeplabV3Plus model and the mean average precision MAP value of the YOLOV5m model, adjust the hyperparameters of the DeeplabV3Plus model and the YOLOV5m model according to the required values of the evaluation indicators, and retrain until the requirements are met; at the same time, use the tensorboard tool module under the tensorflow framework to draw the train loss and val loss curves;

[0020] 2) Prediction: Use the validation set of the infrared shoreline segmentation dataset to test the DeeplabV3 Plus model to obtain a mask, and then calculate the IoU of each category to statistically obtain the mIoU; use the test set of the infrared target detection dataset to test the YOLOV5m model, and obtain the precision AP value of each category of target detection and the mean average precision MAP value through the MAP drawing program.

[0021] In summary, to solve the problem that the unmanned boat cannot navigate autonomously and intelligently due to insufficient light under dark conditions, the present invention proposes a method for fusing infrared shoreline segmentation and target detection. Specifically:

[0022] 1. The present invention uses infrared thermal imaging technology to solve the problems of target visualization and data acquisition for dark environments. Frame extraction, structural similarity deduplication, and manual screening are performed on the collected videos to build an original database. The annotation tool "Labelimg" is used for rectangular box annotation for the water target recognition task, and the annotation tool "Labelme" is used for polygon box annotation for the feasible region recognition task, thereby building an infrared shoreline segmentation dataset and an infrared target detection dataset.

[0023] 2. For the feasible region recognition task, the present invention trains the DeeplabV3 Plus network based on the infrared shoreline segmentation dataset, and uses the human-in-the-loop dataset and network optimization scheme to optimize the dataset and network weights, and finally obtains high-performance weights.

[0024] 3. For the water target recognition task, the present invention trains the YOLOv5m network based on an infrared target detection dataset, and optimizes the dataset and network weights using a human-in-the-loop dataset and a network optimization scheme, finally obtaining high-performance weights.

[0025] 4. Based on network training, the present invention obtains the water target detection weights and the feasible region recognition weights under the Pytorch framework network model. It uses a dual-thread architecture to perform online inference on the two networks for the same input image, conducts decision-level fusion on the inference results, and combines the target detection results and semantic segmentation results of water targets to achieve the positioning of water targets.

[0026] 5. The present invention converts the weight file of the Pytorch framework network model into the weight file of the TensorRT framework network model, migrates the weight file of the TensorRT framework network model to the edge computing platform, and realizes the recognition of water targets and feasible regions based on the edge computing platform. On the edge computing platform, the mAP of target detection of the present invention is not less than 92.65%, the mIOU of shoreline segmentation is not less than 74.15%, and the inference speed is above 20 FPS.

[0027] It can be seen that the present invention mainly first segments the shoreline based on the visual image processing technology of infrared thermal imaging, and simultaneously identifies the obstacles on the water surface. Then, the DeeplabV3 Plus model for infrared shoreline segmentation and the YOLOV5m model for target detection involved in this method are simultaneously deployed on the edge computing platform to achieve real-time recognition of water targets and feasible regions. This kind of system structure for deploying the general deep learning model effectively integrates the information between different stages and different participants based on the intelligent perception system of infrared light to achieve the recognition of water targets and feasible regions, with very good effects and greatly meeting the actual navigation needs of unmanned boats. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of the infrared shoreline segmentation and target detection fusion method for the unmanned boat on the water under dark conditions involved in the present invention;

[0029] Figure 2 is a graphical schematic diagram of the system involved in the infrared shoreline segmentation and target detection fusion method in the present invention;

[0030] Figure 3 is a graphical schematic diagram of the infrared shoreline segmentation and target detection fusion in the present invention;

[0031] Figure 4 is a schematic diagram of the experimental results in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] For the convenience of those skilled in the art, the present invention will be further described below in conjunction with embodiments and the accompanying drawings. The content mentioned in the embodiments does not limit the present invention.

[0033] In view of the two major challenges faced by unmanned boats during night navigation: first, the visible light perception for navigating the route of unmanned boats is basically ineffective at night, and second, the current perception system cannot identify obstacles during water navigation. The present invention takes the infrared light target recognition system as the core, makes full use of modern artificial intelligence algorithms, continuously iterates and optimizes the recognition performance of the weights of the deep convolutional network model on the premise of continuously expanding the data set, and then completes the real-time operation of the unmanned boat through application deployment. Accordingly, the present invention provides a method for fusing infrared shoreline segmentation and target detection of a surface unmanned boat under dark conditions, as Figure 1 and 2 shown, which specifically includes the following steps.

[0034] Step S1, data set establishment.

[0035] Mount various types of infrared thermal imagers on the unmanned boat, including the thermal imager of the DJI M300 drone, and make the infrared thermal imager take low-altitude videos in the water area scene to simulate the navigation perspective of the unmanned boat and shoot the water surface. Then, extract frames from the video, remove duplicates based on structural similarity, and perform manual screening to obtain the original images. Then, use the annotation tool "Labelimg" to perform rectangular box annotation on the water targets, and use the annotation tool "Labelme" to perform polygon box annotation on the feasible region to obtain the infrared shoreline segmentation data set and the infrared target detection data set. The infrared shoreline segmentation data set includes three categories: background, water, and obstacle; the infrared target detection data set includes five categories: boat, person_shore, person_boat, swimming, and dolphin1. Divide both the infrared shoreline segmentation data set and the infrared target detection data set into three sub-data sets: training set, validation set, and test set, according to the ratio of 8:1:1. The number of samples in each category is the same in each sub-data set.

[0036] It should be noted that the present invention is implemented using a deep learning solution, which requires a large amount of multi-scene live data as support. The water area scenes collected in this step should include various different water areas such as the open ocean, inland rivers, riverbanks, and lakes as experimental scenes. To improve the richness of the scenes, the collection time is mainly at night and early morning to enrich the infrared image data at different night times. A total of five videos at different angles are collected, and after frame extraction, duplicate removal, data cleaning, and annotation, the original data set is obtained.

[0037] In addition, in terms of data processing, the double histogram equalization and Gamma transformation methods are used to process images, improving the overall contrast of the images and enhancing details.

[0038] Furthermore, multiple thermal imagers are used to collect data to address the disadvantage of visible light perception for night-time boating. Currently, most research solutions for vision-based perception use visible light images, but in practical applications, the safety of night-time driving in intelligent transportation must inevitably be considered. Considering that the color adjustment schemes of different thermal imagers are different, the present invention uses different thermal imagers and collects data using two color adjustment methods, white heat and iron red, so that the network model weights can adapt to multiple thermal imagers.

[0039] Step S2, model establishment.

[0040] S21, The DeeplabV3 Plus model with resNet as the backbone feature extraction network is used as the network model for shoreline segmentation. The DeeplabV3 Plus model is compiled using the compilation function, and the hyperparameters of the DeeplabV3 Plus model are set. The DeeplabV3 Plus model is repeatedly trained and verified using the training set and validation set of the infrared shoreline segmentation dataset to obtain the training weights based on the infrared shoreline segmentation dataset. During training and validation, the encoder main body of the DeeplabV3 Plus model decomposes the standard convolution into depthwise convolution and pointwise convolution. The depthwise convolution uses spatial convolution independently for each channel, and the pointwise convolution is used to combine the output of the depthwise convolution. In the encoder, for the initially effective feature layer compressed four times, parallel dilated convolutions are used to extract features with different rates respectively, then concatenated and merged, and then 1×1 convolution is used to compress the features to obtain the feature map. In the decoder, the initially effective feature layer compressed twice is adjusted in channel number using 1×1 convolution, and then stacked with the upsampled result of the effective feature after dilated convolution output by the encoder. After the stacking is completed, two depthwise separable convolutions are performed to obtain the final effective feature layer, and then a 1×1 convolution is used to adjust the channels to Num_Classes, and finally upsampling is performed using resize so that the width and height of the final output layer are the same as those of the input image.

[0041] S22. The YOLOV5m model with darknet-53 as the backbone feature extraction network is used as the network model for object detection. The YOLOV5m model is compiled using a compilation function, the hyperparameters of the YOLOV5m model are set, and the YOLOV5m model weights based on the voc2012 dataset are used as pre-training weights for transfer learning. Subsequently, the YOLOV5m model is repeatedly trained and verified using the training set and validation set of the infrared object detection dataset to obtain the training weights based on the infrared object detection dataset. During training and verification, the YOLOV5m model extracts features from the input image through darknet53. A total of three feature layers are extracted in the feature utilization part. The three feature layers are located at different positions of the backbone part darknet53, namely the middle layer, the middle and lower layer, and the bottom layer. The shapes of the three feature layers are (52, 52, 256), (26, 26, 512), and (13, 13, 1024) respectively. The three feature layers are subjected to 5 convolutional processes. After the processing, a part is used to output the prediction results corresponding to the feature layer, and a part is used to perform deconvolution umSampling2d and then combine with other feature layers.

[0042] Preferably, the aforementioned darknet-53 is composed of the darknetConv2D and the residual network residual module. In the darknet53, the residual convolution first performs a 3*3 convolution with a stride of 2, then saves this convolution layer, then performs a 1*1 convolution and a 3*3 convolution, and adds this result to the layer as the final result. Then, a large number of residual skip connections are used for five downsamplings with a stride of 2 and a convolution kernel size of 3. The feature dimensions are 64, 128, 256, 512, and 1024 respectively. The average pooling layer and the fully connected layer are not used. L2 regularization is performed during each convolution, and BatchNormalization normalization and the LeakyReLU activation function are performed after the convolution. The LeakyReLU activation function is:

[0043]

[0044] It is worth mentioning that the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model in the present invention at least include the size (input_shape) of the image samples in the input dataset, the batch size (batch_size), the number of iterations (epochs), the learning rate (lr), and the number of classes (num_classes). In this embodiment, the set hyperparameters are as follows:

[0045] The size of the image samples in the dataset to be input: input_shape = 416*416*3;

[0046] Batch size: Freeze-batch_size = 8, UnFreeze-batch_size = 4; Usually 2^N, such as 32, 64, 128...;

[0047] Number of epochs: Freeze_epochs = 50, UnFreeze_epochs = 100;

[0048] Learning rate: Freeze_lr = 1e-3, UnFreeze_lr = 1e-4;

[0049] Number of classes: num_classes = 10.

[0050] In S23, the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model that have been repeatedly trained and verified are continuously optimized using the cross-entropy loss function and the Adam loss function optimizer to obtain the training weights based on the infrared shoreline segmentation dataset and the infrared target detection dataset. Among them, the cross-entropy loss function is a smooth function, and its essence is the application of cross-entropy in information theory to classification problems. Its formula is:

[0051]

[0052] The Adam loss function optimizer is an optimization method that calculates the adaptive learning rate for each parameter, that is, it stores the exponentially decaying average of the past squared gradients and also maintains the exponentially decaying average of the past gradients :

[0053] .

[0054] Among them, is the exponential moving average, is the squared gradient, is the gradient over the time step sequence.

[0055] If and are initialized as zero vectors, they will be biased towards 0, so bias correction is done by calculating the bias-corrected and to offset these biases:

[0056]

[0057] The gradient update rule is:

[0058]

[0059] In the above formula:

[0060] S24. Evaluate and predict the DeeplabV3 Plus model and the YOLOV5m model under the foregoing training weights.

[0061] 1) Evaluation: Screen the obtained training weights, and select the weights with the lowest total loss and val loss as the weights of the DeeplabV3 Plus model and the YOLOV5m model; respectively use the test sets of the infrared shoreline segmentation dataset and the infrared target detection dataset to test the DeeplabV3 Plus model and the YOLOV5m model, obtain the mIoU of the DeeplabV3Plus model and the mean average precision MAP value of the YOLOV5m model, and adjust the hyperparameters of the DeeplabV3Plus model and the YOLOV5m model according to the required values of the evaluation metrics, and retrain until the requirements are met. In order to check how many epochs of training have reached saturation and prevent overfitting (in this embodiment, after freezing the network parameters and training for 50 epochs, then unfreeze all parameters and train for 50 epochs), here, the tensorboard tool module under the tensorflow framework is also used to draw the train loss and val loss curves.

[0062] 2) Prediction: Use the validation set of the infrared shoreline segmentation dataset to test the DeeplabV3 Plus model to obtain a mask, and then calculate the IoU of each category to statistically obtain the mIoU; use the validation set of the infrared target detection dataset to test the YOLOV5m model, and obtain the precision AP value of each category of target detection and the mean average precision MAP value through the MAP drawing program. The purpose of this prediction is to verify the final effect of the method involved in the present invention. If the effect is not ideal, increase the training data volume or continue to adjust the hyperparameters of the model until the expected effect is achieved.

[0063] S25. Establish a Pytorch framework network model for unmanned boat infrared shoreline segmentation and target detection under dark conditions. The Pytorch framework network model includes the DeeplabV3 Plus model for infrared shoreline segmentation and the YOLOV5m model for target detection after optimizing the hyperparameters. The DeeplabV3 Plus model and the YOLOV5m model are fused at the decision level in a serial dual-threaded inference manner, as shown in Figure 3In order to enable the Pytorch framework network model to be put into online use on the unmanned boat, the present invention converts the weight file of the Pytorch framework network model into the weight file of the TensorRT framework network model and migrates it to the edge computing platform on the unmanned boat. Preferably, the edge computing platform is an NVIDIA embedded platform.

[0064] It should be noted that the present invention performs decision-level fusion on the DeeplabV3 Plus model and the YOLOv5m model to comprehensively ensure safe boating. Since the recognition of underwater obstacles is very challenging for the segmentation network, while the object detection network can still maintain good performance in complex environments, the present invention adopts the object detection scheme to assist in realizing obstacle recognition.

[0065] Step S3, model application: Transmit the infrared shoreline segmentation data and infrared object detection data obtained by processing the water area scene video captured in real time by the infrared thermal imager carried by the unmanned boat through the method involved in step S1 to the edge computing platform, and process them by the TensorRT framework network model to obtain the recognition results of water targets and the feasible region.

[0066] Next, in this embodiment, a comparison experiment of the shoreline segmentation network model was carried out. Three segmentation networks were tried, and a comparison experiment was given in terms of accuracy, speed, and deployment. The experimental results are shown in Table 1 below. The results show that the DeeplabV3 Plus model with ResNet as the backbone reaches 20 FPS in recognition speed, meets the actual navigation requirements, and can be deployed on the embedded platform of the unmanned boat.

[0067] Table 1

[0068] Network model mIoU FPS Deployment DeeplabV3(Resnet) 0.956802 20 Implement deployment DeeplabV3(Xception) 0.953353 5 Cannot be deployed U-Net 0.979853 12 Implement deployment

[0069] In addition, in this embodiment, a comparison experiment of the object detection network model was carried out. An infrared object detection data set was established, with a total of 4,186 pictures, which were divided into a training set and a test set according to a ratio of 8:2, obtaining 3,347 pictures in the training set and 839 pictures in the test set. The target number of iteration rounds was set to 400, and at the same time, the loss value of the training set was monitored and aborted when the change was extremely small for a long time to ensure that the network was fully trained to convergence. In this experiment, according to actual needs, the default parameters were optimized, as shown in Table 2 below.

[0070] Table 2

[0071] Parameter name Before optimization After optimization initial learning rate 0.01 0.00816 final OneCycleLR learning rate 0.2 0.25725 SGD momentum 0.937 0.98 warmup_bias_lr 0.1 0.11521 image HSV-Hue augmentation 0.015 0.01734 image HSV-Saturation augmentation 0.7 0.9 image HSV-Value augmentation 0.4 0.44829 box loss gain 0.05 0.03384 cls loss gain 0.5 0.6195 warmup_epochs 3.0 2.71044 warmup_momentum 0.8 0.66111

[0072] In this experiment, multiple network models such as SSD, YOLOv3, and YOLOv5 were practiced. The performance comparison of each model on the thermal image water target dataset is shown in Table 3 below. The recognition speed of the target detection network model is determined by FPS, and the recognition ability is determined by the mean average precision (mAP). Only when the IoU between the predicted box and the actual box is greater than 0.5 is it included in the calculation. Considering factors such as detection accuracy, recognition speed, training time, and weights, the results in Table 3 show that choosing YOLOv5m as the basic network can achieve fast inference of water targets and rapid deployment of the model. The final effect display of this experiment is as Figure 4 shown, Figure 4 The left figure in the middle shows the recognition result of the white-hot infrared image, and the right figure shows the recognition result of the iron-oxide red infrared image.

[0073] Table 3

[0074]

[0075] The above embodiments are preferred implementation solutions of the present invention. In addition, the present invention can also be implemented in other ways. Any obvious replacement without departing from the concept of the technical solution of the present invention is within the protection scope of the present invention.

[0076] In order to make it more convenient for those of ordinary skill in the art to understand the improvements of the present invention over the prior art, some drawings and descriptions of the present invention have been simplified. And for the sake of clarity, some other elements have also been omitted in this application document. Those of ordinary skill in the art should be aware that these omitted elements may also constitute the content of the present invention.

Claims

1. An infrared shoreline segmentation and target detection fusion method for unmanned surface vessels under dark conditions, characterized in that Including: Step S1, dataset establishment: The unmanned boat is equipped with multiple infrared thermal imagers, which are used to take low-altitude photos of the water area scene from the perspective of the unmanned boat's navigation, shoot videos of the water surface, process the videos to obtain original images, and use annotation tools to annotate the original image data to obtain an infrared shoreline segmentation dataset and an infrared target detection dataset. Both of these datasets are divided into a training set, a test set, and a validation set according to a predetermined ratio. Step S2, model establishment: The DeeplabV3 Plus model is used as the network model for shoreline segmentation, and the YOLOV5m model is used as the network model for target detection; set the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model, and use the YOLOV5m model weights based on the voc2012 dataset as the pre-training weights for transfer learning. Subsequently, use the training set of the infrared shoreline segmentation dataset to repeatedly train and validate the DeeplabV3 Plus model, and use the training set of the infrared target detection dataset to repeatedly train and validate the YOLOV5m model to obtain the training weights based on the infrared shoreline segmentation dataset and the infrared target detection dataset; then use the test set and validation set of the infrared shoreline segmentation dataset to evaluate and predict the DeeplabV3 Plus model under this training weight, and use the test set and validation set of the infrared target detection dataset to evaluate and predict the YOLOV5m model under this training weight. Continuously adjust the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model according to the evaluation and test results, and establish a Pytorch framework network model that cascades the optimized DeeplabV3 Plus model and YOLOV5m model at the decision level; convert the weight file of the Pytorch framework network model into the weight file of the TensorRT framework network model and transfer it to the edge computing platform on the unmanned boat. Step S3, model application: Transmit the infrared shoreline segmentation data and infrared target detection data obtained by processing the water area scene video taken in real time by the infrared thermal imager carried by the unmanned boat through the method involved in Step S1 to the edge computing platform, and process them by the TensorRT framework network model to obtain the recognition results of water targets and the feasible region.

2. The infrared shoreline segmentation and target detection fusion method for the unmanned surface vehicle under dark conditions according to claim 1, wherein: Step S1, dataset establishment: Mount the infrared thermal imager of the DJI M300 drone on the unmanned boat. The infrared thermal imager takes low-altitude photos in the water area scene, simulating the navigation perspective of the unmanned boat, and takes videos of the water surface. Then, the videos are frame-sampled, duplicate-removed, and screened to obtain original images. Then, the annotation tool "Labelimg" is used to perform rectangular box annotation on the water targets, and the annotation tool "Labelme" is used to perform polygon box annotation on the feasible region to obtain the infrared shoreline segmentation dataset and the infrared target detection dataset. The infrared shoreline segmentation dataset includes three categories: background, water, and obstacle; the infrared target detection dataset includes five categories: boat, person_shore, person_boat, swimming, and dolphin1. Both the infrared shoreline segmentation dataset and the infrared target detection dataset are divided into three sub-datasets: training set, validation set, and test set, according to the ratio of 8:1:

1. The number of samples in each category is the same in each sub-dataset.

3. The infrared shoreline segmentation and target detection fusion method for the unmanned surface vehicle under dark conditions according to claim 2, wherein: When the DeeplabV3 Plus model is used as the network model for shoreline segmentation: Resnet is used as the backbone feature extraction network. The encoder main part decomposes the standard convolution into depth convolution and pointwise convolution. The depth convolution independently uses spatial convolution for each channel, and the pointwise convolution is used to combine the output of the depth convolution; in the encoder, for the initially effective feature layer compressed four times, parallel dilated convolutions are used to extract features with different rates respectively, then concat is performed for merging, and then 1×1 convolution is used to compress the features to obtain the feature map; In the decoder, the initially effective feature layer compressed twice is adjusted in channel number using 1×1 convolution, and then stacked with the result of upsampling the effective feature after dilated convolution output by the encoder. After the stacking is completed, two depthwise separable convolutions are performed to obtain the final effective feature layer. Then, a 1×1 convolution is used to adjust the channels to Num_Classes, and finally resize is used for upsampling so that the width and height of the final output layer are the same as those of the input image.

4. The infrared shoreline segmentation and target detection fusion method for unmanned surface vessels under dark conditions according to claim 2 or 3, characterized in that: When the YOLOV5m model is used as the network model for target detection: Darknet-53 is used as the backbone feature extraction network. The input image is feature-extracted through darknet53. A total of three feature layers are extracted in the feature utilization part. The three feature layers are subjected to 5 convolutional processes. After the processing, part of them is used to output the prediction results corresponding to the feature layer, and part of them is used to perform deconvolution umSampling2d and then combine with other feature layers.

5. The infrared shoreline segmentation and target detection fusion method for the unmanned surface vehicle under dark conditions according to claim 4, characterized in that: The darknet-53 consists of darknetConv2D and residual network residual modules. In the darknet53, the residual convolution first performs a 3*3 convolution with a stride of 2, then saves this convolution layer, then performs a 1*1 convolution and a 3*3 convolution, and adds this result to the layer as the final result. Then, a large number of residual skip connections are used for five downsamplings with a stride of 2 and a convolution kernel size of 3. The feature dimensions are 64, 128, 256, 512, and 1024 respectively. The average pooling layer and the fully connected layer are not used. L2 regularization is performed during each convolution, and BatchNormalization normalization and the LeakyReLU activation function are performed after the convolution. The LeakyReLU activation function is: The feature utilization part of the YOLOV5m model extracts multi-feature layers for object detection. A total of three feature layers are extracted. The three feature layers are located at different positions of the backbone part darknet53, namely the middle layer, the middle-lower layer, and the bottom layer. The shapes of the three feature layers are (52, 52, 256), (26, 26, 512), and (13, 13, 1024) respectively. The three feature layers are subjected to 5 convolutional processes. After the processing, part of them is used to output the prediction results corresponding to the feature layer, and part of them is used to perform deconvolution UmSampling2d and then combine with other feature layers.

6. The infrared shoreline segmentation and target detection fusion method for the unmanned surface vehicle under dark conditions according to claim 5, characterized in that: The hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model include the size of the image samples in the input dataset, the batch size, the number of iterations, the learning rate, and the number of classes.

7. The infrared shoreline segmentation and target detection fusion method for the unmanned surface vehicle under dark conditions according to claim 6, wherein: In step S2, after repeatedly training and validating the DeeplabV3 Plus model and the YOLOV5m model, the cross-entropy loss function and the Adam loss function optimizer are used to continuously optimize the hyperparameters of the two models to obtain the training weights based on the infrared shoreline segmentation dataset and the infrared target detection dataset.

8. The infrared shoreline segmentation and target detection fusion method for the unmanned surface vehicle under dark conditions according to claim 7, characterized in that: In step S2, when evaluating and predicting the DeeplabV3 Plus model and the YOLOV5m model under the training weights: 1) Evaluation: Screen the obtained training weights and select the weights with the lowest total loss and val loss as the weights of the DeeplabV3 Plus model and the YOLOV5m model; use the test sets of the infrared shoreline segmentation dataset and the infrared target detection dataset to test the DeeplabV3 Plus model and the YOLOV5m model respectively, obtain the mIoU of the DeeplabV3 Plus model and the mean average precision MAP value of the YOLOV5m model, and adjust the hyperparameters of the DeeplabV3 Plus model and the YOLOV5m model according to the required values of the evaluation metrics and retrain until the requirements are met; at the same time, use the tensorboard tool module in the tensorflow framework to draw the train loss and val loss curves; 2) Prediction: Use the validation set of the infrared shoreline segmentation dataset to test the DeeplabV3 Plus model to obtain a mask, and then calculate the IoU of each category to calculate the mIoU; Use the test set of the infrared target detection dataset to test the YOLOV5m model, and obtain the precision AP value of each category of target detection and the mean average precision MAP value through the MAP drawing program.

Images