A sea rescue method based on improved YOLOV4 target detection

Abstract

This invention discloses a maritime rescue method based on improved YOLOv4 target detection, belonging to the field of maritime rescue. It includes: an improved pixel-by-pixel filtering image preprocessing method (de-raining, de-fogging) for processing images captured by a high-definition camera mounted on a drone, converting blurry images into clear images to provide effective data for subsequent image detection; an improved YOLOv4 target detection algorithm for accurately identifying people in the water; and the use of a Feature Pyramid Transformer (FPT) to enhance the semantic extraction capability of images of small floating objects (people awaiting rescue), thereby improving the search and rescue success rate. Adding a Feature Pyramid Transformer (FPT) after the traditional feature pyramid structure enhances the feature map of semantic information for small targets, which is beneficial for detecting small objects at sea.

Description

Technical Field

[0001] This invention discloses a maritime rescue method based on improved YOLOv4 target detection, belonging to the field of maritime rescue. Background Technology

[0002] With the development of science and technology, the strengthening of national economies, the proposal of my country's marine economic plan, and the expansion of military maritime activities, maritime and air activities such as marine fishing, marine transportation, and military patrols have increased. These activities inevitably lead to various accidents. In maritime rescue missions, due to the uncertainty of the location of those being searched, limited search and rescue methods, and simple life-saving equipment, a large number of vessels are needed to search and rescue target areas, especially when those being rescued are scattered. When manned vessels are used for maritime rescue, the cost is high, and the information obtained by all rescue vessels cannot be effectively utilized, resulting in low rescue efficiency. Drones (UAVs) are a new method for rescuing people who have fallen into the water by using autonomous drone operations. However, when drones cruise and photograph objects on the sea surface, it is difficult to obtain a large number of clear and accurate images, and people in the water appear small in the captured images, making it difficult for traditional target detection algorithms to quickly detect them. Summary of the Invention

[0003] To overcome the shortcomings of the prior art, this invention provides a maritime rescue method based on improved YOLOv4 target detection, which employs an image preprocessing method (de-raining and de-fogging) based on improved pixel-by-pixel filtering and an improved YOLOv4 target detection algorithm.

[0004] A maritime rescue method based on improved YOLOv4 target detection includes the following steps:

[0005] Step S1: Preprocess the images captured by the drone using image filtering methods, processing the input rainy day image I through pixel-level filters. r ∈R H×W ;

[0006]

[0007] Among them, I r ∈R H×W This represents the final de-raining and de-hazing image. This indicates a pixel-by-pixel filtering operation, where each pixel is processed by its own unique convolutional kernel. Including the kernel of all pixels;

[0008] Step S2: After obtaining the preprocessed image, perform appropriate cropping transformation on the image and convert it into tensor format before feeding it into the deep learning network;

[0009] Step S3: After obtaining feature maps of different sizes, new feature maps are generated by capturing contextual information using three improved transformer-based operations: query, key, and value. These new feature maps encode nonlocal interactions across space and scale. The inputs to the feature maps are q... i ,k j ,v j , where i represents any space or scale of any size and position on the feature map, and j is any other space or scale independent of i.

[0010] Preferably, step S1 of the present invention specifically includes the following steps:

[0011] Specifically, when for the p-th I r Each pixel undergoes a unique convolution kernel during dehazing and deraining operations, which is the vector at the p-th position of K. This vector is defined as K. p ∈R K×K Using p as a 2D coordinate of a pixel, predict the deraining and dehazing pixels in the following way.

[0012]

[0013] Where the range of t is from arrive However, due to the issue of pixel-wise convolution kernel size, it is difficult to achieve satisfactory results for images with large rain streaks when using a convolution kernel that predicts parameters for three channels per pixel. To address this problem, this method allows the neural network to predict multi-scale dilated convolution kernels, and then the results are weighted after dilated convolution to obtain the final de-raining and de-fogging images.

[0014] Preferably, step S2 of the present invention specifically includes the following steps: the backbone network performs five convolutional residual operations on the input image to compress the feature size and expand the number of channels to obtain higher semantic information; after max pooling, the last feature layer is upsampled and superimposed with a tensor of the same size as the previous layer, and then upsampled again and superimposed with the tensor of the previous layer to fully reflect the bottom layer information; convolution is performed on the processed tensor to reduce the number of tensor channels and reduce redundant information; the superimposed tensor is downsampled and superimposed with a tensor of the same size, so that tensors of different sizes can obtain top-level information, thereby completing the construction of the feature map.

[0015] Preferably, step S3 of the present invention specifically includes the following steps:

[0016] The following three operation methods are adopted, with the query of itself, higher level, and lower level respectively, so that each level is a feature map that interacts across space and scale.

[0017] To address the issue of image quality degradation caused by complex environmental factors such as rain and fog in images captured by maritime drones, this invention proposes an improved pixel-by-pixel filtering method for de-raining and de-fogging. This method treats the de-raining and de-fogging problem as a pixel-by-pixel filtering problem. Since the filtering operation is highly optimized, the image quality recovery speed is very fast. To address the lack of a large number of paired datasets with and without rain and fog in traditional neural network training, this method employs a rain and fog data augmentation algorithm to artificially generate a large number of paired rain and fog datasets, providing reliable dataset support for subsequent neural network training.

[0018] Based on the features of floating objects images captured by UAVs in complex marine environments, this paper proposes a visual detection model for floating objects (people in need of rescue) at sea, based on cross-scale feature interaction and a parallel residual network. This model uses a parallel residual network to improve image feature extraction capabilities and reduce false detections caused by variations in lighting and shape of floating objects (people in need of rescue). It employs a Feature Pyramid Transformer (FPT) to enhance the semantic extraction capability for smaller floating objects (people in need of rescue). Furthermore, this invention embeds the FPT network into the YOLOv4 object detection algorithm model after the traditional feature pyramid structure, effectively improving the detection performance of small objects. Attached Figure Description

[0019] Figure 1 A diagram of the image preprocessing structure for pixel-by-pixel filtering;

[0020] Figure 2 A flowchart for improved maritime rescue methods. Detailed Implementation

[0021] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be understood that after reading the teachings of this invention, those skilled in the art can make various alterations or modifications to the invention, and these equivalent forms also fall within the scope defined by the appended claims.

[0022] like Figure 2 As shown, this invention provides a maritime rescue method based on improved pixel-by-pixel filtering image preprocessing (rain and fog removal) and improved YOLOv4 target detection. The improved pixel-by-pixel filtering image preprocessing (rain and fog removal) method is used to process images captured by a high-definition camera mounted on a drone, converting blurry images into clear images to provide effective data for subsequent image detection. The improved YOLOv4 target detection algorithm is used to accurately identify people who have fallen into the water, thereby facilitating search and rescue.

[0023] To address the issue of image quality being affected by environmental factors such as rain and fog when images are taken by drones in harsh marine climates, this invention proposes an image de-raining and de-fogging method based on pixel-by-pixel filtering. In harsh marine climates, rain can cause effects such as occlusion, fog, and motion blur. Figure 1 As shown, this invention uses image filtering methods to preprocess images captured by a drone, employing pixel-level filters to process the input rainy day image I. r ∈R H×W .

[0024]

[0025] Among them, I r ∈R H×W This represents the final de-raining and de-hazing image. This indicates a pixel-by-pixel filtering operation, where each pixel is processed by its own unique convolutional kernel. This includes the kernel for all pixels. Specifically, when dealing with the p-th I... r Each pixel undergoes a unique convolution kernel during dehazing and deraining operations, which is the vector at the p-th position of K. This vector is defined as K. p ∈R K×K This invention uses p as a 2D coordinate of a pixel, and can predict de-raining and de-hazing pixels in the following way.

[0026]

[0027] Where the range of t is from arrive However, due to the issue of pixel-wise convolution kernel size, it is difficult to achieve satisfactory results for images with large rain streaks when using a convolution kernel that predicts parameters for three channels for each pixel. To address this problem, this invention allows the neural network to predict multi-scale dilated convolution kernels, and then the results are weighted after dilated convolution to obtain the final de-raining and de-fogging image.

[0028] Due to the limitations of traditional pixel-wise convolution kernel size, it is difficult to achieve satisfactory detection results if there are large rain streaks in the image. To address this issue, this invention allows the neural network to predict multi-scale dilated convolution kernels, and then the results are weighted after dilated convolution to obtain the final de-raining and de-fogging image.

[0029] To improve the accuracy of UAV visual recognition of floating objects at sea, this invention proposes to employ a target detection model based on cross-scale feature interaction and parallel residual networks for visual detection of floating objects on the sea surface. After obtaining the preprocessed image, the image is appropriately cropped and transformed, and then converted into tensor format before being fed into the deep learning network. The backbone network performs five convolutional residual operations on the input image to compress the feature size and expand the number of channels, thereby obtaining higher semantic information. After max pooling, the last feature layer is upsampled and superimposed with a tensor of the same size as the previous layer. This upsampling is repeated before superimposing with the tensor of the previous layer to fully represent the lower-level information. Convolution operations are then applied to the processed tensors to reduce the number of tensor channels and decrease redundant information. The superimposed tensors are downsampled and superimposed with tensors of the same size, allowing tensors of different sizes to acquire top-level information, thus completing the construction of the feature map.

[0030] After obtaining feature maps of different sizes, three improved transformer-based operations are used to capture contextual information through query, key, and value operations, generating new feature maps that encode nonlocal interactions across space and scale. The inputs to the feature maps are q. i ,k j ,v j , where i represents any space or scale of any size and position on the feature map, and j is any other space or scale independent of i.

[0031] Feature interaction can effectively improve the semantic information of small targets, thereby improving detection accuracy. Therefore, increasing the frequency of feature interaction has a positive effect on the detection of floating objects (people awaiting rescue) at sea. This invention proposes three operation modes, using itself, higher levels, and lower levels as queries for other levels, respectively. This makes each level a feature map that interacts across space and scale.

[0032] Self-Transformer captures common object features on the same feature map, representing a non-local interaction; the output feature map has the same scale as its input feature map. Grounding Transformer is a top-down non-local interaction, connecting "concepts" in high-level feature maps with "pixels" in lower-level feature maps. The output feature map has the same scale as the lower-level feature maps.

[0033] The Rendering Transformer works in a bottom-up manner, aiming to render high-level "concepts" by merging visual attributes into low-level "pixels." This local interaction is based on the fact that rendering an "object" with features or attributes from another distant object is meaningless; it's done not by pixels but by the entire feature map. The feature map is then reordered by size and merged with the original pyramid features. Three convolutional operations reduce the number of tensor channels, resulting in a feature map that enhances the semantic information of small targets. This allows for efficient and rapid detection of the location of people who have fallen into the water.

[0034] This invention employs a target detection model based on cross-scale feature interaction and a parallel residual network for visual detection of floating objects on the sea surface. The cross-scale feature interaction scheme can effectively solve the problem of missed detection due to the small size of people in need of rescue on the sea surface. The parallel residual network can effectively extract the low-level features of floating object images taken by UAVs, and the loss function based on the intersection-union ratio of the predicted box and the target box can effectively improve the training speed of the convolutional neural network.

Claims

1. A maritime rescue method based on improved YOLOv4 target detection, characterized in that... Includes the following steps: Step S1: Preprocess the images captured by the drone using image filtering methods, processing the input rainy day images through pixel-level filters. ; ； in, This represents the final de-raining and de-hazing image. This indicates a pixel-by-pixel filtering operation, where each pixel is processed by its own unique convolutional kernel. Including the kernel of all pixels; Specifically, the steps include the following: Specifically, when for the first indivual Each pixel has a unique convolution kernel when performing rain and haze removal operations, that is, in The A vector at positions, which is defined as ;use As a 2D coordinate of a pixel, predict the deraining and dehazing pixel point in the following way. ; Where the range of t is from arrive However, due to the issue of pixel-wise convolution kernel size, it is difficult to achieve satisfactory results for images with large rain streaks when using a convolution kernel that predicts parameters for three channels for each pixel. To address this issue, this method allows the neural network to predict multi-scale dilated convolution kernels, and then the results are weighted after dilated convolution to obtain the final de-raining and de-fogging images. Step S2: After obtaining the preprocessed image, perform appropriate cropping transformation on the image and convert it into tensor format before feeding it into the deep learning network; Specifically, the process includes the following steps: The backbone network performs five convolutional residual operations on the input image to compress the feature size and expand the number of channels, thereby obtaining higher semantic information; after max pooling, the last feature layer is upsampled and superimposed with a tensor of the same size as the previous layer, and then upsampled again and superimposed with the tensor of the previous layer to fully reflect the bottom layer information; convolution is performed on the processed tensor to reduce the number of tensor channels and reduce redundant information; the superimposed tensor is downsampled and superimposed with a tensor of the same size, so that tensors of different sizes can obtain top-level information, thereby completing the construction of the feature map; Step S3: After obtaining feature maps of different sizes, new feature maps are generated by capturing contextual information using three improved transformer-based operations: query, key, and value. These new feature maps encode nonlocal interactions across space and scale. The inputs to the feature maps are... , Represents a space or scale of arbitrary size and position on the feature map. Is with Any other space or scale that is irrelevant; Specifically, the steps include the following: The following three operation methods are adopted, with the query of itself, higher level, and lower level respectively, so that each level is a feature map that interacts across space and scale.

Images