An unmanned boat obstacle recognition method based on image analysis and its model construction method

Through the unmanned boat obstacle recognition method based on image analysis, using sonar image data and dynamic pruning model training strategy, the problem of low obstacle detection efficiency in deep water environments is solved, and efficient and automated obstacle recognition is achieved. , improves detection accuracy and robustness.

CN118781478BActive Publication Date: 2024-11-12ZHEJIANG INTELLIGENT SHIP RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411267326.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-11
Publication Date
2024-11-12
Estimated Expiration
2044-09-11

AI Technical Summary

Technical Problem

The existing unmanned boat obstacle detection methods are inefficient and cannot be automated in deep water environments. The electromagnetic wave detection method has poor propagation capabilities in water. Visual detection methods rely on manpower and high hardware costs and cannot quickly avoid obstacles.

Method used

Using an unmanned boat obstacle recognition method based on image analysis, the model is constructed through sonar image data acquisition, preprocessing and building a model, including image chunking, feature extraction, attention mechanism and object detection module, combined with dynamic pruning model training strategy, lightweight construction Quantified underwater obstacle detection model.

Benefits of technology

It improves the automation and intelligence of underwater obstacle detection by unmanned boats, improves detection speed and accuracy, reduces energy consumption, and enhances robustness in harsh environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118781478B_ABST
    Figure CN118781478B_ABST
Patent Text Reader

Abstract

The present invention provides an unmanned boat obstacle recognition method based on image analysis and a model construction method thereof, which belong to the technical field of underwater image processing. The present invention collects obstacle images during underwater driving by a sonar image acquisition system deployed on the unmanned boat. After data preprocessing, an efficient feature extraction module based on multi-convolution kernel feature units, channel attention and multi-level feature fusion units is designed to extract features from the preprocessed obstacle images. Subsequently, based on the feature extraction module, a convolutional neural network detection model for underwater obstacles is constructed. In this underwater obstacle detection model, a method for block processing of the original image and an attention mechanism are used to further improve the degree of acquisition of image features, so as to improve the detection accuracy. At the same time, during the actual training of the model, a model training strategy of dynamic convolution layer pruning is adopted to further improve the calculation speed of the model and the degree of model lightweight.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of underwater image processing, and in particular relates to an unmanned boat obstacle recognition method based on image analysis and a model construction method thereof. Background Art

[0002] Unmanned boats are a key force in promoting the development of multiple fields such as marine resource exploration, environmental monitoring and protection. With its autonomous navigation, efficient operation and powerful information processing capabilities, unmanned boats have become an important carrier for performing high-risk tasks such as reconnaissance, anti-submarine, and mine hunting, greatly improving the flexibility and safety of maritime operations. In the civilian field, unmanned boats are widely used in marine resource exploration, and can go deep into unknown waters, accurately map the seabed topography and discover potential resources. In addition, unmanned boats also have multiple functions such as water rescue and scientific research investigation, providing more comprehensive and efficient marine services for human society. However, when unmanned boats are actually traveling underwater, there are various obstacles such as reefs, fish schools, and sunken ships underwater. These obstacles will affect the actual driving of unmanned boats. If these underwater obstacles cannot be avoided smoothly, the unmanned boats will be damaged, thereby affecting the normal mission of the unmanned boats. Therefore, improving the recognition ability of unmanned boats for underwater obstacles is of great significance to improving the safety performance of unmanned boats themselves and ensuring the normal implementation of various underwater tasks.

[0003] The existing unmanned boat obstacle detection methods mainly include the following two methods: electromagnetic wave detection method and visual detection method.

[0004] Electromagnetic wave detection method: When detecting obstacles, the unmanned boat first actively emits electromagnetic waves and detects the underwater environment. After encountering obstacles, the electromagnetic waves are reflected back and received by the equipment. Subsequently, through sophisticated data processing, including signal amplification, filtering and time difference calculation, the system can accurately determine the location and distance of the obstacle. Furthermore, by using the reflection characteristics of electromagnetic waves, the unmanned boat can identify the type, size and shape of the obstacle.

[0005] Visual inspection method: This method deploys image acquisition equipment around the unmanned boat and transmits the collected images of the waters around the unmanned boat to the video window in the information terminal through wireless transmission equipment. Relevant personnel conduct actual analysis of the actual waters around the unmanned boat transmitted to the terminal to determine the obstacles in the waters around the unmanned boat.

[0006] However, the visual detection method relies on wireless devices for data transmission when performing obstacle detection, which increases hardware costs. In addition, since the transmission of information takes a lot of time, it is impossible to give quick avoidance instructions to the unmanned boat. In addition, the visual detection method also has high requirements for human resources and cannot achieve automated obstacle detection. When the electromagnetic wave detection method is used for actual underwater obstacle detection, due to the poor propagation ability of electromagnetic waves in water, the emitted detection electromagnetic waves will be absorbed by water, so the electromagnetic wave detection method will be limited in deep water environments. Summary of the invention

[0007] In view of the above problems, the first aspect of the present invention provides a method for constructing an unmanned boat obstacle recognition model based on image analysis, comprising the following steps:

[0008] S1, collects sonar image data including underwater obstacles when the unmanned boat is traveling underwater, and constructs the original data set;

[0009] S2, preprocessing the collected underwater sonar image data to construct a standard data set for training the unmanned boat obstacle recognition model;

[0010] S3, constructing an underwater obstacle recognition model, wherein the model includes an image segmentation module, a feature extraction module, an attention mechanism module and a target detection module;

[0011] The image block division module adopts an image block division method with equal intervals to divide the image in the standard image data into a plurality of blocks of equal size;

[0012] The feature extraction module adopts a feature extraction unit with multiple convolution kernels to take into account both the global features and local detail features of the data, and fuses the features of different channels through a channel attention unit; at the same time, the feature extraction module designs a framework for multi-depth feature extraction and a multi-level feature fusion unit;

[0013] The attention mechanism module performs a feature extraction operation based on the divided image, and then performs a weighted fusion of the image features of different small blocks;

[0014] The target detection module performs target detection based on the fused image features and outputs the specific obstacle distribution in the waters around the unmanned boat;

[0015] S4, builds a model training strategy based on dynamic pruning, and trains the model through a standard data set to finally obtain a lightweight underwater obstacle detection model.

[0016] Preferably, the feature extraction module includes a multi-depth feature extraction unit and a multi-depth feature fusion unit;

[0017] The multi-depth feature extraction unit mainly includes a feature extraction unit FTC and a feature stratification unit FFC. The feature extraction unit FTC has three feature extraction paths with different convolution kernel sizes and a channel attention mechanism for fusing the output features of the three different paths, and uses convolution kernels with different sizes for feature extraction; the feature stratification unit FFC is composed of a 3*3 convolution layer and an average pooling layer connected together. In the multi-depth feature extraction unit, four FFC units are horizontally connected in sequence, and each FFC unit is connected to six FTC units connected in sequence, thereby forming a multi-depth feature extraction framework, which is used to increase the multi-scale perception capability of the unmanned boat;

[0018] The multi-depth feature fusion unit is implemented based on the spatial attention mechanism, which optimizes the fusion of features at different depth levels to enhance the environmental adaptability of the unmanned boat during obstacle detection, and adjusts the weights and thresholds of feature layers at different depths in dim light and bad weather.

[0019] Preferably, the feature extraction unit FTC is specifically:

[0020] Design a feature extraction network with three paths. Path 1: The input first passes through two convolution layers with a convolution kernel size of 3*3, then passes through a maximum pooling layer, and then passes through two convolution layers with a convolution kernel size of 3*3, and then passes through the Softmax activation function to obtain the output of path 1; Path 2: The input passes through two convolution layers with a convolution kernel size of 5*5, and then passes through a maximum pooling layer and a Softmax activation function to directly obtain the output of path 2; Path 3: The input is processed by a convolution layer with a convolution kernel size of 7*7, and then passes through the Softmax activation function to directly obtain the output of this path;

[0021] Then, the channel attention mechanism is used to assign attention weights to the output features of the three different channels, and the proportion of the output features of different channels is automatically selected; the larger convolution kernel channel extracts more prominent global features when performing feature extraction, while the smaller convolution kernel extracts more prominent detail features when performing feature extraction;

[0022] The three weighted path outputs are then passed through a fully connected layer and then through a Relu activation function to obtain the final output of this feature extraction unit FTC.

[0023] Preferably, the image segmentation module divides the image in the standard image data into 9 small blocks of equal size, and then uses the feature extraction module to perform feature extraction operations on each small portion of the image after segmentation, to obtain ;

[0024]

[0025] in, Represents the feature extraction module operation function, Represents the image after segmentation Small pieces of images, Indicates the extracted The feature matrix of the small patch image, .

[0026] Preferably, the specific processing process of the attention mechanism module is:

[0027] S11, feature matrix conversion: The obtained image feature matrix of each small block is sequentially converted to the linear transformation matrix Multiply to convert the image feature matrix of each small block Converted into a one-dimensional feature vector :

[0028]

[0029] in, Represents the one-dimensional feature vector corresponding to the i-th image block;

[0030] S12, attention weight calculation: Calculate the attention weight corresponding to each small part of the image. The calculation process is shown in the following formula:

[0031]

[0032] in, Indicates the calculated i The attention weight of a small image patch, Indicates The transpose of the one-dimensional feature vector corresponding to each image patch, represents the normalization function, which maps the attention weight value to the interval middle, ;

[0033] S13, weighted image feature calculation: using the calculated The attention weight corresponding to the small image and its image feature matrix Multiply them together to get the weighted image feature matrix ,and :

[0034]

[0035] S14, the weighted feature matrix of all the small blocks of images Combine them to get the weighted overall image feature matrix :

[0036]

[0037] in, Represents a combination operation function.

[0038] Preferably, the input of the target detection module is first a batch normalization BN layer, and then passes through three convolution layers with a convolution kernel size of 3*3, and then passes through an average pooling layer and a Relu activation function. The output of the Relu activation function at this time is added to the input of the target detection module again, and then the sum of the addition passes through two maximum pooling layers and the Relu activation function to obtain the final output of the model.

[0039] Preferably, the model training strategy of dynamic pruning is to delete the convolutional layer feature extraction path with low contribution to feature extraction during the model training process, including the following process:

[0040] S21, calculate the output feature vector of each convolutional layer feature extraction path in the feature extraction link in turn The mean of all elements in :

[0041]

[0042] in, is the output feature vector of the convolutional layer feature extraction pathway. element values, , Indicates that there are a total of element value;

[0043] S22, model output feature vector The mean calculation of :

[0044]

[0045] in, ,

[0046]

[0047] S23, feature correlation value calculation: sequentially calculate the output feature vectors of all convolutional layer feature extraction paths in the feature extraction link of this underwater obstacle detection model and the model feature output vector The characteristic correlation value of :

[0048]

[0049] Its characteristic correlation value , represents the output feature vector of each convolutional layer feature extraction pathway; in each training, the convolutional layer feature extraction pathway corresponding to the output features with feature correlation values ​​within the last 10% is deleted to obtain a new underwater obstacle detection model network structure;

[0050] S24, after obtaining the new obstacle detection model network structure, continue to train the model, delete a path every several trainings, and use the underwater obstacle detection model obtained last time as the final underwater obstacle detection model actually deployed.

[0051] The second aspect of the present invention provides an unmanned boat obstacle recognition method based on image analysis, which deploys the underwater obstacle detection model constructed by the construction method described in the first aspect to the unmanned boat terminal, and includes the following process:

[0052] Deploy high-precision sonar equipment around the unmanned boat to collect sonar images of the waters around the unmanned boat in real time;

[0053] Performing image filtering, image normalization and image standardization processing on the collected initial image;

[0054] The processed image is sent to the underwater obstacle detection model for detection to obtain the specific obstacle distribution in the waters around the unmanned boat;

[0055] Intelligent control and avoidance are carried out based on the detected obstacle conditions in the surrounding waters.

[0056] The third aspect of the present invention provides an unmanned boat obstacle identification device based on image analysis, the device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the underwater obstacle detection model constructed by the construction method described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor executes an unmanned boat obstacle identification method based on image analysis.

[0057] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program or instruction of the underwater obstacle detection model constructed by the construction method described in the first aspect. When the program or instruction is executed by a processor, the processor executes an unmanned boat obstacle recognition method based on image analysis.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. Improve the degree of automation and intelligence: This method can improve the degree of automation and intelligence of underwater obstacle detection by training an underwater obstacle detection model based on a convolutional neural network. At the same time, a feature extraction unit based on multiple convolution kernels is designed and a multi-level feature extraction architecture is used. When performing obstacle detection underwater, obstacles of different sizes can be accurately identified intelligently. At the same time, the model training strategy based on dynamic convolutional layer pruning can reduce the model's memory and improve the model's obstacle detection speed. This allows the unmanned boat to have a faster detection time for surrounding obstacles when the model is actually deployed on the unmanned boat, ultimately further improving the unmanned boat's perception of the surrounding environment and its intelligence level;

[0060] 2. Improve the lightweight of the model: A model training strategy based on dynamic pruning of convolutional layers is constructed. Unimportant convolutional layers in the model are deleted during model training, which can reduce the model's memory, improve the model's detection speed, and improve the model's lightweight. For unmanned boats, the energy consumed during obstacle detection will also be further reduced, which is conducive to improving the endurance of unmanned boats in underwater operations;

[0061] 3. Improve the accuracy of obstacle detection: Compared with extracting image features from the entire image, the present invention first divides the image into blocks when performing feature extraction operations, then extracts features from each small block of the image after the blocks, and then combines the extracted small block image features through the attention mechanism. Through such feature extraction operations, when the unmanned boat is performing underwater obstacle detection and model building, the ability of fine detection can be further improved due to the image block subdivision operation. And through the attention mechanism, the image features of different small blocks are effectively integrated, so that the unmanned boat's obstacle detection accuracy will be further improved, and good detection effects can be maintained for obstacles of different sizes;

[0062] 4. Improve the robustness of obstacle detection: In the present invention, a channel attention mechanism is added when constructing the feature extraction unit, which can better grasp the global and local features of the image, and a spatial attention mechanism is added to the feature fusion unit to obtain better fusion features to improve the robustness of obstacle detection. When the unmanned boat is actually driving, when the surrounding water conditions (turbidity, water flow velocity, lighting conditions) change, the unmanned boat can still maintain a good detection rate for obstacles. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 The figure is an overall logic block diagram of the unmanned boat obstacle identification method of the present invention.

[0064] Figure 2 This is a structural diagram of the underwater obstacle detection model of the present invention.

[0065] Figure 3 Schematic diagram of image segmentation according to the present invention.

[0066] Figure 4 Schematic diagram of the overall network structure of the feature extraction module of the present invention.

[0067] Figure 5 This is a schematic diagram of the FTC network structure of the feature extraction unit of the present invention.

[0068] Figure 6 Schematic diagram of the dynamic pathway pruning operation flow of the present invention.

[0069] Figure 7 This is an actual flow chart for the deployment and application of the model of the present invention.

[0070] Figure 8 The following is an example of a visual detection result.

[0071] Fig. 9 This is a schematic diagram of the structure of the unmanned boat obstacle identification equipment. DETAILED DESCRIPTION

[0072] The invention will be further described below in conjunction with specific embodiments.

[0073] The present invention collects sonar images of obstacles during underwater travel through a high-precision sonar system deployed on the unmanned boat. After data preprocessing, an efficient feature extraction module based on multi-convolution kernel feature units, channel attention and multi-level feature fusion units is designed to extract features from the preprocessed obstacle images. Subsequently, based on the feature extraction module, a convolutional neural network detection model for underwater obstacles is constructed. In this underwater obstacle detection model, the method of block processing of the original image and the attention mechanism are used to further improve the degree of acquisition of image features, so as to improve the detection accuracy. At the same time, during the actual training of the model, a model training strategy of dynamic convolutional layer pruning is adopted to further improve the calculation speed of the model and the lightweight degree of the model. Finally, the optimized model is deployed to the unmanned boat that is actually traveling, so as to realize real-time underwater obstacle detection and analysis of the unmanned boat and ensure navigation safety. The overall process is as follows: Figure 1 As shown:

[0074] Initial data set construction: Deploy high-precision sonar equipment on the surface of the unmanned boat to collect images containing underwater obstacles when the unmanned boat is traveling underwater. Then filter, normalize and standardize the collected underwater images containing obstacles, and finally construct a data set for training the underwater obstacle detection model.

[0075] Feature extraction module design: The feature extraction module is based on a feature extraction unit with multiple convolution kernels to take into account both the global features and local detail features of the data and fuse the features of different channels through the channel attention unit. At the same time, this feature extraction module designs a multi-depth feature extraction framework and a multi-level feature fusion unit to further enhance the efficiency of feature extraction. Through the multi-level feature fusion unit, the feature extraction module can take into account both detail features and global features to ensure that the unmanned boat can still maintain the ability to stably detect obstacles in a constantly changing and complex underwater environment, thereby enhancing the robustness of the unmanned boat in obstacle detection.

[0076] Construct an underwater obstacle detection model: Design an underwater obstacle detection model based on a convolutional neural network, including a feature extraction module, an attention mechanism module, and a target detection module. This detection model first divides the image into blocks when performing feature extraction, and then performs feature extraction on the divided image. Subsequently, the image features of different small areas are efficiently fused through the attention mechanism to further improve the detection accuracy of the model. By extracting features after dividing the image into blocks and then fusing them using the attention mechanism, a good detection rate is maintained for smaller obstacles during the actual driving and detection of the unmanned boat, ensuring the unmanned boat's comprehensive detection capabilities for obstacles of different sizes;

[0077] Constructing a model training strategy based on dynamic pruning: During actual model training, in order to further improve the model's calculation speed and the model's obstacle detection speed, the present invention constructs a model training strategy based on dynamic pruning, and finally obtains a lightweight underwater obstacle detection model; thus, when the unmanned boat is actually driving, the final lightweight underwater obstacle detection model is used to achieve faster calculation and processing, which can ensure a faster detection speed for underwater obstacles and further improve the driving safety of the unmanned boat;

[0078] Actual deployment and use of the model: When the unmanned boat is actually traveling underwater, high-precision sonar equipment is deployed to collect sonar images of the water conditions around the unmanned boat in real time, and the collected images are sent to the lightweight underwater obstacle detection model to analyze and judge the obstacles around the unmanned boat in real time.

[0079] 1. Construction of training dataset

[0080] Underwater obstacle image acquisition: As a data-driven technology, the present invention needs to obtain the surrounding obstacle images of the unmanned boat when it is actually traveling underwater for subsequent model training. The present invention uses multiple high-precision sonar devices deployed on the surface of the unmanned boat to collect sonar images of the surrounding waters to obtain sonar images containing underwater obstacles from multiple perspectives, and defines the collected images containing underwater obstacles as ;

[0081] Image filtering: During the underwater obstacle image acquisition process, the underwater obstacle images collected are often There is interference from noise factors in the image, and the presence of noise will be detrimental to the accurate recognition of underwater obstacles in the image. The present invention constructs an image filtering processing module, in which a Gaussian filter is used to filter the collected underwater obstacle image. Perform Gaussian filtering to remove the noise in the image and obtain the denoised image ;

[0082]

[0083] in, Represents Gaussian filter processing function;

[0084] Image normalization: Image processed by Gaussian filtering The noise information contained in the image will be greatly reduced, but The range of pixel values ​​in is very large, which causes the model to consume more resources during training and actual use. Therefore, in order to speed up the convergence of the model and reduce the memory consumed by the model, the present invention uses the maximum and minimum normalization method to denoise the image. Perform normalization operation, as shown in the following formula, to normalize all pixel values ​​in the image to The image after normalization operation has been obtained ;

[0085]

[0086] in, Representing images Middle Line The pixel value of the pixel point in the column, Represents the image after normalization Middle Line The pixel value of the pixel point in the column, Respectively represent images The minimum pixel value and the maximum pixel value among all pixels;

[0087] Image normalization processing method: In order to further improve the generalization effect after model training, the present invention performs normalization on the image after normalization operation. The standard image dataset has been obtained by standardization operation , the standardized formula is as shown in the formula:

[0088]

[0089] in, Represents the image after normalization operation Middle Line The pixel value of the pixel point in the column, Respectively represent the image The pixel average value of all pixels and the standard deviation calculated for all pixels;

[0090] Data calibration: based on the obtained standard image data , use the image annotation software LabelImg to annotate the image The obstacles in the image are marked with rectangular frames to obtain the position information of the underwater obstacles in the image, including the specific x, y coordinates and the length s and width h of the obstacles.

[0091] 2. Construction of underwater obstacle detection model

[0092] The underwater obstacle detection model constructed by the present invention is mainly composed of an image segmentation module, a feature extraction module, an attention mechanism and a target detection module. The overall model structure is as follows: Figure 2 As shown. In the underwater obstacle detection model constructed by the present invention, unlike the method of extracting features from the entire picture, a method of extracting features from blocks of the image is adopted. Subsequently, the features of different blocks of the image are combined using the attention mechanism module. When combining the features, not only the inherent features of the image pixels are considered, but also the position information of the image blocks is considered as feature information. Therefore, the understanding of the features of the image data can be further enhanced through this attention mechanism module to improve the accuracy of the unmanned boat's prediction of underwater obstacles. Finally, the target detection module is used to further process the output features of the attention mechanism module to obtain the final detection result.

[0093] 1. Image segmentation module

[0094] In the obstacle detection model designed by the present invention, different from the method of extracting features from the entire image, the method adopted by the present invention is to extract features from the image in blocks, and then combine features through the attention mechanism to obtain the overall features of the image. By first dividing the image into blocks and then integrating them, the unmanned boat's perception of the surrounding environment can be further enhanced, the unmanned boat's receptive field can be enhanced, and the unmanned boat's perception of the surrounding water conditions can be made more refined, so that the unmanned boat has the ability to detect obstacles when it is far away from the obstacles. Specifically, the image is first divided into blocks. In the present invention, an equally spaced image block method is selected to divide the image in the standard image data into 9 small blocks of equal size. The image block diagram is shown as follows. Figure 3 shown.

[0095] 2. Feature Extraction Module

[0096] The feature extraction module constructed by the present invention includes a multi-depth feature extraction unit and a multi-depth feature fusion unit. The overall structure view is as follows: Figure 4 As shown in the figure, the multi-depth feature extraction unit mainly includes a feature extraction unit (FTC) and a feature stratification unit (FFC). The feature extraction unit (FTC) has three feature extraction paths with different convolution kernel sizes and a channel attention mechanism for fusing the output features of the three different paths. The use of convolution kernels with different sizes for feature extraction can enhance the robustness of the unmanned boat in actual obstacle detection underwater. Facing different water conditions and obstacles of different sizes, this unmanned boat can maintain a good obstacle detection capability. The feature stratification unit (FFC) is composed of a 3*3 convolution layer and an average pooling layer. In the multi-depth feature extraction unit, four FFC units are horizontally connected in sequence. For each FFC unit, six FTC units are connected in sequence, thus forming a multi-depth feature extraction framework. This framework can increase the multi-scale perception capability of the unmanned boat, enhance the detection range of the unmanned boat obstacle detection, and improve the detection accuracy of long-distance obstacles and obstacles of small size. The multi-depth feature fusion unit is implemented based on the spatial attention mechanism, which optimizes the fusion of features at different depth levels to enhance the environmental adaptability of the unmanned boat during obstacle detection. In dim light and bad weather, the weights and thresholds of feature layers at different depths are adjusted to enhance the accuracy and robustness of obstacle detection of the unmanned boat in complex environments.

[0097] Feature extraction unit (FTC) design: The feature extraction unit (FTC) is the core unit in the feature extraction module constructed by the present invention. In FTC, a feature extraction network with three paths is designed. Path 1: The input first passes through two convolution layers with a convolution kernel size of 3*3, then passes through a maximum pooling layer, and then passes through two convolution layers with a convolution kernel size of 3*3, and then passes through the Softmax activation function to obtain the output of path 1 after normalization; Path 2: The input passes through two convolution layers with a convolution kernel size of 5*5, and then passes through a maximum pooling layer and a Softmax activation function to directly obtain the output of path 2; Path 3: After the input is processed by a convolution layer with a convolution kernel size of 7*7, it passes through the Softmax activation function to directly obtain the output of this path.

[0098] Then, the channel attention mechanism is used to assign attention weights to the output features of the three different channels, and the proportion of the output features of different channels can be automatically selected. The larger convolution kernel channel extracts more prominent global features when performing feature extraction, while the smaller convolution kernel extracts more prominent detail features when performing feature extraction. Using this channel attention mechanism, both global features and detail features can be optimally taken into account to achieve the best feature extraction effect.

[0099] Then, the three weighted path outputs are passed through a fully connected layer and then through a Relu activation function to obtain the final output of this feature extraction unit (FTC). The specific structure of FTC is as follows Figure 5 shown.

[0100] Feature Fractional Cell (FFC) Design: In the present invention, a feature fractional cell is constructed as one of the units of the multi-depth feature extraction unit. This FFC structure consists of a convolutional layer with a convolution kernel size of 3*3 connected to an average pooling layer;

[0101] Design of multi-depth feature extraction unit: A multi-depth feature extraction unit is constructed based on the designed feature extraction unit FTC and the designed feature stratification unit FFC. Its input features pass through the FFC unit in sequence, and each feature stratification unit is connected to 6 FTC units downward.

[0102] Design of multi-scale feature fusion unit: Based on the features of different depth levels obtained by the multi-depth feature extraction unit, the multi-scale feature fusion unit is used to fuse the feature information of different depth levels. The multi-scale feature fusion unit mainly includes a spatial attention mechanism unit, through which the features of different depth levels are automatically fused to achieve the optimal feature fusion effect.

[0103] Image block feature extraction: Based on the image block operation and the image feature extraction module, this feature extraction module is used in turn to perform feature extraction operations on each small part of the image after block division, and the

[0104]

[0105] in, represents the feature extraction module operation function described in the S2 process, Represents the image after segmentation Small pieces of images, Indicates the extracted The feature matrix of the small patch image, .

[0106] 3. Attention Mechanism Module

[0107] The model attention mechanism module design allows the unmanned boat to pay more attention to obstacle feature data information when performing obstacle detection, increase the accuracy of obstacle detection, and reduce the misjudgment rate of obstacle detection. The process includes:

[0108] Feature matrix conversion: The image feature matrix of each small block is converted into a linear transformation matrix in turn. Multiply to convert the image feature matrix of each small block Converted into a one-dimensional feature vector ;

[0109]

[0110] in, Represents the one-dimensional feature vector corresponding to the i-th image block;

[0111] Attention weight calculation: Calculate the attention weight corresponding to each small part of the image. The calculation process is shown in the following formula:

[0112]

[0113] in, represents the calculated attention weight of the i-th image patch, Indicates The transpose of the one-dimensional feature vector corresponding to each image patch, Represents the normalization operation function, which can map the attention weight value to the interval middle, ;

[0114] Weighted image feature calculation: Using the calculated The attention weight corresponding to the small image and its image feature matrix Multiply them together to get the weighted image feature matrix ,and :

[0115]

[0116] The weighted feature matrix of all the small blocks of images is obtained Combine them to get the weighted overall image feature matrix :

[0117]

[0118] in, Represents a combination operation function.

[0119] 4. Object Detection Module

[0120] The input of the target detection module is first a batch normalization (BN) layer, then three convolution layers with a convolution kernel size of 3*3, and then an average pooling layer and a Relu activation function. The output of the Relu activation function at this time is added to the input of the target detection module again, and then the sum is passed through two maximum pooling layers and the Relu activation function to get the final output of the model. Designing the target detection module in this way can further improve the detection effect of the model, and the multiple pooling layers in the module can further reduce the risk of overfitting of the model and reduce the complexity of model training. And because the real ocean environment is complex and changeable, using this target detection module can accurately identify obstacles in a complex and changeable ocean environment, improve the adaptability of unmanned boats to the underwater environment, and enhance the robustness of obstacle detection in complex waters.

[0121] 3. Model Training

[0122] For the trained underwater obstacle detection model, in order to further improve the lightweight degree of the model and improve the speed of the model in underwater obstacle detection, the present invention proposes a model training strategy of fusion path dynamic pruning. The specific process is as follows: Figure 6 shown.

[0123] S1, use the standard image data set to train the underwater obstacle detection model. Since there are a large number of different convolutional layer feature extraction pathways in the feature extraction link of the constructed underwater obstacle detection model. Therefore, during the model training process, deleting the convolutional layer feature extraction pathways that have a low contribution to feature extraction can ensure that the computing resources consumed by the model during actual detection are further reduced; so deploying the lightweight model to the unmanned boat can reduce its demand for the unmanned boat terminal memory and computing resources; ultimately, it ensures that without affecting the performance of the model, it can be effectively deployed under the limited back-end computing power of the unmanned ship and improve the actual detection speed of the model for obstacles.

[0124] S2, calculates the output feature vector of each convolutional layer feature extraction path in the feature extraction link in turn The mean of all elements in

[0125]

[0126] in, is the output feature vector of the convolutional layer feature extraction pathway. element values, , Indicates that there are a total of element value;

[0127] S3, model output feature vector The mean calculation of :

[0128]

[0129] in, ;

[0130]

[0131] S4, feature correlation value calculation: Calculate the output feature vectors of all convolutional layer feature extraction paths in the feature extraction link of this underwater obstacle detection model in turn and the model feature output vector The characteristic correlation value of .

[0132]

[0133] Its characteristic correlation value , represents the output feature vector of each convolutional layer feature extraction pathway And the final model output The larger the feature correlation value, the higher the correlation between this feature and the model output, that is, the more important this feature is. In each training, the convolutional layer feature extraction path corresponding to the output features with feature correlation values ​​within the last 10% is deleted to obtain a new underwater obstacle detection model network structure.

[0134] S5, after obtaining the new obstacle detection model network structure, continue to train the model, and perform a path deletion every 50 trainings (i.e., perform a cycle from step S2 to step S4 every 50 training iterations). The total number of training iterations in the present invention is 400, and the underwater obstacle detection model obtained at the last time is used as the underwater obstacle detection model that is finally actually deployed.

[0135] 4. Model Deployment and Application

[0136] The actual deployment and application flow chart of the underwater obstacle detection model is as follows: Figure 7 The optimized underwater obstacle detection model is deployed to the unmanned boat to help the unmanned boat realize intelligent surrounding obstacle detection. The specific implementation steps include:

[0137] Image acquisition: deploy high-precision sonar equipment around the unmanned boat to collect real-time images of the water conditions around the unmanned boat;

[0138] Image processing: The collected initial sonar image is sent to the image processing module (including image filtering module, image normalization module, and image standardization module) for processing to enhance the image quality and improve the detection accuracy of the subsequent model for underwater obstacles;

[0139] Underwater obstacle detection model deployment: The final optimized underwater obstacle detection model is deployed to the terminal of the unmanned boat, and the processed image is sent to the underwater obstacle detection model for detection to obtain the specific obstacle distribution in the waters around the unmanned boat;

[0140] Intelligent avoidance: Based on the detected obstacle conditions in the surrounding waters, the unmanned boat can carry out intelligent active avoidance based on the analysis of the detected obstacles to maintain the safety of the unmanned boat itself and ensure the normal implementation of the unmanned boat's underwater mission.

[0141] 5. Experimental Results

[0142] In order to verify the effectiveness of the method proposed in this invention, the following experiments were designed:

[0143] Calculate the intersection over union (IOU), which refers to the intersection over union of the marked area in the image and the area predicted by the model. The larger the value, the higher the model accuracy. The experimental verification data set uses public underwater sonar images, and the obstacles are steel pipes. The experimental comparison model uses the well-known target detection algorithms YOLOV8, SPP-net and Fast R-CNN. The experimental results are shown in Table 1. The visual detection results are shown in Figure 8 shown.

[0144] Table 1 Experimental results

[0145]

[0146] Analysis of Table 1 shows that the obstacle detection accuracy of the method proposed in the present invention is the highest. Compared with the second-ranked target detection method YOLOV8, the IOU is 0.67% higher, which verifies the effectiveness of the method of the present invention.

[0147] like Fig. 9As shown, the present invention also provides an unmanned boat obstacle recognition device based on image analysis, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; the memory stores a computer execution program; the memory stores a computer execution program of the underwater obstacle detection model constructed by the construction method as described above; when the processor executes the computer execution program stored in the memory, the processor can execute an unmanned boat obstacle recognition method based on image analysis. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0148] The device may be provided as a terminal, a server or other forms of device. In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0149] The present invention also provides a computer-readable storage medium, which stores a computer execution program of the underwater obstacle detection model constructed by the construction method as described above. When the computer execution program is executed by a processor, the processor can execute an unmanned boat obstacle recognition method based on image analysis.

[0150] Specifically, a system, device or equipment equipped with a readable storage medium may be provided, on which a software program code for implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system, device or equipment reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the above-mentioned embodiments, so the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0151] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0152] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for constructing an unmanned boat obstacle recognition model based on image analysis, characterized in that: The following steps are involved: S1, using sonar equipment to collect sonar image data containing underwater obstacles when the unmanned boat is traveling underwater, and constructing an original data set; S2, preprocessing the collected sonar image data to construct a standard data set for training the unmanned boat obstacle recognition model; S3, constructing an underwater obstacle recognition model, wherein the model includes an image segmentation module, a feature extraction module, an attention mechanism module and a target detection module; The image block division module adopts an image block division method with equal intervals to divide the image in the standard image data into a plurality of blocks of equal size; The feature extraction module adopts a feature extraction unit with multiple convolution kernels to take into account both the global features and local detail features of the data, and fuses the features of different channels through a channel attention unit; at the same time, the feature extraction module designs a framework for multi-depth feature extraction and a multi-level feature fusion unit; The attention mechanism module performs a feature extraction operation based on the divided image, and then performs a weighted fusion of the image features of different small blocks; The target detection module performs target detection based on the fused image features and outputs the specific obstacle distribution in the waters around the unmanned boat; S4, build a model training strategy based on dynamic pruning, and train the model through a standard data set to finally obtain a lightweight underwater obstacle detection model; The dynamic pruning model training strategy is to delete the convolutional layer feature extraction paths that have a low contribution to feature extraction during the model training process. The process includes: S21, calculate the output feature vector of each convolutional layer feature extraction path in the feature extraction link in turn The mean of all elements in : in, is the output feature vector of the convolutional layer feature extraction pathway. element values, , Indicates that there are a total of element value; S22, model output feature vector The mean calculation of : in, , S23, feature correlation value calculation: sequentially calculate the output feature vectors of all convolutional layer feature extraction paths in the feature extraction link of this underwater obstacle detection model and the model feature output vector The characteristic correlation value of : Its characteristic correlation value , represents the output feature vector of each convolutional layer feature extraction pathway; in each training, the convolutional layer feature extraction pathway corresponding to the output features with feature correlation values ​​within the last 10% is deleted to obtain a new underwater obstacle detection model network structure; S24, after obtaining the new obstacle detection model network structure, continue to train the model, delete a path every several trainings, and use the underwater obstacle detection model obtained last time as the final underwater obstacle detection model actually deployed.

2. The method for constructing an unmanned boat obstacle recognition model based on image analysis according to claim 1, characterized in that: The feature extraction module includes a multi-depth feature extraction unit and a multi-depth feature fusion unit; The multi-depth feature extraction unit mainly includes a feature extraction unit FTC and a feature stratification unit FFC. The feature extraction unit FTC has three feature extraction paths with different convolution kernel sizes and a channel attention mechanism for fusing the output features of the three different paths, and uses convolution kernels with different sizes for feature extraction; the feature stratification unit FFC is composed of a 3*3 convolution layer and an average pooling layer connected together. In the multi-depth feature extraction unit, four FFC units are horizontally connected in sequence, and each FFC unit is connected to six FTC units connected in sequence, thereby forming a multi-depth feature extraction framework, which is used to increase the multi-scale perception capability of the unmanned boat; The multi-depth feature fusion unit is implemented based on the spatial attention mechanism, which optimizes the fusion of features at different depth levels to enhance the environmental adaptability of the unmanned boat during obstacle detection, and adjusts the weights and thresholds of feature layers at different depths in dim light and bad weather.

3. The method for constructing an unmanned boat obstacle recognition model based on image analysis as claimed in claim 2, characterized in that: The feature extraction unit FTC is specifically: Design a feature extraction network with three paths. Path 1: The input first passes through two convolution layers with a convolution kernel size of 3*3, then passes through a maximum pooling layer, and then passes through two convolution layers with a convolution kernel size of 3*3, and then passes through the Softmax activation function to obtain the output of path 1; Path 2: The input passes through two convolution layers with a convolution kernel size of 5*5, and then passes through a maximum pooling layer and a Softmax activation function to directly obtain the output of path 2; Path 3: The input is processed by a convolution layer with a convolution kernel size of 7*7, and then passes through the Softmax activation function to directly obtain the output of this path; Then, the channel attention mechanism is used to assign attention weights to the output features of the three different channels, and the proportion of the output features of different channels is automatically selected; the larger convolution kernel channel extracts more prominent global features when performing feature extraction, while the smaller convolution kernel extracts more prominent detail features when performing feature extraction; The three weighted path outputs are then passed through a fully connected layer and then through a Relu activation function to obtain the final output of this feature extraction unit FTC.

4. The method for constructing an unmanned boat obstacle recognition model based on image analysis according to claim 1, characterized in that: The image segmentation module divides the sonar image in the standard image data into 9 small blocks of equal size, and then uses the feature extraction module to perform feature extraction operations on each small part of the image after segmentation. ; in, Represents the feature extraction module operation function, Represents the image after segmentation Small pieces of images, Indicates the extracted The feature matrix of the small patch image, .

5. The method for constructing an unmanned boat obstacle recognition model based on image analysis according to claim 1, characterized in that: The specific processing process of the attention mechanism module is as follows: S11, feature matrix conversion: The obtained image feature matrix of each small block is sequentially converted to the linear transformation matrix Multiply to convert the image feature matrix of each small block Converted into a one-dimensional feature vector : in, Represents the one-dimensional feature vector corresponding to the i-th image block; S12, attention weight calculation: Calculate the attention weight corresponding to each small part of the image. The calculation process is shown in the following formula: in, Indicates the calculated i The attention weight of a small image patch, Indicates The transpose of the one-dimensional feature vector corresponding to each image patch, represents the normalization function, which maps the attention weight value to the interval middle, ; S13, weighted image feature calculation: using the calculated The attention weight corresponding to the small image and its image feature matrix Multiply them together to get the weighted image feature matrix ,and : S14, the weighted feature matrix of all the small blocks of images Combine them to get the weighted overall image feature matrix : in, Represents a combination operation function.

6. The method for constructing an unmanned boat obstacle recognition model based on image analysis according to claim 1, characterized in that: The input of the target detection module is first a batch normalization BN layer, then passes through three convolution layers with a convolution kernel size of 3*3, and then passes through an average pooling layer and a Relu activation function. The output of the Relu activation function at this time is added to the input of the target detection module again, and then the sum of the addition passes through two maximum pooling layers and the Relu activation function to obtain the final output of the model.

7. An unmanned boat obstacle recognition method based on image analysis, characterized in that: The underwater obstacle detection model constructed by the construction method according to any one of claims 1 to 6 is deployed to the unmanned boat terminal, and includes the following processes: Deploy a high-precision sonar system around the unmanned boat to collect sonar images of the waters around the unmanned boat in real time; Perform image filtering, image normalization and image standardization on the collected initial sonar image; The processed sonar image is sent to the underwater obstacle detection model for detection to obtain the specific obstacle distribution in the waters around the unmanned boat; Intelligent control and avoidance are carried out based on the detected obstacle conditions in the surrounding waters.

8. An unmanned boat obstacle recognition device based on image analysis, characterized in that: The device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the underwater obstacle detection model constructed by the construction method according to any one of claims 1 to 6; when the processor executes the computer execution program stored in the memory, the processor executes an unmanned boat obstacle recognition method based on image analysis.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction of the underwater obstacle detection model constructed by the construction method according to any one of claims 1 to 6. When the program or instruction is executed by the processor, the processor executes an unmanned boat obstacle recognition method based on image analysis.