Underwater target identification method for underwater intelligent robot

Abstract

The invention discloses an underwater target identification method for an underwater intelligent robot, which comprises the following steps: firstly, enhancing the definition and color restoration effect of an underwater image through image enhancement processing, and then utilizing an underwater target identification network model which is designed in a targeted manner and is improved based on a feature fusion attention network framework to identify the underwater target. A new defogging target detection model is obtained, a clear fog-free image is output in an end-to-end mode, and recognition of an underwater target is achieved; according to the method, a good foundation is laid for the underwater intelligent robot to carry out underwater tasks, and an underwater target recognition network model improved based on a feature fusion attention network framework is deployed in a detection module of the underwater intelligent robot, so that the feasibility of deploying an underwater target recognition algorithm in mobile terminals such as the underwater intelligent robot is improved; the accuracy and real-time performance of underwater target recognition in an underwater intelligent robot are improved, and the universality and practicability of an underwater target recognition algorithm based on a neural network in mobile terminal equipment are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater target detection, and in particular to an underwater target recognition method for an underwater intelligent robot. Background Art

[0002] Underwater target recognition is one of the most important tasks of a dedicated underwater robot. Vision is an effective way to find the target. Considering the complexity of the underwater environment, the scattering effect of water on light, the refraction of the water body itself, the interference caused by suspended matter in the water, the convergence of sunlight by underwater ripples, and the garbage on the water, the target image to be detected will be affected by atomization, color distortion, contrast reduction, blurred image details, backscattering noise, etc. Underwater target recognition needs to solve a series of problems such as color distortion, contrast reduction, blurred image details, suspended matter interference, and spot effect. Therefore, processing the underwater environment image is a necessary preparation for completing various underwater tasks.

[0003] Nowadays, most object recognition algorithms based on convolutional neural networks are deployed on PCs or servers, and edge devices are mainly responsible for image acquisition and preprocessing. On some edge devices, most image detection technology applications are based on traditional object detection algorithms, which are far less robust and versatile than object recognition algorithms based on convolutional neural networks. Image detection and deep learning technologies based on Linux systems are still in the development stage, and there is no complete solution for cross-X86 platform porting and application.

[0004] Deep learning usually uses a graphics processing unit (GPU) to deploy a network model for learning to achieve better recognition results. This method has high requirements for computing power and consumes a lot of power. So far, compared with traditional algorithms, target recognition algorithms based on convolutional neural networks often have higher accuracy and better versatility. However, whether in the training stage or the inference stage, the algorithm has high requirements for the computing power of the platform. In order to finally implement the algorithm, it often costs a lot, which also limits the development and innovation of target recognition algorithms in the civilian field.

[0005] With the continuous development of deep neural networks, network models tend to be deeper and more complex. While the accuracy of target detection is improved, the network model will also be affected by the large amount of model calculation and slow operation speed. Underwater intelligent robots that perform underwater target recognition are usually designed based on lightweight Linux systems on mobile terminals to perform target detection tasks. In order to deploy and apply target detection networks based on deep learning on mobile devices such as underwater intelligent robots and meet the requirements of real-time reasoning, further research on underwater target recognition algorithms is needed. In recent years, most cameras perform target detection tasks based on target detection algorithms based on traditional image detection. This method has poor robustness and is easily affected by environmental interference. In order to study the feasibility of deploying target recognition algorithms on mobile terminals, improve the accuracy and real-time performance of target recognition on mobile terminals, and improve the versatility and practicality of target recognition algorithms based on neural networks, an underwater target recognition method based on image detection and deep learning technology based on lightweight systems such as Linux is proposed, which plays a key role in popularizing target recognition algorithms based on neural networks and reducing hardware deployment costs.

[0006] Therefore, how to improve the clarity of underwater images, output fog-free images, and deploy underwater target recognition methods based on neural networks based on lightweight system image detection and deep learning technology has become a problem that needs further research and solution. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the present invention provides an underwater target recognition method for an underwater intelligent robot, which can be deployed in a lightweight system of an underwater intelligent robot to improve the accuracy and real-time performance of the underwater intelligent robot in performing underwater target recognition tasks.

[0008] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0009] An underwater target recognition method for an underwater intelligent robot, the method comprising:

[0010] S1. Obtain underwater sample images that have been labeled with target categories to form an underwater sample image dataset;

[0011] S2. Based on the feature fusion attention network architecture, an underwater target recognition network model for target recognition in underwater images is constructed;

[0012] S3, using the underwater sample image in the underwater sample image data set as the input image, performing enhancement processing on the input image, and inputting it into the underwater target recognition network model for training to obtain a trained underwater target recognition network model;

[0013] S4. Deploy the trained underwater target recognition network model to the detection module of the underwater intelligent robot, and establish communication between the image acquisition module of the underwater intelligent robot and the detection module. Use the underwater image acquired by the image acquisition module of the underwater intelligent robot as the input image, and after enhancement processing, input it into the underwater target recognition network model deployed in the detection module for recognition and detection to obtain the underwater target recognition result.

[0014] As a preferred solution, before the input image is input into the underwater target recognition network model, the input image is first enhanced using a multi-scale enhancement algorithm with color restoration, wherein each channel of the input original image is enhanced as follows:

[0015]

[0016] Among them, S(x,y) represents the original image before enhancement; Represents the enhanced output of the i-th color channel of the original image; C i (x, y) represents the color restoration factor of the i-th color channel of the original image; B(x, y) represents the bilateral filtered image of the original image; G and b are the gain value and offset value of the multi-scale enhancement algorithm with color restoration, respectively; * represents the convolution operation; ω k represents the enhancement weight of the kth scale in the multi-scale enhancement algorithm with color restoration, and K represents the number of scales included in the multi-scale enhancement algorithm with color restoration.

[0017] Specifically, the color restoration factor C of the i-th color channel of the original image is i (x,y) is calculated as follows:

[0018]

[0019] In the formula, I j (x,y) represents the pixel value of the original image in the i-th color channel, represents the sum of the pixel values ​​of the three color channels of the original image at each pixel position; α and β are color restoration adjustment parameters, which are used to control the intensity and influence of color restoration respectively.

[0020] Specifically, the bilateral filtered image B(x,y) of the original image is calculated as follows:

[0021]

[0022] Where B(x, y) represents the pixel value after filtering; I(p) and I(q) represent the pixel value at the original image at the pixel coordinate p=(x p ,y p ) and q=(x q,y q ), S represents the pixel coordinate domain of the original image; G d (||pq||) represents the spatial kernel function, which is used to measure p=(x p ,y p ),q=(x q ,y q ) The pixel space distance between two pixel coordinates; G r (|I(p)-I(q)|) represents the pixel value kernel function, which is used to measure the pixel value difference between I(p) and I(q); W p Represents the normalized weight coefficient of bilateral filtering.

[0023] Specifically, the normalized weight coefficient W of the bilateral filter is p It is calculated as follows:

[0024] W p =∑ q∈S G d (||pq||)G r (|I(p)-I(q)|);

[0025]

[0026]

[0027] In the formula, ||pq|| means p=(x p ,y p ),q=(x q ,y q ) represents the spatial distance between two pixel coordinates; |I(p)-I(q)|) represents the absolute value of the difference between the pixel values ​​of I(p) and I(q); σ d represents the parameter that controls the range of weight distribution in the spatial domain; σ r Represents the parameter that controls the grayscale domain weight distribution range.

[0028] As a preferred solution, the underwater target recognition network model includes a pre-convolutional layer, a local residual attention feature group structure, a feature attention module and two post-convolutional layers connected in sequence; the pre-convolutional layer is used to pre-process the input image of the underwater target recognition network model and extract its shallow features; the local residual attention feature group structure is used to extract deep noise features in the input image, retain shallow information, and fuse them; the feature attention module is used to further extract spatial features and pixel features in the input image; the two post-convolutional layers are used to perform the final feature fusion and reconstruction processing to obtain the output of the underwater target recognition network model.

[0029] Specifically, the local residual attention feature group structure includes three cascaded fully connected local residual attention feature groups, and the processing process of the local residual attention feature group structure is expressed as:

[0030] GS(F′)=Concat[G3(G2(G1(F′)))];

[0031] Among them, F′ is the input image of the local residual attention feature group structure, GS(·) represents the local residual attention feature group structure; Concat(·) represents full connection; G1(·), G2(·), G3(·) represent the operation of three local residual attention feature groups; each local residual attention feature group contains N cascaded basic block units and one convolution unit, and its processing process is expressed as:

[0032]

[0033] Among them, G i (·) represents the operation of the i-th local residual attention feature group; Conv(·) represents the convolution operation; BS i (·) represents the operation of the i-th basic block unit contained in the local residual attention feature group; It is an image pixel superposition operation; each basic block unit includes a cascaded local residual module, a convolution module and a feature attention module, and its processing process is expressed as:

[0034]

[0035] F″ is the input image of the basic block unit, BS i (·) represents the operation of the i-th basic block unit; FA(·) represents the feature attention operation; LR(·) represents the local residual operation, and Conv(·) represents the convolution operation; δ(·) is the Relu activation function; It is the image pixel superposition operation.

[0036] Specifically, the feature attention module includes a cascaded channel attention unit and a pixel attention unit;

[0037] The processing of the channel attention unit is expressed as:

[0038]

[0039] F″′ is the input image of the feature attention module, CA(·) represents the channel attention operation; Conv(·) represents the convolution operation; AvgPool(·) represents the average pooling operation; σ(·) is the Sigmoid activation function, and δ(·) is the Relu activation function; the output image of the channel attention unit is multiplied by the input image F″′ as the input image F of the feature attention module. CA ,Right now:

[0040]

[0041] The processing of the pixel attention unit is expressed as:

[0042] PA(F CA )=σConv(δ(Conv(F CA ))));

[0043] PA(·) represents the pixel attention operation; the output image of the pixel attention unit is then compared with its own input image F CA After multiplication, it is used as the output of the feature attention module. Therefore, the processing process of the feature attention module is expressed as:

[0044]

[0045] F″′ is the input image of the feature attention module, FA(·) represents the feature attention operation; It is the image pixel multiplication operation.

[0046] Specifically, the processing process of the underwater target recognition network model is expressed as:

[0047]

[0048] Wherein, F is the input image of the underwater target recognition network model, FFA(·) represents the processing operation of the underwater target recognition network model; Conv(·) represents the convolution operation; It is the image pixel superposition operation.

[0049] As a preferred solution, the loss function of the underwater target recognition network model training is specifically:

[0050]

[0051] Where: L represents the loss function; y i Represents the true target category label of the i-th underwater sample image in training; represents the predicted target object category of the model for the i-th sample image; ||·||1 represents the L1 norm operation; N represents the number of underwater sample images for training.

[0052] Compared with the prior art, the present invention has the following technical effects:

[0053] 1. The present invention proposes an underwater target recognition method for an underwater intelligent robot. First, the clarity and color restoration effect of the underwater image are enhanced by image enhancement processing, and then a new defogging target detection model is obtained by using a targeted designed underwater target recognition network model based on a feature fusion attention network architecture to output a clear fog-free image in an end-to-end manner and realize the recognition of underwater targets.

[0054] 2. The method of the present invention adopts a multi-scale enhancement algorithm with color restoration based on bilateral filtering to improve the clarity of underwater images and the robustness during training, and then uses the trained underwater target recognition network model to deploy it to the detection module of the underwater intelligent robot, so as to output a clear fog-free image in an end-to-end manner and perform underwater target recognition. This is conducive to obtaining better target recognition accuracy and real-time performance while reducing the network level of the underwater target recognition network model, and can make the underwater target recognition network model have better lightweight characteristics, and can be well applied and deployed in lightweight systems such as Linux, to achieve underwater image denoising, enhancement, defogging and underwater target recognition, and lay a good foundation for underwater intelligent robots to carry out underwater tasks.

[0055] 3. The present invention deploys an underwater target recognition network model improved based on the feature fusion attention network architecture into the detection module of the underwater intelligent robot, which can effectively help improve the feasibility of deploying the underwater target recognition algorithm on mobile terminals such as underwater intelligent robots, improve the accuracy and real-time performance of underwater target recognition in underwater intelligent robots, and enhance the versatility and practicality of the underwater target recognition algorithm based on neural networks in mobile devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solution and advantages of the invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0057] Figure 1 It is a flow chart of the underwater target recognition method for an underwater intelligent robot disclosed in the present invention;

[0058] Figure 2 A schematic diagram of the network structure of the underwater target recognition network model constructed in the method of the present invention;

[0059] Figure 3 This is an example diagram of the detection module equipment;

[0060] Figure 4 A schematic diagram of the communication mechanism between the detection modules of multiple underwater intelligent robots and the surface monitoring computer;

[0061] Figure 5 This is an example diagram showing the ROS master communication mechanism;

[0062] Figure 6 This is the schematic diagram of the ROS communication mechanism;

[0063] Figure 7 Schematic diagram of the subscriber and publisher process. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0065] The present invention will be further described in detail below in conjunction with the accompanying drawings.

[0066] In view of the problems that the underwater target recognition algorithm in the prior art has poor feasibility and poor recognition accuracy when deployed on a mobile terminal, the present invention provides an underwater target recognition method for an underwater intelligent robot, such as Figure 1 As shown, the method comprises the following steps:

[0067] S1. Obtain underwater sample images that have been labeled with target categories to form an underwater sample image dataset;

[0068] S2. Based on the feature fusion attention network architecture, an underwater target recognition network model for target recognition in underwater images is constructed;

[0069] S3, using the underwater sample image in the underwater sample image data set as the input image, performing enhancement processing on the input image, and inputting it into the underwater target recognition network model for training to obtain a trained underwater target recognition network model;

[0070] S4. Deploy the trained underwater target recognition network model to the detection module of the underwater intelligent robot, and establish communication between the image acquisition module of the underwater intelligent robot and the detection module. Use the underwater image acquired by the image acquisition module of the underwater intelligent robot as the input image, and after enhancement processing, input it into the underwater target recognition network model deployed in the detection module for recognition and detection to obtain the underwater target recognition result.

[0071] At present, underwater intelligent robots face five major challenges in underwater target recognition tasks and the deployment of target recognition algorithms: First, although Gaussian filtering can process images through weighted mean filtering, it only considers the spatial distribution of pixels and fails to take into account grayscale differences, resulting in poor image edge preservation effects; second, the defogging algorithm based on the physical model has advantages in processing noise, but its effect is limited by the estimation of atmospheric light and transmittance, which has a significant impact; third, the learning method of the convolutional network requires additional estimation of transmittance and global atmospheric light in the scattering model during the defogging process, and the uncertainty of these two parameters may lead to error amplification; fourth, image detection and deep learning technology based on the Linux system is still in the development stage, and there is currently no perfect cross-X86 platform transplantation and application solution; finally, when using the deep learning method to deploy the network model for learning, the demand for computing power is high, and power consumption cannot be ignored. In general, compared with traditional algorithms, the implementation of target recognition algorithms based on convolutional neural networks usually requires a higher cost, which restricts its deployment, application and innovative development on the lightweight system of underwater intelligent robots.

[0072] In view of this, the present invention proposes an underwater target recognition method for an underwater intelligent robot. First, the clarity and color restoration effect of the underwater image are enhanced by image enhancement processing, and then a new defogging target detection model is obtained by using an underwater target recognition network model improved based on a feature fusion attention network architecture to output a clear fog-free image in an end-to-end manner and recognize underwater targets. The method uses a multi-scale enhancement algorithm with color restoration improved based on bilateral filtering to improve the clarity of the underwater image and the robustness during training, and then the trained underwater target recognition network model is deployed to the detection module of the underwater intelligent robot to output a clear fog-free image in an end-to-end manner and perform underwater target recognition. This is conducive to obtaining good target recognition accuracy and real-time performance under the premise of reducing the network level of the underwater target recognition network model, and can make the underwater target recognition network model have good lightweight characteristics, and can be well applied and deployed in lightweight systems such as Linux to realize underwater image denoising, enhancement, defogging and underwater target recognition, which lays a good foundation for underwater intelligent robots to carry out underwater tasks. The present invention deploys an underwater target recognition network model improved based on the feature fusion attention network architecture into the detection module of the underwater intelligent robot, which can effectively help improve the feasibility of deploying the underwater target recognition algorithm on mobile terminals such as underwater intelligent robots, improve the accuracy and real-time performance of underwater target recognition in underwater intelligent robots, and enhance the versatility and practicality of the underwater target recognition algorithm based on neural network in mobile devices.

[0073] The underwater target recognition method for an underwater intelligent robot of the present invention is described in more detail below.

[0074] 1. Image preprocessing

[0075] After obtaining the target image dataset, we first need to extract frames, remove duplicates, and filter them to improve the quality of the dataset. Through the target image preprocessing technology, the data enhancement method of geometry and pixel transformation can effectively expand the existing samples, not only increasing the amount of data, but also enriching the diversity of data. The processed image samples will be randomly divided into training set, validation set, and test set in a ratio of 8:1:1.

[0076] The frame extraction stage processes the long video collected on site. Due to the similarity of the scene and target position in the video, an equidistant sampling strategy can be adopted, for example, one image is extracted every 25 frames. In addition, after the video is cropped, frame extraction is required to extract the temporal features in the continuous frames to facilitate the training of the subsequent network model. The key frames are extracted in a 1-second frame manner. In the image deduplication stage, the image data obtained by shooting and frame extraction are processed to reduce duplicate images. In order to reduce the manual workload, the software can be used for rapid deduplication, followed by manual review, and the structural similarity algorithm is used for similarity analysis. The screening stage involves manual elimination of invalid data. Before data annotation, low-quality or invalid data is removed by manual screening to ensure the quality of subsequent images. The excluded content includes images with low relevance to the task, images with too low pixels and blur, and images with poor shooting angles that are difficult to annotate. Finally, data enhancement is performed through geometric and pixel transformations to expand the samples while increasing data diversity. This method not only increases the amount of training data, but also avoids sample imbalance, improves the convergence effect and generalization ability of the model, and thus improves the overall performance of the model.

[0077] 2. Image Annotation

[0078] The image data is annotated using the LabelImg tool and after review, a preliminary data set is formed. Figure 2 As shown in the figure, it is mainly used to mark the target category and its area location in the image. During the labeling process, missing labels and incorrect labeling may have a negative impact on the training results. The target category label uses English characters, and the labeling of the target area should be as accurate as possible to ensure that the target object is completely surrounded. This is especially important for convex polygonal objects, which should completely frame their main body. Each labeled image will generate a corresponding XML file, which contains the image name and the specific coordinate information of each annotation box.

[0079] 3. Preparatory work for training underwater target recognition network model

[0080] In the actual application experiment, a total of 5,000 image samples were collected, and after expansion through data enhancement technology, the total number reached 6,000. To avoid data overlap, the data set is divided into training set, validation set and test set in a ratio of 8:1:1, with the number of samples being 4,800, 600 and 600 respectively. The deep learning environment configuration of this actual application experiment is as follows: the CPU is Intel(R) Xeon(R) Bronze 3204 (1.9GHz), the GPU is NVIDIA GeForce RTX 2060 (6144MiB), the compiler used is Python 3.8, the deep learning framework is Pytorch 1.6, and the acceleration module is CUDA 11.8.

[0081] 4. Multi-scale enhancement algorithm with color restoration improved by bilateral filtering

[0082] Due to the complex underwater environment and poor imaging quality, it is necessary to process the images taken by the camera through image enhancement algorithms to improve the clarity, contrast and other useful information of the image, remove useless information such as interference noise, and improve the accuracy of target detection. In recent years, with the continuous development of deep learning, underwater image enhancement has also been further developed due to the addition of deep learning. Deep convolutional neural networks have powerful supervised learning models and have great advantages in processing low-level visual tasks. It can better improve image quality and accuracy while reducing dependence on underwater image imaging expertise, and perform more scalable and automated analysis of images.

[0083] Therefore, the present invention proposes a method of enhancing underwater images by using a multi-scale Retinex enhancement algorithm with color restoration based on bilateral filtering (improved MSRCR algorithm) to improve the clarity and robustness of underwater images during training, and then modifying its network structure based on the feature fusion attention network architecture to obtain a new underwater target recognition network model, so as to output a clear fog-free image and obtain target recognition results in an end-to-end manner. The following is a detailed introduction to the multi-scale enhancement algorithm with color restoration based on bilateral filtering proposed by the method of the present invention.

[0084] There are many ways to implement the Retinex algorithm. The Retinex theory is a model about how the human visual system perceives color, which aims to explain the mechanism of people's perception of color and brightness. The mathematical model of this theory was proposed by Land et al., and its formula is shown below.

[0085] S(x,y)=R(x,y)·L(x,y)

[0086] In this formula, an image is composed of arranged pixels, and x and y represent the horizontal and vertical coordinates of a pixel, respectively. Retinex theory believes that the image S(x,y) perceived by the human eye can be decomposed into the incident image L(x,y) under the influence of light and the reflectivity R(x,y) of the reflected image. Among them, R(x,y) represents the intrinsic properties of the pixel and is independent of the incident light.

[0087] The SSR (Single Scale Retinex) algorithm takes the logarithm of both sides of the formula so that the reflected image component can be obtained by subtracting the incident image component from the image component perceived by the human eye. The formula is as follows.

[0088] log[R(x,y)=log[S(x,y)]-log[L(x,y)]

[0089] In this formula, since the incident image component is difficult to obtain directly, a Gaussian filter is used to perform a convolution operation on the original image to replace the incident image component. The corresponding formula is shown below.

[0090] log[R(x,y)=log[S(x,y)]-log[S(x,y)*G(x,y)]

[0091] In the formula, * represents convolution operation; G(x,y) represents Gaussian filtering, and the Gaussian filtering formula is as follows.

[0092]

[0093] In the formula, x and y represent the deviation from the center pixel; σ represents the standard deviation of the Gaussian kernel. When the σ value is too small, the filter range is very small, resulting in an insignificant smoothing effect. At this time, more original details and noise of the image are retained; when the σ value is too large, the smoothing effect on the image is better, but excessive smoothing will also cause the image to lack edge details.

[0094] Based on the SSR algorithm, the MSR (Multi-Scale Retinex) algorithm introduces multiple Gaussian kernels of different scales to perform Retinex processing on the input image. Finally, the MSR algorithm generates the final image by weighted fusion of the processing results of these different scales. The corresponding MSR algorithm formula is shown below.

[0095]

[0096] Where K represents the number of scales contained in the multi-scale enhancement algorithm with color restoration, usually 3, representing high, medium, and low scales respectively; ω k represents the enhancement weight of the kth scale in the multi-scale enhancement algorithm with color restoration, and is taken

[0097] The MSRCR (Multi-Scale Retinex with Color Restoration) algorithm is a further optimization of the MSR algorithm, which aims to solve the problem that the MSR algorithm may introduce noise and cause local detail color distortion during image processing. To this end, the MSRCR algorithm introduces a color restoration function to adjust the weights between different color channels in the original image. This process can adjust the color of the darker area information, thereby effectively eliminating color distortion in the image.

[0098] The formula of the MSRCR algorithm of the prior art is as follows:

[0099]

[0100]

[0101] In the formula, C i (x, y) represents the color restoration factor of the i-th color channel of the original image; α and β are color restoration adjustment parameters, α is used to control the intensity of color restoration, and β is used to control the degree of influence of color restoration; I j (x,y) represents the pixel value of the original image in the i-th color channel, It shows the sum of the pixel values ​​of the three color channels of the original image at each pixel position.

[0102] Finally, a gain value G and an offset value b are added to the formula to obtain the final MSRCR algorithm, and the formula is shown below.

[0103]

[0104] The Gaussian filter used in the MSRCR algorithm is a linear smoothing filter. Although it can blur the image, it may cause loss of details, especially at the edge and texture. In order to solve this problem, the present invention proposes to replace the Gaussian filter with a bilateral filter in the improved MSRCR algorithm to enhance the effect of underwater images.

[0105] Bilateral filtering is a nonlinear filter that considers not only the spatial relationship of pixels in an image, but also the difference between their grayscale values ​​or color values. By weighted averaging pixels, bilateral filtering can effectively remove noise and smooth the image while retaining the clarity of edge information, thereby achieving high-quality filtering. The formula for bilateral filtering is shown below.

[0106]

[0107] Where B(x, y) represents the pixel value of the image after bilateral filtering; I(p) and I(q) represent the pixel value of the original image at the pixel coordinate p=(x p ,y p ) and q=(x q ,y q ), S represents the pixel coordinate domain of the original image; G d (||pq||) represents the spatial kernel function, which is used to measure p=(x p ,y p ),q=(x q ,y q ) The pixel space distance between two pixel coordinates is usually represented by a Gaussian function, and the weight of the space distance can be controlled by adjusting the parameters; G r (|I(p)-I(q)|) represents the pixel value kernel function, which is used to measure the pixel value difference between I(p) and I(q) pixel values. It is usually represented by a Gaussian function, and the weight of the pixel value difference can be controlled by adjusting the parameters. p Represents the normalized weight coefficient of bilateral filtering, which is used to ensure that the pixel value after filtering is within a certain range. p The formula is as follows.

[0108]

[0109]

[0110]

[0111] In the formula, ||pq|| means p=(x p ,y p ),q=(x q ,y q ) represents the spatial distance between two pixel coordinates; |I(p)-I(q)|) represents the absolute value of the difference between the pixel values ​​of I(p) and I(q); σ d represents the parameter that controls the range of weight distribution in the spatial domain; σ r Indicates the parameter that controls the grayscale domain weight distribution range. The present invention adopts a bilateral filter function to replace the Gaussian filter function in the existing MSRCR algorithm, and obtains an improved MSRCR underwater image enhancement algorithm based on the bilateral filter function, and its algorithm formula is as follows:

[0112]

[0113] Among them, S(x,y) represents the original image before enhancement; Represents the enhanced output of the i-th color channel of the original image; C i(x, y) represents the color restoration factor of the i-th color channel of the original image; B(x, y) represents the bilateral filtered image of the original image; G and b are the gain value and offset value of the multi-scale enhancement algorithm with color restoration, respectively; * represents the convolution operation; ω k represents the enhancement weight of the kth scale in the multi-scale enhancement algorithm with color restoration, and K represents the number of scales included in the multi-scale enhancement algorithm with color restoration. Usually K is 3, representing high, medium, and low scales respectively.

[0114] In the multi-scale enhancement algorithm with color restoration adopted by the present invention, bilateral filtering is used to replace the traditional Gaussian filtering to enhance the underwater image. In the filtering process, not only the spatial relationship of pixels in the image is considered, but also the grayscale or color difference between them is integrated. Among them, by weighted averaging the pixels, not only the noise is effectively removed and the image is smoothed, but also the clarity of the edge information in the image is retained.

[0115] 5. Improved underwater target recognition network model based on feature fusion attention network architecture

[0116] The structure of the underwater target recognition network model proposed in this invention based on the improved feature fusion attention network architecture is as follows: Figure 2 As shown. The underwater target recognition network model includes a pre-convolution layer, a local residual attention feature group structure, a feature attention module and two post-convolution layers connected in sequence; the pre-convolution layer is used to pre-process the input image of the underwater target recognition network model and extract its shallow features; the local residual attention feature group structure is used to extract the deep noise features in the input image, retain the shallow information, and fuse them; the feature attention module is used to further extract the spatial features and pixel features in the input image; the two post-convolution layers are used to perform the final feature fusion reconstruction processing to obtain the output of the underwater target recognition network model.

[0117] Such a network architecture design helps to more effectively capture the pixel features of different levels in underwater images. Finally, the feature attention module is used to fuse all the extracted features to obtain a clear fog-free image in an end-to-end manner, which can help to more accurately realize underwater target recognition. At the same time, a multi-scale enhancement algorithm with color restoration based on bilateral filtering is used to improve the clarity of underwater images and the robustness during training. The trained underwater target recognition network model is then deployed to the detection module of the underwater intelligent robot to output a clear fog-free image and perform underwater target recognition in an end-to-end manner. This is conducive to obtaining better target recognition accuracy and real-time performance while reducing the network level of the underwater target recognition network model. The underwater target recognition network model has a good lightweight feature and can be well applied and deployed in lightweight systems such as Linux to achieve underwater image denoising, enhancement, defogging and underwater target recognition, laying a good foundation for underwater intelligent robots to carry out underwater tasks.

[0118] 5.1 Network architecture of underwater target recognition network model

[0119] After the input image of the underwater target recognition network model is processed by the pre-convolution layer, it is first input into the local residual attention feature group structure for deep noise feature extraction. The local residual attention feature group structure includes three cascaded fully connected local residual attention feature groups. The processing process of the local residual attention feature group structure is expressed as:

[0120] GS(F′)=Concat[G3(G2(G1(F′)))];

[0121] Among them, F′ is the input image of the local residual attention feature group structure, GS(·) represents the local residual attention feature group structure; Concat(·) represents full connection; G1(·), G2(·), G3(·) represent the operation of three local residual attention feature groups; each local residual attention feature group contains N cascaded basic block units and one convolution unit, and its processing process is expressed as:

[0122]

[0123] Among them, G i (·) represents the operation of the i-th local residual attention feature group; Conv(·) represents the convolution operation; BS i (·) represents the operation of the i-th basic block unit contained in the local residual attention feature group; It is an image pixel superposition operation; each basic block unit includes a cascaded local residual module, a convolution module and a feature attention module, and its processing process is expressed as:

[0124]

[0125] F″ is the input image of the basic block unit, BS i (·) represents the operation of the i-th basic block unit; FA(·) represents the feature attention operation; LR(·) represents the local residual operation, and Conv(·) represents the convolution operation; δ(·) is the Relu activation function; It is the image pixel superposition operation.

[0126] The design of the local residual attention feature group structure enables the underwater target recognition network model to directly learn the mapping relationship required to remove noise from the input underwater image, thereby simplifying the denoising process and breaking the limitation of the physical model-based dehazing method in estimating atmospheric light and transmittance inaccurately.

[0127] The feature attention module includes a cascaded channel attention unit and a pixel attention unit. The processing of the channel attention unit is expressed as:

[0128]

[0129] F′″ is the input image of the feature attention module, CA(·) represents the channel attention operation; Conv(·) represents the convolution operation; AvgPool(·) represents the average pooling operation; σ(·) is the Sigmoid activation function, and δ(·) is the Relu activation function; the output image of the channel attention unit is multiplied by the input image F″′ as the input image F of the feature attention module. CA ,Right now:

[0130]

[0131] The processing of the pixel attention unit is expressed as:

[0132] PA(F CA )=σ(Conv(δ(Conv(F CA ))));

[0133] PA(·) represents the pixel attention operation; the output image of the pixel attention unit is then compared with its own input image F CA After multiplication, it is used as the output of the feature attention module. Therefore, the processing process of the feature attention module is expressed as:

[0134]

[0135] F″′ is the input image of the feature attention module, FA(·) represents the feature attention operation; It is the image pixel multiplication operation.

[0136] The feature attention module obtains weights through adaptive learning to fuse features. The process can retain the low-level information and pass it to the deep layer. Due to the existence of the weight mechanism, the underwater target recognition network model network can more effectively focus on effective information such as thick zero areas, high-frequency textures and color fidelity, and better obtain fog-free images, thereby improving the underwater target recognition effect.

[0137] Therefore, combined with Figure 2 ,The processing process of the underwater target recognition network model can be expressed as:

[0138]

[0139] Wherein, F is the input image of the underwater target recognition network model, FFA(·) represents the processing operation of the underwater target recognition network model; Conv(·) represents the convolution operation; It is the image pixel superposition operation.

[0140] 5.2 Training of underwater target recognition network model

[0141] Using the training data previously prepared for the training of the underwater target recognition network model and the hardware equipment configured for the deep learning environment, the underwater target recognition network model is trained using the training set, the training loss is calculated, and the parameters of the underwater target recognition network model are optimized based on the training loss. The training results are then verified and tested using the validation set and the test set until the parameter tuning of the underwater target recognition network model converges and the training is completed.

[0142] The loss function for underwater target recognition network model training is preferably:

[0143]

[0144] Where: L represents the loss function; y i Represents the true target category label of the i-th underwater sample image in training; represents the predicted target category of the model for the i-th sample image; ||·||1 represents the L1 norm operation; N represents the number of underwater sample images trained. The use of this loss function is helpful to optimize the convergence performance of underwater target recognition network model training.

[0145] 6. Development and operation environment of underwater target recognition network model configured on the detection module

[0146] The development environment is built by installing the Ubuntu 20.04 system on the detection module of the underwater intelligent robot, and then installing the development environment required for the development and operation of the target detection algorithm. The development environment conditions are shown in Table 1. After installing the system and the dependent software environment, you can install the ROS development environment.

[0147] Table 1 Development environment condition parameter description

[0148]

[0149] Examples of detection module devices built based on the above development environment configuration are as follows Figure 3 shown.

[0150] 7. The underwater target recognition network model is deployed to the detection module of the underwater intelligent robot

[0151] After completing the model training on a high-performance computer, the target recognition algorithm will be converted into a TensorRT model through C++ to accelerate the reasoning process in the artificial intelligence module system. TensorRT uses an optimized algorithm structure to achieve faster and more efficient reasoning speed on NVIDIA GPUs. The usage process includes two parts: build and deployment:

[0152] During the construction phase, TensorRT generates optimization decisions to convert and optimize the network. To avoid additional calculations, unused network layers are deleted and multiple network layers are merged. Through vertical layer fusion and horizontal layer fusion, the network and tensors are fused to reduce the calculation process and transmission time. Vertical fusion is to integrate the Conv layer, BN layer or offset layer, and activation layer into a CBR (Convolution, Bias and ReLU). Horizontal fusion is to fuse several identical inputs to reduce the depth and width of the network, reduce repeated calls to the same function, and improve the inference speed.

[0153] During the deployment phase, when the system receives data to be processed, it executes the previously developed optimization strategy and performs reasoning and recognition. This process is designed to minimize latency while maximizing the system's throughput.

[0154] In order to realize that the target detection algorithm meets the requirements of the target detection task on the underwater intelligent robot, the above algorithm is deployed to the detection module of the underwater intelligent robot and finally verified. When the target detection model is deployed to the detection module of the underwater intelligent robot, in order to meet the requirements of lightweight and real-time, the present invention adopts offline reasoning for verification. Offline reasoning is that the user uses the model trained by the deep learning experimental platform to convert it into an offline model supported by the detection module of the underwater intelligent robot, and then loads and executes the offline model. This offline reasoning can use batch loading models to perform reasoning tasks, and the model will run faster. The deployment process is to convert the weight parameter file saved after learning the deep learning experimental platform into a format supported by the detection module of the underwater intelligent robot, and then replace the weight file of the target detection algorithm in the detection module of the underwater intelligent robot.

[0155] After obtaining the recognition result of the underwater target, the detection module of the underwater intelligent robot can upload the underwater target recognition result data to the surface monitoring computer through the wireless communication mechanism. Figure 4 A schematic diagram of the communication mechanism between the detection modules (Topic) of multiple underwater intelligent robots and the surface monitoring computer (Computer) is shown. The communication transmission between them may need to be achieved with the help of distributed network communication nodes (Node).

[0156] 8. ROS establishes communication between the camera and the detection module

[0157] The communication mechanism of ROS (Robot Operating System) is as follows Figure 5 As shown in the figure, a loosely coupled distributed communication method is adopted to achieve effective communication between various ROS nodes. Based on a unified communication standard, the loosely coupled distributed framework of ROS allows each node to run its functions independently, and can even be distributed on different hosts for collaborative work. Various sensors on underwater intelligent robots, such as cameras, radars, GPS, etc., need to transmit data to achieve reasonable control of underwater intelligent robots, and the implementation of process communication is the key to transmitting data.

[0158] The basic communication methods of ROS can be divided into three types:

[0159] 1. Topic communication (publish and subscribe mode)

[0160] 2. Service communication (request response mode)

[0161] 3. Parameter server (parameter sharing mode)

[0162] roscore is a collection of nodes and programs that constitute the core services of the ROS system, hence the name "ros core". To achieve topic communication, you must first run roscore, which is started by entering the roscore command in the terminal of the Ubuntu system. The roscore command will start the following functions:

[0163] ROS Master: The main node responsible for communication.

[0164] ROS Parameter Server: A server for storing and managing parameters.

[0165] Rosout Logging Node: Node used to output logs.

[0166] A node is an execution unit of a ROS program, a process that executes a specific task or an independent executable file. A message is used to describe the type of data in topic transmission. Users can define interfaces by writing .msg files, and corresponding code files will be generated during the compilation process.

[0167] ROS Master serves as the control center for all nodes, such as Figure 6 As shown, it provides naming and registration services, and tracks publishers and subscribers of topics and services. Its role is to enable ROS nodes to locate each other and provide parameter servers. Once the nodes locate each other, they can start communicating. ROS Master interacts with ROS client libraries (such as roscpp and rospy) through an XML-RPC-based API to store and retrieve information. API (Application Programming Interface) is a set of standards that define the exchange of information and commands between applications and computer operating systems or network operating systems.

[0168] XML-RPC is a remote procedure call (RPC) technology that uses the HTTP protocol as a transmission mechanism and uses XML text format to transmit commands and data. RPC (Remote Procedure Call) allows a local machine to call a procedure (method) on a remote machine. It is a distributed computing technology designed to improve the "interoperability" between different machines.

[0169] The ROS communication process mainly uses topic communication. Topics use an asynchronous communication mechanism. Nodes transmit data through a publish / subscribe model. Data is transmitted from publishers to subscribers. There can be multiple subscribers or publishers for the same topic.

[0170] In ROS, publishers and subscribers are the core components for implementing inter-node communication. They publish and receive messages through topics, forming a flexible and efficient data transmission mechanism. For example, the camera node publishes image data so that other nodes in the system can receive real-time images, and the target detection node processes these image data by subscribing to them. The subscriber processes the received messages by registering callback functions. The system automatically calls the defined callback function every time it receives a message from the ( / camera) topic, so that the target detection node can respond to the data in real time. This topic-based publish / subscribe mechanism enables nodes in the ROS system to exchange data efficiently without having to understand each other's internal implementation. Through this decoupled design, ROS improves the flexibility and scalability of the system and simplifies the communication process between nodes, thereby supporting developers to build more complex and feature-rich robotic applications.

[0171] The creation process of publishers and subscribers in ROS is as follows Figure 7 shown.

[0172] To create a publisher, you must first customize the topic message, and then write the following in the publisher code:

[0173] Initialize the ROS node;

[0174] Register node information with ROS Master, including the published topic name and message type in the topic;

[0175] Create message data;

[0176] Set the frequency of publishing messages to implement a publishing information cycle.

[0177] Creation content of subscribers in ROS:

[0178] Initialize the ROS node;

[0179] Subscribe to the required topics;

[0180] Wait for topic messages in a loop, and enter the callback function after receiving the message;

[0181] Complete message processing in the callback function.

[0182] To realize the communication between the camera and the detection module based on the above content, the following ROS project needs to be established in the Ubuntu system. The ROS workspace (catkin_work space) is used to store function packages, which generally contains three subspaces: src: source space, build: build space, and devel: development space. Before creating any program, you must first create a workspace, that is, create a workspace, compile a workspace, set the environment, and check the environment variables in the terminal of the Ubuntu system. Then create the detection and camera function packages in the src subdirectory of the workspace, and then compile the function package. After the function package is created, two files will be generated in the directory of the function package, one is the package.xml manifest file, which provides relevant information about the function package and defines the dependencies between function packages, and the other is the CMakeLists.txt script file, which is used for compilation.

[0183] 9. Overview

[0184] Aiming at target monitoring in complex underwater environments, the present invention provides an underwater target recognition method for an underwater intelligent robot. The method uses a multi-scale enhancement algorithm with color restoration based on bilateral filtering to denoise and perform image enhancement preprocessing on underwater images. Compared with other image denoising and enhancement methods, it is a nonlinear filtering that considers spatial grayscale distribution and has better edge retention, which can help improve the clarity of underwater images and the robustness during model training. At the same time, the present invention proposes to fuse feature information of different depth layers through a deep feature information fusion layer to achieve end-to-end image clarity processing without relying on the model. The mapping relationship required for noise removal is directly learned from the input underwater image through end-to-end training, thereby simplifying the denoising process and breaking the limitation that the defogging method based on the physical model is not accurate enough in estimating atmospheric light and transmittance. The underwater target recognition method of the underwater intelligent robot proposed in the present invention can be deployed and applied on lightweight systems such as Linux, which is conducive to the popularization of the target recognition algorithm based on the neural network on mobile terminals such as underwater intelligent robots, thereby improving the feasibility of deploying the target recognition algorithm on the mobile terminal, the accuracy and real-time performance of high target recognition on the mobile terminal, and enhancing the versatility and practicality of the underwater target recognition algorithm based on the neural network on mobile devices.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, it should be understood by those skilled in the art that various changes may be made in form and details without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. An underwater target recognition method for an underwater intelligent robot, characterized in that: The method comprises: S1. Obtain underwater sample images that have been labeled with target categories to form an underwater sample image dataset; S2. Based on the feature fusion attention network architecture, an underwater target recognition network model for target recognition in underwater images is constructed; S3, using the underwater sample image in the underwater sample image data set as the input image, performing enhancement processing on the input image, and inputting it into the underwater target recognition network model for training to obtain a trained underwater target recognition network model; S4. Deploy the trained underwater target recognition network model to the detection module of the underwater intelligent robot, and establish communication between the image acquisition module of the underwater intelligent robot and the detection module. Use the underwater image acquired by the image acquisition module of the underwater intelligent robot as the input image, and after enhancement processing, input it into the underwater target recognition network model deployed in the detection module for recognition and detection to obtain the underwater target recognition result.

2. The underwater target recognition method for an underwater intelligent robot according to claim 1, characterized in that: Before inputting the input image into the underwater target recognition network model, the input image is first enhanced using a multi-scale enhancement algorithm with color restoration, where each channel of the input original image is enhanced as follows: Among them, S(x,y) represents the original image before enhancement; Represents the enhanced output of the i-th color channel of the original image; C i (x, y) represents the color restoration factor of the i-th color channel of the original image; B(x, y) represents the bilateral filtered image of the original image; G and b are the gain value and offset value of the multi-scale enhancement algorithm with color restoration, respectively; * represents the convolution operation; ω k represents the enhancement weight of the kth scale in the multi-scale enhancement algorithm with color restoration, and K represents the number of scales included in the multi-scale enhancement algorithm with color restoration.

3. The underwater target recognition method for an underwater intelligent robot according to claim 2, characterized in that: The color restoration factor C of the i-th color channel of the original image i (x,y) is calculated as follows: In the formula, I j (x,y) represents the pixel value of the original image in the i-th color channel, Shows the sum of the pixel values ​​of the three color channels of the original image at each pixel position; α and β are color restoration adjustment parameters, which are used to control the intensity and influence of color restoration respectively.

4. The underwater target recognition method for an underwater intelligent robot according to claim 2, characterized in that: The bilateral filtered image B(x,y) of the original image is calculated as follows: Where B(x,y) represents the pixel value after filtering; I(p) and I(q) represent the pixel coordinates p=(x p ,y p ) and q=(x q ,y q ), S represents the pixel coordinate domain of the original image; G d (||pq||) represents the spatial kernel function, which is used to measure p=(x p ,y p ),q=(x q ,y q ) The pixel space distance between two pixel coordinates; G r (|I(p)-I(q)|) represents the pixel value kernel function, which is used to measure the pixel value difference between I(p) and I(q); W p Represents the normalized weight coefficient of bilateral filtering.

5. The underwater target recognition method for an underwater intelligent robot according to claim 4, characterized in that: Normalized weight coefficient W of bilateral filtering p It is calculated as follows: W p =∑ q∈S G d (||p-q||)G r (|I(p)-I(q)|); In the formula, ||pq|| means p=(x p ,y p ),q=(x q ,y q ) represents the spatial distance between two pixel coordinates; |I(p)-I(q)|) represents the absolute value of the difference between the pixel values ​​of I(p) and I(q); σ d represents the parameter that controls the range of weight distribution in the spatial domain; σ r Represents the parameter that controls the grayscale domain weight distribution range.

6. The underwater target recognition method for an underwater intelligent robot according to claim 1, characterized in that: The underwater target recognition network model comprises a pre-convolutional layer, a local residual attention feature group structure, a feature attention module and two post-convolutional layers connected in sequence; the pre-convolutional layer is used to pre-process the input image of the underwater target recognition network model and extract its shallow features; The local residual attention feature group structure is used to extract deep noise features in the input image, retain shallow information, and fuse them; the feature attention module is used to further extract spatial features and pixel features in the input image; The two post-convolutional layers are used for the final feature fusion and reconstruction processing to obtain the output of the underwater target recognition network model.

7. The underwater target recognition method for an underwater intelligent robot according to claim 6, characterized in that: The local residual attention feature group structure includes three cascaded fully connected local residual attention feature groups, and the processing process of the local residual attention feature group structure is expressed as: GS(F′)=Concat[G3(G2(G1(F′)))]; Among them, F′ is the input image of the local residual attention feature group structure, GS(·) represents the local residual attention feature group structure; Concat(·) represents full connection; G1(·), G2(·), G3(·) represent the operation of three local residual attention feature groups; each local residual attention feature group contains N cascaded basic block units and one convolution unit, and its processing process is expressed as: Among them, G i (·) represents the operation of the i-th local residual attention feature group; Conv(·) represents the convolution operation; BS i (·) represents the operation of the i-th basic block unit contained in the local residual attention feature group; It is an image pixel superposition operation; each basic block unit includes a cascaded local residual module, a convolution module and a feature attention module, and its processing process is expressed as: F″ is the input image of the basic block unit, BS i (·) represents the operation of the i-th basic block unit; FA(·) represents the feature attention operation; LR(·) represents the local residual operation, and Conv(·) represents the convolution operation; δ(·) is the Relu activation function; It is the image pixel superposition operation.

8. The underwater target recognition method for an underwater intelligent robot according to claim 6 or 7, characterized in that: The feature attention module includes a cascaded channel attention unit and a pixel attention unit; The processing of the channel attention unit is expressed as: F′″ is the input image of the feature attention module, CA(·) represents the channel attention operation; Conv(·) represents the convolution operation; AvgPool(·) represents the average pooling operation; σ(·) is the Sigmoid activation function, and δ(·) is the Relu activation function; the output image of the channel attention unit is multiplied by the input image F″′ as the input image F of the feature attention module. CA ,Right now: The processing of the pixel attention unit is expressed as: PA(F CA )=σ(Conv(δ(Conv(F CA )))); PA(·) represents the pixel attention operation; the output image of the pixel attention unit is then compared with its own input image F CA After multiplication, it is used as the output of the feature attention module. Therefore, the processing process of the feature attention module is expressed as: F′″ is the input image of the feature attention module, FA(·) represents the feature attention operation; It is the image pixel multiplication operation.

9. The underwater target recognition method for an underwater intelligent robot according to claim 8, characterized in that: The processing process of the underwater target recognition network model is expressed as: Wherein, F is the input image of the underwater target recognition network model, FFA(·) represents the processing operation of the underwater target recognition network model; Conv(·) represents the convolution operation; It is the image pixel superposition operation.

10. The underwater target recognition method for an underwater intelligent robot according to claim 1, characterized in that: The loss function of the underwater target recognition network model training is specifically: Where: L represents the loss function; y i Represents the true target category label of the i-th underwater sample image in training; represents the predicted target object category of the model for the i-th sample image; ||·||1 represents the L1 norm operation; N represents the number of underwater sample images for training.

Images