Underwater image super-resolution and target identification method

By adopting super-resolved target recognition model and innovative training strategies in underwater image processing, the problems of low underwater image quality and lack of data are solved, and the image clarity and recognition performance are significantly improved, which is suitable for complex underwater environments.

CN119067848AActive Publication Date: 2024-12-03UNIV OF SCI & TECH OF CHINA

Patent Information

Application Number
CN202411154805.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-03
Estimated Expiration
2044-08-22

AI Technical Summary

Technical Problem

The quality of underwater images is low, the efficiency of traditional processing methods is low, and the data set is scarce, which leads to difficult target recognition, insufficient algorithm accuracy and speed, and poor robustness in complex environments.

Method used

The super-resolution target recognition model, including encoder, decoder and feature pyramid network, uses orthogonal bidirectional attention module and upper scaler, and combines training strategies for paired similar graph consistency loss and cross-label supervision to improve image quality and Identify performance.

Benefits of technology

It significantly improves the signal-to-noise ratio and clarity of underwater images, improves the target recognition performance, alleviates the scarcity of data sets, and enhances the robustness and generalization capabilities of the algorithm.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067848A_ABST
    Figure CN119067848A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of underwater imaging detection, and discloses an underwater image super-resolution and target identification method, and the method specifically comprises the steps: an encoder comprises a convolution layer and a depth feature extraction part; the method comprises the following steps: down-sampling an input underwater image by using a convolutional layer, extracting to obtain an initial feature, and then inputting the initial feature into a depth feature extraction part consisting of a plurality of orthogonal bidirectional attention modules in sequence; inputting the feature input into an upper scaler and a convolutional layer in sequence to obtain a reconstructed image of the underwater image; inputting the features output by the encoder into the feature pyramid network; and connecting the plurality of aligned regions of interest, and inputting the connected regions of interest into a convolutional layer to obtain a classification result of the target and a boundary box of the target. The signal-to-noise ratio and definition of the image can be remarkably improved, and a foundation is laid for improvement of target recognition performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of underwater imaging detection, and in particular to an underwater image super-resolution and target recognition method. Background Art

[0002] Underwater imaging detection technology has become the focus of research in the current rapid rise of underwater resource development, environmental monitoring and marine military. The breakthrough of deep learning technology has injected new vitality into the development of this field. As a key technology for underwater intelligent detection, the research on underwater image target detection is becoming increasingly active and is widely used in aquatic biological detection, water environment exploration, seabed modeling, salvage and rescue, submarine pipeline detection, anti-mine and anti-submarine tasks. These studies not only broaden the boundaries of underwater detection, but also provide strong technical support for the further development of related fields. However, due to the significant differences between the underwater environment and the terrestrial environment, especially the absorption and scattering of light by water, the quality of underwater images collected by electronic equipment is seriously degraded, manifested as serious color distortion, lack of details, reduced contrast and blur, which greatly hinders the accurate acquisition of information. Traditional underwater image processing methods, such as enhancement, restoration and super-resolution reconstruction algorithms, are generally limited by the dependence on degradation models and low processing efficiency, so they are limited in terms of application scope and real-time performance.

[0003] At present, a major constraint on the large-scale application of underwater unmanned detection technology is the performance shortcomings of the detection algorithm and the lack of underwater image data sets. In most cases, human intervention is still required to assist in processing. At the same time, underwater biological targets are small and densely distributed, and the existence of overlapping occlusions still increases the difficulty of target recognition. Therefore, improving the accuracy and speed of the algorithm, expanding the underwater image data set, enhancing the robustness in complex environments, broadening the generalization ability of the algorithm, and optimizing the computational efficiency of the model have become key problems that need to be overcome in this field. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an underwater image super-resolution and target recognition method.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A method for underwater image super-resolution and target recognition, wherein the super-resolution target recognition model adopted includes an encoder, a decoder and a feature pyramid network; specifically, the following steps are included:

[0007] Step 1: The encoder includes a convolutional layer and a deep feature extraction part. The convolutional layer is used to downsample the input underwater image I to extract the initial feature F0, and then F0 is sequentially input into the deep feature extraction part composed of A orthogonal bidirectional attention modules. The A orthogonal bidirectional attention modules output the features F1, F2, ..., F extracted at different levels respectively. A , for F0, F1, F2, ..., F A Splice and obtain the feature C1 output by the encoder;

[0008] Step 2: The decoder includes an upscaler and a convolutional layer; the feature C1 input is sequentially input to the upscaler and the convolutional layer to obtain a reconstructed image of the underwater image.

[0009] Step three, input the feature C1 output by the encoder into the feature pyramid network; the feature pyramid network includes a bottom-up path, a top-down path, and a lateral connection path; the bottom-up path uses a deep residual network; the region selection network is used to extract the region of interest from each feature map in the top-down path and align them; after connecting multiple aligned regions of interest, they are input into the convolutional layer to obtain the classification result of the target and the bounding box of the target.

[0010] Furthermore, the orthogonal bidirectional attention module includes a plurality of alternating convolutional layers and attention modules, for alternately performing convolution operations and attention operations on the input feature F;

[0011] The attention module includes two bidirectional long short-term memory networks. The first bidirectional long short-term memory network scans pixels one by one from left to right and from right to left along each row of feature F to obtain two feature maps. The first bidirectional long short-term memory network splices the two feature maps and outputs them to combine the left context information and the right context information of each pixel. The second bidirectional long short-term memory network scans each column of the feature map in a bottom-up and top-down order along the feature map output by the first bidirectional long short-term memory network, and obtains the overall feature map by splicing; the convolution layer is used to convert the overall feature map into D channels to form a feature weight α, where D = W×H, W, H are the width and height of the input feature F respectively; for the feature vector p of the pixel at position (w, h) in the feature weight α w,h ∈R D , use the convolution layer to change the dimension to C×W×H and then use the softmax function to normalize it to get a new feature weight α w,h ∈R D , and then generate a new feature weight α for each pixel w,h Weighted, we get the context feature F att :

[0012] New feature weight of the i-th channel Obtained by the following formula:

[0013]

[0014] is the feature vector of the pixel at position (w, h) in the feature weight α of the i-th channel;

[0015] For the pixel at position (w,h), the context feature F with the same size as F is involved att Obtained by the following formula:

[0016]

[0017] Among them, f w,h ∈R D is the convolution feature at position (w,h) in F, F att The feature at position (w,h) in the image.

[0018] Furthermore, the upscaler includes a plurality of convolution groups, each of which includes a normal convolution and a sub-pixel convolution; let Conv(n c ,k,n f ) is a convolutional layer, where n c ,k,n f are the number of filter channels, kernel size, and number of filters respectively; for n c The input features of feature maps, ordinary convolution Conv(n c ,3,r 2 ·n f ) Increase the number of feature maps of the input features to r 2 ·n f , and the size is H×W×r 2 ·n f The feature tensor of , where r represents the upscaling factor (customizable magnification); H, W represent the height and width of the feature map of the input feature respectively; sub-pixel convolution PixelShuffle(r) converts the size of H×W×r 2 ·n f The elements in the feature tensor of are rearranged to a size of (r·H)×(r·W)×n f The feature tensor of .

[0019] Furthermore, the training strategy of the super-resolution target recognition model includes:

[0020] Pairwise Similarity Figure 1 Caused loss:

[0021] Two super-resolution target recognition models with the same structure but different initialization methods are used, the encoder and the decoder constitute an autoencoder network, the autoencoder networks of the two super-resolution target recognition models are respectively defined as a first autoencoder network A(θ1) and a second autoencoder network A(θ2), and the feature pyramid networks of the two super-resolution target recognition models are respectively recorded as a first feature pyramid network F(θ1) and a second feature pyramid network F(θ2); A(θ1) and F(θ1) constitute a first super-resolution target recognition model, and A(θ2) and F(θ2) constitute a second super-resolution target recognition model;

[0022] In the autoencoder network, the feature map P from the encoder is e , that is, the feature C1, and the feature map P output by the decoder d , that is, the reconstructed image Calculate similarity Figure 1 Causative losses;

[0023] For a given batch size B, P e and P d The shapes of are B×C×W×H and B×C'×W'×H', where H and W are the feature maps P e The width and height of the feature map P are shown in Figure 2. d The width and height of C' is the corresponding number of feature channels; in order to calculate similarity Figure 1 Consistency loss, first we need to obtain the similarity graph matrix, that is, the Gram matrix, and return the similarity value between two pairs of inputs according to the output of the autoencoder network A(θ1), A(θ2); in order to reduce the impact of changes in similarity values, the similarity graph matrix is ​​L2 normalized to obtain a pairwise similarity graph matrix ψ with a shape of B×B, that is, for the i-th super-resolution object recognition model:

[0024]

[0025] ‖·‖2 represents the bi-norm, i∈{1,2}, ψ1 and ψ2 are the similarity graph matrices generated by the autoencoder networks A(θ1) and A(θ2), respectively;

[0026] The similarities proposed Figure 1 Causative loss sim for:

[0027]

[0028] L D (·,·) is the root mean square error, Respectively represent the feature graph P in A(θ1) and A(θ2) e The similarity graph matrix of Represents the feature map P output by A(θ1) and A(θ2) respectively d The similarity graph matrix of .

[0029] Cross-label supervision:

[0030] The classification results output by the first super-resolution target recognition model and the second super-resolution target recognition model are used as pseudo labels Y1 and Y2 respectively, and the pseudo labels Y1 and Y2 are used as supervision signals: pseudo label Y2 is used as the supervision of the first super-resolution target recognition model, and pseudo label Y1 is used as the supervision of the second super-resolution target recognition model. The cross entropy loss function and the Dice loss function are used together to constrain the loss of the first super-resolution target recognition model. and the loss of the second super-resolution object recognition model

[0031]

[0032] where α is a hyperparameter that balances the contribution of the two loss terms; l ce (·,·) is the cross entropy loss function; l dice (·,·) is the Dice loss function; i∈{1,2}; then the total cross-label supervision loss l Y for:

[0033]

[0034] The total loss function of the training strategy for:

[0035]

[0036] Compared with the prior art, the beneficial technical effects of the present invention are:

[0037] The present invention proposes a novel and efficient underwater image super-resolution and target recognition method, which can significantly improve the signal-to-noise ratio and clarity of the image, laying a foundation for improving target recognition performance. At the same time, the present invention also designs an innovative model training strategy to effectively alleviate the problem of scarcity of underwater image data sets, thereby meeting the urgent needs of practical applications. This series of innovative measures can lay a solid foundation for the widespread application of underwater unmanned detection technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a schematic diagram of the architecture of the super-resolution target recognition model proposed in the present invention;

[0039] Figure 2 is a schematic diagram of an orthogonal bidirectional attention module and an attention module in an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of an upper scaler in an embodiment of the present invention;

[0041] Figure 4 is a schematic diagram of an upper scaler in an embodiment of the present invention;

[0042] Figure 5 Schematic diagram of the training strategy in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0044] 1. Overall architecture:

[0045] The architecture of the super-resolution target recognition model proposed in the present invention is as follows: Figure 1 The super-resolution target recognition model of the present invention is based on the classic autoencoder structure, adopts a dense upsampling framework, performs initial feature extraction and deep feature extraction in the encoder part, and performs image reconstruction in the decoder.

[0046] Taking a school of fish as an underwater target, for an input underwater image I containing a school of fish, the image is first downsampled using a convolutional layer (Conv) to extract the initial feature F0, and then F0 is sent to the deep feature extraction part consisting of A orthogonal bidirectional attention modules (Orthogonal Bidirectional Attention, OBA). The features extracted at different levels include F0, F1, F2, ..., F A , are transferred to the decoder for image super-resolution reconstruction. The decoder consists of an upscaler and a convolutional layer Conv. This is the reconstructed image output by the decoder.

[0047] The feature C1 output by the encoder is sent to the bottom-up path in the feature pyramid network (FPN). The bottom-up path uses a deep residual network (ResNet). The top-down path uses a region selection network (RPN) to form a region of interest extraction and then connects it. The connected features are then fused through the convolutional layer to obtain the target classification and target bounding box regression to achieve target recognition.

[0048] 2. Orthogonal Bidirectional Attention Module (OBA):

[0049] like Figure 2 As shown, in the orthogonal bidirectional attention module, convolution operations and attention operations are alternated to further extract features from the input feature F.

[0050] In the attention module, a bidirectional long short-term memory network (LSTM) scans pixels one by one along each row of F from left to right and from right to left, respectively, and then connects the two obtained feature maps to combine the left context information and the right context information of each pixel. Similarly, another bidirectional LSTM is used to scan each column of the obtained feature map in a bottom-up and top-down order. After obtaining the overall feature map by scanning in four directions, a convolutional layer is used to convert the feature map into D channels, where D = w × h, to form a feature weight α, and then the weight generated for each pixel is weighted to obtain the final attention weight F. att For the feature vector p of a pixel at position (w,h) w,h ∈R D , after normalization using the softmax function, we get an attention weight α w,h ∈R D Therefore, the pixel at position (w,h) participates in the context feature F of the same size as F. att , can be obtained by the following formula:

[0051]

[0052] where f w,h ∈R D is the convolution feature at position (w,h) in F, f att The feature at position (w,h) in the image.

[0053] 3. Upscaler:

[0054] like Figure 3 and Figure 4 As shown, the upscaler consists of multiple convolution groups, each of which contains a normal convolution and a sub-pixel convolution. c The input features of feature maps, ordinary convolution Conv(n c ,3,r 2 ·n f ) Increase the number of feature maps to R 2 ·N f Sub-pixel convolution PixelShuffle(r) converts H×W×r 2 ·n f The elements in the feature tensor are rearranged to a size of (r·H)×(r·W)×n f The feature tensor of .

[0055] 4. Training strategy:

[0056] 4.1, Pairwise Similarity Figure 1 Caused loss:

[0057] like Figure 5 As shown in the figure, the present invention uses two super-resolution target recognition models with the same structure and different random initialization methods. For the sake of simplicity, the lower autoencoder network is defined as A(θ1), A(θ2), and the upper feature pyramid network is defined as F(θ1), F(θ2). In the autoencoder network, based on the feature map P from the encoder, e , i.e., feature C1, and the decoder output P d , that is, reconstruct the image Calculate similarity Figure 1 Consistency loss. For a given batch size B, the shapes of these outputs are B×C×W×H and B×C'×W'×H', where H and W are the spatial dimensions of the feature map and C is the number of feature channels. The similarity graph matrix is ​​the Gram matrix, which returns the similarity value between two pairs of inputs based on the network output. In order to reduce the impact of changes between values, the similarity graph is L2 normalized. This calculation results in a pairwise similarity graph matrix (ψ) of shape B×B. That is, for each network:

[0058]

[0059] The training loss is

[0060]

[0061] L D is the root mean square error, ψ1 and ψ2 are the similarity graph matrices generated by networks A(θ1) and A(θ2), respectively.

[0062] resemblance Figure 1 The role of consistency training is equivalent to adding multiple perturbations to the features learned by the encoder and forcing the output of the decoder to be consistent with the output before the perturbation. It is used to cope with complex underwater environments, which in disguise increases the sample size and improves the robustness of the algorithm.

[0063] 4.2, Cross-label Supervision:

[0064] like Figure 5 As shown in the figure, when the two networks output the classification results, they obtain the corresponding classification labels Y1 and Y2, which are used as supervisory signals. The classification label Y2 is used as the supervision of network 1, and Y1 is used as the supervision of network 2, and the cross entropy loss function is used for constraint.

[0065] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0066] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for underwater image super-resolution and target recognition, It is characterized in that The super-resolution target recognition model used includes an encoder, a decoder, and a feature pyramid network; specifically, the following steps are included: Step 1: The encoder includes a convolutional layer and a deep feature extraction part; the convolutional layer is used to downsample the input underwater image I and extract the initial feature F 0 , then F 0 The deep feature extraction part is composed of A orthogonal bidirectional attention modules, and the A orthogonal bidirectional attention modules output the features F extracted at different levels. 1 、F 2 ,…,F A , for F 0 、F 1 、F 2 ,…,F A Splice and get the feature C output by the encoder 1 ; Step 2: The decoder includes an upscaler and a convolutional layer; the feature C 1 The input is fed into the upscaler and convolutional layer in turn to obtain the reconstructed image of the underwater image. Step 3: The feature C output by the encoder 1 Input into the feature pyramid network; the feature pyramid network includes a bottom-up path, a top-down path and a lateral connection path; the bottom-up path uses a deep residual network; the region selection network is used to extract the region of interest from each feature map in the top-down path and align them; after connecting multiple aligned regions of interest, they are input into the convolution layer to obtain the classification result of the target and the bounding box of the target.

2. The underwater image super-resolution and target recognition method according to claim 1, It is characterized in that The orthogonal bidirectional attention module includes a plurality of alternating convolutional layers and attention modules, which are used to alternately perform convolution operations and attention operations on the input feature F; The attention module includes two bidirectional long short-term memory networks. The first bidirectional long short-term memory network scans pixels one by one from left to right and from right to left along each row of feature F to obtain two feature maps. The first bidirectional long short-term memory network splices the two feature maps and outputs them to combine the left context information and the right context information of each pixel. The second bidirectional long short-term memory network scans each column of the feature map in a bottom-up and top-down order along the feature map output by the first bidirectional long short-term memory network, and obtains the overall feature map by splicing; the convolution layer is used to convert the overall feature map into D channels to form a feature weight α, where D = W×H, W, H are the width and height of the input feature F respectively; for the feature vector p of the pixel at position (w, h) in the feature weight α w,h ∈R D , use the convolution layer to change the dimension to C×W×H and then use the softmax function to normalize it to get a new feature weight α w,h ∈R D , and then generate a new feature weight α for each pixel w,h Weighted, we get the context feature F att : New feature weight of the i-th channel Obtained by the following formula: is the feature vector of the pixel at position (w, h) in the feature weight α of the i-th channel; For a pixel at position (w,h), the context feature F of the same size as F is involved att Obtained by the following formula: Among them, f w,h ∈R D is the convolution feature at position (w,h) in F, F att The feature at position (w,h) in the image.

3. The underwater image super-resolution and target recognition method according to claim 1, It is characterized in that The upscaler includes multiple convolution groups, each of which includes a normal convolution and a sub-pixel convolution; let Conv(n c ,k,n f ) is a convolutional layer, where n c ,k,n f are the number of filter channels, kernel size, and number of filters respectively; for n c The input features of feature maps, ordinary convolution Conv(n c ,3,r 2 ·n f ) Increase the number of feature maps of the input features to r 2 ·n f , and the size is H×W×r 2 ·n f The feature tensor of , where r represents the up-scaling factor; H, W represent the height and width of the feature map of the input feature respectively; sub-pixel convolution PixelShuffle(r) converts the size of H×W×r 2 ·n f The elements in the feature tensor of are rearranged to a size of (r·H)×(r·W)×n f The feature tensor of .

4. The underwater image super-resolution and target recognition method according to claim 1, It is characterized in that The training strategy of the super-resolution target recognition model includes: Pairwise similarity graph consistency loss: Two super-resolution target recognition models with the same structure but different initialization methods are used. The encoder and the decoder constitute an autoencoder network. The autoencoder networks of the two super-resolution target recognition models are defined as the first autoencoder network A(θ 1 ), the second autoencoder network A(θ 2 ), the feature pyramid networks of the two super-resolution target recognition models are respectively denoted as the first feature pyramid network F(θ 1 ), the second feature pyramid network F(θ 2 ); A(θ 1 ) and F(θ 1 ) forms the first super-resolution target recognition model, A(θ 2 ) and F(θ 2 ) constitute the second super-resolution target recognition model; In the autoencoder network, the feature map P from the encoder is e , that is, the feature C 1 , and the feature map P output by the decoder d , that is, the reconstructed image Calculate similarity graph consistency loss; For a given batch size B, P e and P d The shapes of are B×C×W×H and B×C'×W'×H', where H and W are the feature maps P e The width and height of the feature map P are shown in Figure 2. d The width and height of the graph, C' is the corresponding number of feature channels; in order to calculate the consistency loss of similar graphs, we first need to obtain the similar graph matrix, that is, the Gram matrix, according to the autoencoder network A(θ 1 ),A(θ 2 ) The output returns the similarity value between the two pairs of inputs; in order to reduce the impact of changes in similarity values, the similarity graph matrix is ​​L2 normalized to obtain a pairwise similarity graph matrix ψ with a shape of B×B, that is, for the i-th super-resolution object recognition model: ‖·‖ 2 represents the bi-norm, i∈{1,2}, ψ 1 and ψ 2 Especially the autoencoder network ψ(θ 1 ),A(θ 2 )The similarity graph matrix generated; The proposed similarity graph consistency loss l sim for: L D (·,·) is the root mean square error, Respectively represent A(θ 1 ),A(θ 2 ) in the feature map P e The similarity graph matrix of Respectively represent A(θ 1 ),A(θ 2 ) Output feature map P d Similarity graph matrix of ; Cross-label supervision: The classification results output by the first super-resolution target recognition model and the second super-resolution target recognition model are used as pseudo labels Y1 and Y2 respectively, and the pseudo labels Y1 and Y2 are used as supervision signals: pseudo label Y2 is used as the supervision of the first super-resolution target recognition model, and pseudo label Y1 is used as the supervision of the second super-resolution target recognition model. The cross entropy loss function and the Dice loss function are used together to constrain the loss of the first super-resolution target recognition model. and the loss of the second super-resolution object recognition model where α is a hyperparameter that balances the contribution of the two loss terms; l ce (·,·) is the cross entropy loss function; l dice (·,·) is the Dice loss function; i∈{1,2}; then the total cross-label supervision loss l Y for: The total loss function of the training strategy for:

Citation Information

Patent Citations

  • Remote sensing image super-resolution reconstruction method and system based on multi-scale network, and medium

    CN118071602A

  • Apparatus for processing image super resolution using graph information and its method

    KR102682543B1

  • Method for enhancing quality and resolution of CT images based on deep learning

    US20240037732A1

Cited By

  • Model training method, underwater target detection method and related device

    CN121259555A