Fish image recognition method and system based on deep learning

Abstract

The invention discloses a fish image recognition method and system based on deep learning, and the method designs a Bilatal-Net underwater image enhancement model which comprises a mixed domain self-attention-based contrast enhancement module, a bilateral feature fusion module and a multi-dimensional loss function module. A contrast enhancement module based on mixed domain self-attention is introduced, a bilateral feature fusion module is designed for detail blurring of an underwater image to enhance detail features of the image, and a multi-dimensional loss function is designed for color deviation of the underwater image. Aiming at the characteristics of large inter-class difference and small intra-class difference of fish images, a FishResNet18 fish recognition model of an attention residual module based on a channel domain and a spatial domain is proposed, extraction of important features in the fish images is stimulated from two aspects of the channel and the spatial domain, the learning effect is improved, meanwhile, the degradation phenomenon of CNN is reduced, and the recognition efficiency is improved. And the recognition efficiency of the fish image is improved from local to global.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and relates to a fish image recognition method and system based on deep learning. Background Art

[0002] Due to the limitations of the special underwater imaging environment, underwater fish images have low contrast, blurred details and color distortion, which affect the efficiency and accuracy of target recognition. In addition, fish images have characteristics such as similarity between classes and large differences within classes, which brings challenges to underwater biological observation and identification. Therefore, the construction of underwater fish image enhancement and recognition technology is of great significance for the construction of marine ranches.

[0003] Existing image enhancement methods can be divided into traditional model-free and model-based methods. They can enhance the contrast of images, retain image details, and improve image color deviation to a certain extent. However, when improving image quality, traditional model-free methods do not take into account the changes in different underwater environments during underwater image imaging, and there are phenomena of over-enhancement or under-enhancement, which cannot be fully suitable for all underwater images; traditional physical model-based methods perform poorly in correcting underwater image color deviation, and need to rely on prior knowledge and manually adjust parameters, which is time-consuming and greatly affected by parameters, and cannot be generalized to complex and changeable underwater scenes. In order to overcome the above limitations, deep learning has been introduced into the field of underwater image enhancement in recent years. By training the network on a large-scale underwater dataset, it can effectively cope with the changing underwater environment. However, when these underwater image enhancement models are used for fish images, there are still different degrees of insufficient enhancement effects. Therefore, it is necessary to improve on the basis of the existing models to improve the efficiency of fish image enhancement. In addition, as deep learning becomes more mature, researchers have also applied deep learning methods to fish image recognition. Convolutional neural networks have been widely recognized in fish recognition tasks. However, the accuracy rate has shown a downward trend in actual recognition tasks. The main reasons for this phenomenon are as follows: (1) Fish images are limited by light imaging, resulting in varying degrees of blur, color cast, and low contrast, which affects recognition efficiency; (2) The current fish image datasets lack data within each class and the number of classes is unbalanced, while deep learning relies on a large amount of robust data. Therefore, overfitting is prone to occur in fish recognition tasks; (3) Fish with large intra-class differences and high inter-class similarities cannot be effectively identified, resulting in recognition errors. Summary of the invention

[0004] The purpose of the present invention is to solve the problems in the prior art and provide a fish image recognition method and system based on deep learning.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention proposes a fish image recognition method based on deep learning, comprising:

[0007] Get enhanced underwater fish images;

[0008] Build the FishResNet18 fish image recognition model based on the ResNet model;

[0009] The enhanced underwater fish images are input into the FishResNet18 fish image recognition model to obtain fish images and realize fish image recognition.

[0010] Preferably, an underwater image enhancement model Bilateral-Net is used to obtain enhanced underwater fish images; the underwater image enhancement model Bilateral-Net is based on CNN, and introduces a contrast enhancement module based on mixed domain self-attention for low image contrast; a bilateral feature fusion module is designed to enhance image detail features for blurred underwater image details; a multidimensional loss function is designed to constrain model parameters from image pixels, image perception, image structure and color balance for underwater image color deviation.

[0011] Preferably, the contrast enhancement module based on mixed domain self-attention consists of three parts: convolution layer, channel self-attention block, and spatial self-attention block. The original underwater image I is input and the features of C×H×W are obtained through the convolution layer. Figure X ; Then, a mixed-domain self-attention mechanism is formed by connecting the channel self-attention block and the spatial self-attention block in series;

[0012] Specific: Features Figure X After the channel self-attention block is input, the SoftMax function first obtains the relative importance Q1 of each pixel in the spatial dimension, then generates the spatial information V1, and finally calculates the channel self-attention weight Z1 and the spatial self-attention weight Z2, and applies the channel self-attention weight Z1 to the feature Figure X Get the input X' of the spatial self-attention block and apply the spatial self-attention weight Z2 to the feature Figure X 'Get the output Z of the contrast enhancement module based on mixed domain self-attention.

[0013] Preferably, the bilateral feature fusion module comprises two branches of a residual connection convolution block and a detail enhancement convolution block, and the features of the two branches are fused to obtain a detail enhanced feature map; wherein the convolution kernel of the detail enhancement convolution block performs a parameter convolution on a channel p of the feature map to obtain an intermediate variable P', and a 1×1×d convolution kernel is used to integrate the intermediate variable P' to obtain the detail feature map O p ; The features of the residual connection convolution block and the detail enhancement convolution block are fused to obtain the final detail enhancement feature map.

[0014] Preferably, the multidimensional loss function integrates image information in four dimensions, namely, image pixels, image perception, image structure, and color information, to adjust the parameters of the model. The multidimensional loss function is expressed as:

[0015] Loss = αL MSE +βL VGG +γL SSIM +L cb

[0016] Where, L MSE is the pixel-wise mean square error loss, L VGG is the perceptual loss, L SSIM is the structural similarity loss function, L cb Represents a color balance term that controls the color stability between image pairs by ensuring that each pixel points to the same direction in the three-dimensional RGB space:

[0017]

[0018] Where λ is a constant factor that adjusts the contribution of the color balance term, Represents image I i is the RGB vector of the j-th pixel, K is the total number of pixels in the image, and α, β, and γ are hyperparameters for adjusting the contribution of pixel, perception, and structure losses.

[0019] Preferably, the FishResNet18 fish image recognition model is constructed based on the ResNet model, specifically:

[0020] The FishResNet18 fish image recognition model constructs an enhancement block based on asymmetric convolution to improve the local feature extraction ability of the model. Then, it combines the attention mechanism and the enhancement block to design an attention residual module based on the channel domain and an attention residual module based on the spatial domain, respectively, to realize the extraction of features in fish images from the channel domain and the spatial domain.

[0021] Preferably, the channel-domain-based attention residual module includes an attention module and an enhancement block. First, the feature map of any layer is input into the attention module. First, through a compression operation, the input features are aggregated in their spatial dimensions to generate a weight that can respond to the global distribution; secondly, the nonlinear dependency between channels is learned by twice excitation of the global distribution weight; finally, the weight of each channel is applied to the input feature map to obtain an output feature map after selective attention, and the output feature map is directly fed into the enhancement block to continue feature extraction;

[0022] Specifically: perform a global average pooling operation on the spatial dimension of each layer of channels to obtain the global information R between C channels, perform two excitations on the global information R between C channels to obtain the weight value s of the feature map, apply the weight value s to the input feature maps M with the number of C, and obtain the recalibrated feature map U, which is directly fed into the enhancement block.

[0023] Preferably, the spatial domain-based attention residual module includes a non-local module and an enhancement block; wherein the non-local module first linearly maps the input feature map through three 1×1 convolutions and compresses the number of channels to obtain the first feature δ, the second feature θ and the third feature φ; the feature map dimension is reconstructed into HW×C / 2 through the Reshape operation, and then the first feature δ and the second feature θ are matrix-dot multiplied to calculate the autocorrelation in the feature, that is, the relationship between each pixel and all other pixels, and the Softmax operation is performed on the correlation to obtain a 0-1 relationship response coefficient; the relationship response coefficient is correspondingly multiplied with the third feature φ to obtain the feature y at the i-th position in the output feature map. i , the feature y at the i-th position in the output feature map i The dimension is reconstructed to H×W×C / 2, using a 1×1 convolution kernel W z The feature y at the i-th position in the output feature map i The dimension is restored to H×W×C, and the features Figure X Perform a residual connection with it to obtain the output Z of the non-local module.

[0024] The present invention proposes a fish image recognition method based on deep learning, comprising:

[0025] An image acquisition module, the image acquisition module is used to acquire enhanced underwater fish images;

[0026] A model building module, wherein the model building module is used to build a FishResNet18 fish image recognition model based on a ResNet model;

[0027] The image processing module is used to input the enhanced underwater fish image into the FishResNet18 fish image recognition model to obtain the fish image and realize fish image recognition.

[0028] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a fish image recognition method based on deep learning when executing the computer program.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The present invention proposes a fish image recognition method based on deep learning. By acquiring enhanced underwater fish images, the quality of fish images can be improved from three aspects: contrast, detail and color. This overcomes the problem that the recognition efficiency is difficult to improve due to the distortion of underwater images and the diversity of fish species. In view of the characteristics of large differences between fish images and small differences within the same category, a FishResNet18 fish recognition model based on the attention residual module in the channel domain and the spatial domain is proposed. The extraction of important features in fish images is stimulated from both the channel and spatial domains. While improving the learning effect, the degradation phenomenon of CNN is reduced, and the recognition efficiency of fish images is improved from local to global. Specifically, the fish image recognition model FishResNet18 is composed of an attention residual module based on the channel and spatial domains. The extraction of important features in fish images is stimulated from both the channel and spatial domains. This improves the feature extraction efficiency from local to global, and overcomes the problem that the differences between underwater fish target optical images are small and the differences within the same category are large.

[0031] Furthermore, the Bilateral-Net underwater image enhancement model is established, including three modules: contrast enhancement module based on hybrid domain self-attention, bilateral feature fusion module and multidimensional loss function. To solve the problem of low image contrast, the model introduces a contrast enhancement module based on hybrid domain self-attention, adaptively learns the image brightness level distribution, suppresses noise, and thus improves image quality; to solve the problem of blurred underwater image details, based on the idea of ​​over-parameterization, a bilateral feature fusion module is designed to enhance image detail features and improve the model's feature extraction ability; finally, to solve the problem of underwater image color deviation, a multidimensional loss function is designed to constrain model parameters from four aspects: image pixels, image perception, image structure, and color balance, to ensure the model's enhancement effect. Then, through four steps of category screening, data expansion and cleaning, data enhancement, and image enhancement, a Fish13-Enh fish image dataset with sufficient number of categories and covering different angles and periods within the class and great similarity between classes is constructed based on the wildfish dataset.

[0032] Furthermore, an enhancement block is constructed based on asymmetric convolution to improve the local feature extraction ability of the model. Then, combined with the attention mechanism and the enhancement block, attention residual modules based on the channel domain and the spatial domain are designed respectively. This stimulates the extraction of important features in fish images from both the channel and spatial domains, while improving the learning effect and reducing the degradation phenomenon of CNN, thereby improving the recognition efficiency of underwater fish images from a local to a global perspective.

[0033] The present invention proposes a fish image recognition system based on deep learning, which acquires fish images and realizes fish image recognition by dividing the system into an image acquisition module, a model building module and an image processing module. The modularization concept is adopted to make each module independent of each other, which is convenient for unified management of each module. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 This is a flow chart of the fish image recognition method based on deep learning of the present invention.

[0036] Figure 2 This is the architecture diagram of the underwater image enhancement model proposed in the present invention.

[0037] Figure 3 This is a structural diagram of a contrast enhancement module based on mixed domain self-attention in the image enhancement model of the present invention.

[0038] Figure 4 It is a structural diagram of the bilateral feature fusion module in the image enhancement model of the present invention.

[0039] Figure 5 This is the fish image recognition model architecture diagram proposed in the present invention.

[0040] Figure 6 It is a structural diagram of the enhancement block in the image recognition model of the present invention.

[0041] Figure 7 This is a structural diagram of the channel domain-based attention residual module in the image recognition model of the present invention.

[0042] Figure 8 It is an algorithm flow chart of the channel attention module in the image recognition model of the present invention.

[0043] Fig. 9 This is a structural diagram of the spatial domain-based attention residual module in the image recognition model of the present invention.

[0044] Fig.10 It is an algorithm flow chart of the non-local module in the attention residual module based on the spatial domain of the present invention.

[0045] Fig.11 This is a diagram of a fish image recognition system based on deep learning according to the present invention.

[0046] Fig.12 The figure is a schematic structural diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0048] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0050] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0051] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0052] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0054] The present invention proposes a fish image recognition method based on deep learning, such as Figure 1 As shown, including:

[0055] S1. Obtain enhanced underwater fish images;

[0056] An underwater image enhancement model Bilateral-Net is used to obtain enhanced underwater fish images; the underwater image enhancement model Bilateral-Net is based on CNN, and introduces a contrast enhancement module based on hybrid domain self-attention for low image contrast; a bilateral feature fusion module is designed to enhance image detail features for blurred underwater image details; a multidimensional loss function is designed to constrain model parameters from image pixels, image perception, image structure and color balance for underwater image color deviation.

[0057] The contrast enhancement module based on mixed domain self-attention consists of three parts: convolution layer, channel self-attention block, and spatial self-attention block. The original underwater image I is input and the C×H×W feature is obtained through the convolution layer. Figure X ; Then, a mixed-domain self-attention mechanism is formed by connecting the channel self-attention block and the spatial self-attention block in series;

[0058] Specific: Features Figure X After the channel self-attention block is input, the SoftMax function first obtains the relative importance Q1 of each pixel in the spatial dimension, then generates the spatial information V1, and finally calculates the channel self-attention weight Z1 and the spatial self-attention weight Z2, and applies the channel self-attention weight Z1 to the feature Figure X Get the input X' of the spatial self-attention block and apply the spatial self-attention weight Z2 to the feature Figure X 'Get the output Z of the contrast enhancement module based on mixed domain self-attention.

[0059] The bilateral feature fusion module consists of two branches: a residual connection convolution block and a detail enhancement convolution block. The features of the two branches are fused to obtain a detail enhanced feature map. The convolution kernel of the detail enhancement convolution block performs a parameter convolution on a channel p of the feature map to obtain an intermediate variable P'. The 1×1×d convolution kernel is used to integrate the intermediate variable P' to obtain the detail feature map O p ; The features of the residual connection convolution block and the detail enhancement convolution block are fused to obtain the final detail enhancement feature map.

[0060] The multidimensional loss function integrates image information in four dimensions: image pixels, image perception, image structure, and color information to adjust the parameters of the model. The multidimensional loss function is expressed as:

[0061] Loss = αL MSE +βL VGG +γL SSIM +L cb

[0062] Where, L MSE is the pixel-wise mean square error loss, L VGG is the perceptual loss, L SSIM is the structural similarity loss function, L cb Represents a color balance term that controls the color stability between image pairs by ensuring that each pixel points to the same direction in the three-dimensional RGB space:

[0063]

[0064] Where λ is a constant factor that adjusts the contribution of the color balance term, Represents image I i is the RGB vector of the j-th pixel, K is the total number of pixels in the image, and α, β, and γ are hyperparameters for adjusting the contribution of pixel, perception, and structure losses.

[0065] S2. Build FishResNet18 fish image recognition model based on ResNet model;

[0066] The FishResNet18 fish image recognition model is constructed based on the ResNet model, specifically:

[0067] The FishResNet18 fish image recognition model constructs an enhancement block based on asymmetric convolution to improve the local feature extraction ability of the model. Then, it combines the attention mechanism and the enhancement block to design an attention residual module based on the channel domain and an attention residual module based on the spatial domain, respectively, to realize the extraction of features in fish images from the channel domain and the spatial domain.

[0068] The channel-domain-based attention residual module includes an attention module and an enhancement block. First, the feature map of any layer is input into the attention module. First, through compression operation, the input features are aggregated in their spatial dimensions to generate a weight that can respond to the global distribution; secondly, the global distribution weight is stimulated twice to learn the nonlinear dependency between channels; finally, the weight of each channel is applied to the input feature map to obtain the output feature map after selective attention, and the output feature map is directly fed into the enhancement block for further feature extraction;

[0069] Specifically: perform a global average pooling operation on the spatial dimension of each layer of channels to obtain the global information R between C channels, perform two excitations on the global information R between C channels to obtain the weight value s of the feature map, apply the weight value s to the input feature maps M with the number of C, and obtain the recalibrated feature map U, which is directly fed into the enhancement block.

[0070] The spatial domain-based attention residual module includes a non-local module and an enhancement block; the non-local module first linearly maps the input feature map through three 1×1 convolutions and compresses the number of channels to obtain the first feature δ, the second feature θ and the third feature φ; the feature map dimension is reconstructed to HW×C / 2 through the Reshape operation, and then the first feature δ and the second feature θ are matrix-multiplied to calculate the autocorrelation in the feature, that is, the relationship between each pixel and all other pixels, and the Softmax operation is performed on the correlation to obtain a 0-1 relationship response coefficient; the relationship response coefficient is multiplied with the third feature φ to obtain the feature y at the i-th position in the output feature map. i , the feature y at the i-th position in the output feature map i The dimension is reconstructed to H×W×C / 2, using a 1×1 convolution kernel W z The feature y at the i-th position in the output feature map i The dimension is restored to H×W×C, and the features Figure X Perform a residual connection with it to obtain the output Z of the non-local module.

[0071] S3. Input the enhanced underwater fish image into the FishResNet18 fish image recognition model to obtain the fish image and realize fish image recognition.

[0072] The following is a detailed description of this method:

[0073] Figure 2 This is the architecture diagram of the Bilateral-Net model for underwater image enhancement proposed in the present invention. The model is based on CNN. To address the problem of low image contrast, a contrast enhancement module based on mixed domain self-attention is introduced to adaptively learn the image brightness level distribution and suppress noise, thereby improving image quality. To address the problem of blurred underwater image details, a bilateral feature fusion module is designed based on the over-parameterization idea to enhance image detail features and improve the feature extraction capability of the model. Finally, to address the problem of underwater image color deviation, a multidimensional loss function is designed to constrain model parameters from four aspects: image pixels, image perception, image structure, and color balance, to ensure the enhancement effect of the model.

[0074] Figure 3This is the structure diagram of the contrast enhancement module based on mixed domain self-attention in the image enhancement model of the present invention. The module consists of three parts: convolution layer, channel self-attention block, and spatial self-attention block. An original underwater image I with a size of 256×256×3 is input, and the features of C×H×W are obtained through the convolution layer. Figure X Then, a mixed-domain self-attention mechanism is formed by connecting the channel self-attention block and the spatial self-attention block in series. Figure X After the channel self-attention block is input, the relative importance Q1 of each pixel in the spatial dimension is first obtained by the SoftMax function, which also has a normalization effect:

[0075] Q1=SoftMax(Reshape(w 1×1 *X)) (1)

[0076] In the formula, w 1×1 Represents a 1×1 convolution kernel;

[0077] Then generate the spatial information V1, and ensure that each channel maintains a high resolution while obtaining spatial information:

[0078] V1=Reshape(w 1×1 *X) (2)

[0079] In the formula, Reshape(.) means reconstructing the feature map of C / 2×H×W into C / 2×HW;

[0080] Finally, calculate the channel self-attention weight Z1:

[0081] Z1=Sigmoid(LN(w 1×1 *(Q1×V1))) (3)

[0082] In the formula, × represents matrix multiplication, LN(.) represents normalization, and restores the channel dimension to C, and applies the channel self-attention weight to the feature Figure X The input X' of the spatial self-attention block is obtained. The calculation of the spatial and channel self-attention blocks is similar. The spatial self-attention weight Z2 is obtained by calculating the relative weight Q2 of the channel dimension and the channel information V2:

[0083] Z2=Sigmoid(Reshape(Q2×V2)) (4)

[0084] In the formula, Reshape(.) reshapes the output size to 1×H×W, and Sigmoid(.) outputs the spatial self-attention weight. Apply the spatial self-attention weight to the feature Figure X 'Get the output Z of the contrast enhancement module based on mixed domain self-attention.

[0085] Figure 4 This is a structural diagram of the bilateral feature fusion module in the image enhancement model of the present invention. The module consists of two branches including a residual connection convolution block and a detail enhancement convolution block. The features of the two branches are fused to obtain a detail enhanced feature map. In the detail enhancement convolution block, the ordinary convolution operation is optimized, and the method of sliding feature extraction of the entire feature map using multiple convolutions is transformed into using a large number of different convolutions to extract features from a single channel of the feature map and then reintegrating to obtain a feature map with rich details. The convolution kernel performs a parameter convolution on a channel p of the feature map to obtain an intermediate variable P':

[0086] P' = concat(wi 3×3 *p) (5)

[0087] Among them, * represents the convolution symbol, concat(.) represents the feature map after concatenating d times of convolution, wi 3×3 represents a 3×3 convolution kernel. Then, a 1×1×d convolution kernel is used to integrate the intermediate variable P' to obtain the detail feature map O p :

[0088] O p =w 1×1×d *P' (6)

[0089] Among them, w 1×1×d Represents a 1×1×d convolution kernel. According to the characteristics of convolution linear operation, the above convolution can be equivalent to a detail enhancement convolution kernel with rich detail features. Therefore, for the feature with input channel number C Figure X , the above detail enhancement process can be expressed as:

[0090] O=W C×(3×3)×Cout *X=(Wi C×(3×3)×d *W C×d×Cout )*X (7)

[0091] Where O represents the detailed features after being extracted and reintegrated by d convolution kernels, and W C×(3×3)×Cout is the detail enhancement convolution kernel, Wi C×(3×3)×d represents the C×(3×3)×d convolution kernel, W C×d×Cout Represents the C×d×Cout convolution kernel, and Cout represents the number of channels of the output feature map.

[0092] All convolutional layers in the residual connection convolution block and the detail enhancement convolution block use 3×3 common convolution kernels and detail enhancement convolution kernels respectively, and jump-connect the original input image to each convolution block to prevent network overfitting and improve the generalization performance of the network. At the same time, it ensures that each convolution block can combine the basic features of the original image in the process of learning features, and finally fuses the features of the two branches to obtain the final detail enhancement feature map.

[0093] The present invention designs a multidimensional loss function, which integrates the image information of four dimensions, namely, image pixels, image perception, image structure and color information, to adjust the parameters of the model, so as to ensure that the network model can enhance underwater images in a targeted manner. The multidimensional loss function is expressed as:

[0094] Loss = αL MSE +βL VGG +γL SSIM +L cb (8)

[0095] Where, L MSE is the pixel-by-pixel mean square error loss. By minimizing the loss between the enhanced image and the label image, the mapping learning that preserves the edge and detail sharpness is achieved. VGG Based on the pre-trained VGG19 network definition, the feature representations of the enhanced image and the reference image in the network are obtained respectively, and the L1 distance between the two is used as the perceptual loss. SSIM is the structural similarity loss function, L cb Represents a color balance term that controls the color stability between image pairs by ensuring that each pixel points to the same direction in the three-dimensional RGB space:

[0096]

[0097] Where λ is a constant factor that adjusts the contribution of the color balance term, Represents image I i The RGB vector of the j-th pixel, Indicates the enhanced image I i where α, β, and γ are the hyperparameters for adjusting the contribution of pixel, perception, and structure losses, which are 0.2, 0.6, and 0.2, respectively.

[0098] Finally, based on the wildfish dataset, the present invention constructs the Fish13-Enh fish image dataset, which has sufficient number of categories and covers different angles and periods within the category and has great similarity between categories. Specifically, it is carried out through four steps: category screening, data expansion and cleaning, data enhancement, and image enhancement. Firstly, the categories with the data volume within the class greater than 100 and the characteristics of small inter-class difference and large intra-class difference are counted and screened out from the WildFish data set. Then, image expansion data sets are downloaded from Bing, Google and various professional websites according to the screened categories. Then, the expanded data are cleaned according to at least 5 prototype images and professional descriptions of each category in the image set of professional websites, and images irrelevant to the category are deleted. Finally, the data is enhanced by flipping and rotating at different angles. After the above processing, the data set has sufficient number of categories and covers the morphology of fish at different angles and different periods. However, due to the limitations of the underwater image imaging environment, the image still has partial distortion in contrast, clarity and color. Therefore, the Bilateral-Net underwater image enhancement model designed by the present invention is used to improve the quality of the fish data set from three aspects of image contrast, detail and color correction. After the above four steps, a high-quality fish image data set Fish-Enh containing 13 categories and a total of 38,456 images is obtained.

[0099] Figure 5 This is the architecture diagram of the fish image recognition model FishResNet18 proposed in this invention, which is improved based on ResNet. First, the enhancement block is constructed based on asymmetric convolution to improve the local feature extraction ability of the model. Then, the attention residual module based on the channel domain and the spatial domain is designed respectively by combining the attention mechanism and the enhancement block. The extraction of important features in fish images is stimulated from both the channel and spatial domains. While improving the learning effect, the degradation phenomenon of CNN is reduced, and the recognition efficiency of underwater fish images is improved from local to global.

[0100] Figure 6 This is the enhanced block structure diagram in the image recognition model of the present invention. The structure is a jump connection formed by an enhanced kernel. The enhanced kernel is used to replace the ordinary convolution and two residual structures are established. In the figure, the input feature map size is H×W×C0, and the output size after the enhanced block is H i ×W i ×C i C0 and C i Represent the number of input channels and output channels respectively. The enhancement kernel consists of three parallel layers, which use d×d, 1×d and d×1 convolution kernels to slide and extract features, and add their outputs to enrich the feature space. Non-square kernels such as 1×d and d×1 are called asymmetric convolution layers. The skeleton information of the convolution kernel is enhanced by adding the asymmetric kernel in each enhancement kernel to the corresponding position of the square kernel.

[0101] The additivity of convolution is defined as in Equation 10. If there are p 2D kernels K with compatible sizes p , for the same input I, sliding convolution with the same step size produces outputs of the same resolution. Assuming that the result of adding these outputs is A, all K p The corresponding positions are added to get K. Convolution of I also gives the output A, so K is called K p The equivalent core of .

[0102]

[0103] Due to the additivity of convolution, the output addition operation of the three parallel layers in the enhanced kernel can equivalently replace the convolution operation in the original network structure. In addition to enhancing the learning effect of the network, the enhanced kernel does not introduce hyperparameters and does not require additional computational burden compared to the original ResNet.

[0104] Figure 7 This is the structural diagram of the channel-domain-based attention residual module in the image recognition model of the present invention. This structure embeds the channel-domain-based attention module into different layers of the network by combining the attention module and the enhancement block, thereby improving the learning effect of each stage of the network. First, the feature map passes through an attention module, that is, the feature map of any layer is input into the attention module, and the selected output is obtained through three steps. First, through the compression operation, the input features are aggregated in their spatial dimensions to generate a weight that can respond to the global distribution. The second step is the excitation operation, which excites the global distribution weight twice to learn the nonlinear dependency between channels, excite valuable features, and suppress useless or unimportant features. Finally, the weights of each channel are applied to the input feature map to obtain the output feature map after selective attention. The output feature map can be directly fed into the enhancement block to continue feature extraction.

[0105] Figure 8 It is the algorithm flow chart of the channel attention module in the image recognition model of the present invention. The compression operation in the first step of the algorithm flow can be understood as a global average pooling operation on the spatial dimension of each layer of channels, that is, for the input feature map of size H×W and number C Perform global pooling to obtain the global information R between C channels, where m1 and m2 refer to the first and second feature maps of the input.

[0106]

[0107] In the formula, m c (x,y) represents the feature map m of the Cth channel c The value at position coordinates (x,y).

[0108] Secondly, in order to limit the complexity of the model while ensuring the generalization ability of the model, R is stimulated twice to obtain the weight value s of the feature map:

[0109] s=σ(g(R,W))=σ(W2δ(W1R)) (12)

[0110] In the formula, δ represents the ReLU function, σ represents the Sigmoid function, W1 represents the weight of the first fully connected layer, the dimension is C / r, r is the scaling factor for reducing the number of channels, which is used to reduce the amount of calculation and reduce the complexity of the model, W2 represents the weight of the second fully connected layer, the dimension is C, and the dimension of s is 1×1×C.

[0111] Finally, the weight s is applied to M to obtain the recalibrated feature map U, which can be directly fed into the enhancement block.

[0112] U c =s c ·m c (13)

[0113] When the attention module is in the shallow layer of the network, it works in a category-independent way, stimulating information characteristics and strengthening shared universal features. When the network is deep, the attention module tends to be specialized and responds to different inputs in a class-specific way. FishResNet18 connects the above attention module and enhancement block three times alternately to form three channel-domain-based attention residual modules, which are embedded in different layers of the network, stimulating valuable features, improving learning effects, and reducing the degradation of CNN.

[0114] Fig. 9 This is a diagram of the structure of the spatial domain-based attention residual module in the image recognition model of the present invention. The structure combines the non-local module and the enhancement block to embed the spatial domain-based attention module into the network and strengthen the learning between the global information of the image. First, the feature map output by the enhancement kernel is input into the non-local module, and the global feature response is added to the network structure to avoid the loss of important information in the global feature during the local convolution process. After the feature screening of the non-local module, the feature map with global useful information is fed to the next layer of the network.

[0115] Fig.10 It is an algorithm flow chart of the non-local module in the attention residual module based on the spatial domain of the present invention. The non-local module realizes the calculation of the weight response of each position of the feature map through matrix multiplication. The algorithm flow is as follows:

[0116] (1) First, three 1×1 convolutions W are performed δ , W θ and W φ The input feature map is linearly mapped and the number of channels is compressed to obtain three features of δ, θ, and φ, where Wδ , W θ and W φ Represent three 1×1 convolutions for outputting the three features of δ, θ, and φ respectively;

[0117] (2) Reshape the feature map to HW×C / 2, then perform matrix dot multiplication on features δ and θ to calculate the autocorrelation of the features, that is, the relationship between each pixel and all other pixels, and perform Softmax operation on the correlation to obtain a 0-1 relationship response coefficient.

[0118] (3) Multiply the relationship response coefficient by the characteristic φ to obtain y i , and y i The dimension of is reconstructed to H×W×C / 2.

[0119] (4) According to the mathematical model of the non-local module, finally, a 1×1 convolution kernel W is used z y i The dimension is restored to H×W×C, and the features Figure X A residual connection is performed with it to obtain the output Z of the non-local module. At this time, the feature map Z has richer global and local features.

[0120] The FishResNet18 model contains three channel-domain-based attention residual modules and one spatial-domain-based attention residual module. The spatial-domain-based attention residual module is placed after the first channel-domain-based attention residual module. The model stimulates the extraction of important features in fish images from both the channel and spatial domains. The channel attention module is connected in series with the enhancement block to form a channel-domain-based attention residual module, which works together in the shallow and deep layers of the network. In the shallow layer of the network, it stimulates information-rich features in a class-independent way and strengthens shared universal features. In the deep layer of the network, it tends to be specialized and abstract, and responds to different inputs in a highly class-specific way. The non-local module is connected and embedded in the enhancement block through two enhancement kernels to form a spatial-domain-based attention residual module. In the process of the enhancement kernel repeatedly extracting local image features, global feature responses are added to maximize the enhancement of global features and solve the problem of undifferentiated feature extraction of the entire image information. With the joint action of the enhancement block and the attention residual module, the model can positively stimulate valuable features and suppress unimportant features. While improving the learning effect, it reduces the degradation phenomenon of CNN and improves the efficiency of feature extraction from local to global.

[0121] Therefore, in order to overcome the shortcomings of the prior art, the present invention proposes a Bilateral-Net underwater image enhancement model to improve the quality of fish images and constructs a high-quality fish dataset Fish-Enh in order to address the problems of fish image distortion and insufficient sample size in each category of the data set. In view of the large differences between fish image categories and small differences within categories, the present invention proposes a FishResNet18 fish recognition model based on an attention residual module in the channel domain and the spatial domain, which stimulates the extraction of important features in fish images from both the channel and spatial domains, while improving the learning effect and reducing the degradation of CNN, thereby improving the recognition efficiency of fish images from a local to a global perspective. The CNN-based Bilateral-Net image enhancement model and the underwater target image recognition model FishResNet18, which is based on an improved ResNet, improve the quality of fish images while improving the recognition efficiency of underwater fish images from a local to a global perspective.

[0122] Example 2

[0123] The present invention proposes a fish image recognition system based on deep learning, such as Fig.11 As shown, including:

[0124] An image acquisition module, the image acquisition module is used to acquire enhanced underwater fish images;

[0125] A model building module, wherein the model building module is used to build a FishResNet18 fish image recognition model based on a ResNet model;

[0126] The image processing module is used to input the enhanced underwater fish image into the FishResNet18 fish image recognition model to obtain the fish image and realize fish image recognition.

[0127] Example 3

[0128] See also Fig.12 As shown, the present invention also provides an electronic device 100 for a fish image recognition method based on deep learning; the electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on the at least one processor 102, and at least one communication bus 104.

[0129] The memory 101 can be used to store the computer program 103, and the processor 102 implements the steps of the fish image recognition method based on deep learning described in Example 1 by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101. The memory 101 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data (such as audio data) created according to the use of the electronic device 100, etc. In addition, the memory 101 can include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices.

[0130] The at least one processor 102 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 102 may be a microprocessor or any conventional processor, etc. The processor 102 is the control center of the electronic device 100, and uses various interfaces and lines to connect various parts of the entire electronic device 100.

[0131] The memory 101 in the electronic device 100 stores a plurality of instructions to implement a fish image recognition method based on deep learning, and the processor 102 can execute the plurality of instructions to implement:

[0132] Get enhanced underwater fish images;

[0133] Build the FishResNet18 fish image recognition model based on the ResNet model;

[0134] The enhanced underwater fish images are input into the FishResNet18 fish image recognition model to obtain fish images and realize fish image recognition.

[0135] Example 4

[0136] If the module / unit integrated in the electronic device 100 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory and read-only memory (ROM, Read-Only Memory).

[0137] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0138] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0139] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A fish image recognition method based on deep learning, characterized in that: include: Get enhanced underwater fish images; Build the FishResNet18 fish image recognition model based on the ResNet model; The enhanced underwater fish images are input into the FishResNet18 fish image recognition model to obtain fish images and realize fish image recognition.

2. The fish image recognition method based on deep learning according to claim 1, characterized in that The underwater image enhancement model Bilateral-Net is used to obtain enhanced underwater fish images; The underwater image enhancement model Bilateral-Net is based on CNN. It introduces a contrast enhancement module based on hybrid domain self-attention to address low image contrast. It designs a bilateral feature fusion module to enhance image detail features to address the blurring of underwater image details. It also designs a multidimensional loss function to constrain model parameters from image pixel points, image perception, image structure, and color balance to address underwater image color deviation.

3. The fish image recognition method based on deep learning according to claim 2, characterized in that The contrast enhancement module based on mixed-domain self-attention consists of three parts: a convolutional layer, a channel self-attention block, and a spatial self-attention block. The original underwater image I is input and a C×H×W feature map X is obtained through the convolutional layer. Then, a mixed-domain self-attention mechanism is formed by connecting the channel self-attention block and the spatial self-attention block in series. Specifically: After the feature map X is input into the channel self-attention block, the SoftMax function is first used to obtain the relative importance Q1 of each pixel in the spatial dimension, and then the spatial information V1 is generated. Finally, the channel self-attention weight Z1 and the spatial self-attention weight Z2 are calculated. The channel self-attention weight Z1 is applied to the feature map X to obtain the input X' of the spatial self-attention block, and the spatial self-attention weight Z2 is applied to the feature map X' to obtain the output Z of the contrast enhancement module based on mixed domain self-attention.

4. The fish image recognition method based on deep learning according to claim 2, characterized in that The bilateral feature fusion module consists of two branches: a residual connection convolution block and a detail enhancement convolution block. The features of the two branches are fused to obtain a detail enhanced feature map. The convolution kernel of the detail enhancement convolution block performs a parameter convolution on a channel p of the feature map to obtain an intermediate variable P'. The 1×1×d convolution kernel is used to integrate the intermediate variable P' to obtain the detail feature map O p ; The features of the residual connection convolution block and the detail enhancement convolution block are fused to obtain the final detail enhancement feature map.

5. The fish image recognition method based on deep learning according to claim 2, characterized in that, The multidimensional loss function integrates image information in four dimensions: image pixels, image perception, image structure, and color information to adjust the model parameters. The multidimensional loss function is expressed as: Loss=αL MSE +βL VGG +γL SSIM +L cb Where, L MSE is the pixel-wise mean square error loss, L VGG is the perceptual loss, L SSIM is the structural similarity loss function, L cb Represents a color balance term that controls color stability between image pairs by ensuring that each pixel points to the same direction in the three-dimensional RGB space: Where λ is a constant factor that adjusts the contribution of the color balance term, I i j Represents image I i is the RGB vector of the j-th pixel, K is the total number of pixels in the image, and α, β, and γ are hyperparameters for adjusting the contribution of pixel, perception, and structure losses.

6. The fish image recognition method based on deep learning according to claim 1, characterized in that The FishResNet18 fish image recognition model is constructed based on the ResNet model, specifically: The FishResNet18 fish image recognition model constructs an enhancement block based on asymmetric convolution to improve the model's local feature extraction ability. Then, combining the attention mechanism and the enhancement block, it designs an attention residual module based on the channel domain and an attention residual module based on the spatial domain, respectively, to extract features from fish images from the channel domain and the spatial domain.

7. The fish image recognition method based on deep learning according to claim 6, characterized in that The channel-domain-based attention residual module includes an attention module and an enhancement block. First, the feature map of any layer is input into the attention module. First, through the compression operation, the input features are aggregated in their spatial dimensions to generate a weight that can respond to the global distribution. Secondly, the nonlinear dependencies between channels are learned by twice-stimulating the global distribution weights. Finally, the weights of each channel are applied to the input feature map to obtain the output feature map after selective attention. The output feature map is directly fed into the enhancement block for further feature extraction. Specifically: A global average pooling operation is performed on the spatial dimension of each layer of channels to obtain the global information R between C channels. The global information R between C channels is stimulated twice to obtain the weight value s of the feature map. The weight value s is applied to the input feature maps M with the number of C to obtain the recalibrated feature map U, which is directly fed into the enhancement block.

8. The fish image recognition method based on deep learning according to claim 6, characterized in that The spatial domain-based attention residual module includes a non-local module and an enhancement block; the non-local module first linearly maps the input feature map through three 1×1 convolutions and compresses the number of channels to obtain the first feature δ, the second feature θ and the third feature φ; the feature map dimension is reconstructed to HW×C / 2 through the Reshape operation, and then the first feature δ and the second feature θ are matrix-multiplied to calculate the autocorrelation in the feature, that is, the relationship between each pixel and all other pixels, and the Softmax operation is performed on the correlation to obtain a 0-1 relationship response coefficient; the relationship response coefficient is multiplied with the third feature φ to obtain the feature y at the i-th position in the output feature map. i , the feature y at position i in the output feature map i The dimension is reconstructed to H×W×C / 2, using a 1×1 convolution kernel W z The feature y at position i in the output feature map i The dimension of is restored to H×W×C, and the feature map X is residually connected to it to obtain the output Z of the non-local module.

9. A fish image recognition method based on deep learning, characterized in that: include: An image acquisition module, configured to acquire enhanced underwater fish images; A model construction module, wherein the model construction module is used to construct a FishResNet18 fish image recognition model based on the ResNet model; The image processing module is used to input the enhanced underwater fish image into the FishResNet18 fish image recognition model to obtain the fish image and realize fish image recognition.

10. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the fish image recognition method based on deep learning as described in any one of claims 1 to 8 are implemented.

Images