Underwater fish image recognition and detection method
By improving the rt-detr model, introducing a parallel structure of triple attention module and combining the gating mechanism, the problem of unsatisfactory fish recognition rate and accuracy in underwater environments is solved, and high accuracy in complex scenarios is achieved Underwater fish detection.
Patent Information
- Application Number
- CN202410910819.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-09
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-07-09
AI Technical Summary
The prior art has poor recognition and accuracy of underwater fish in underwater environments, making it difficult to cope with the detection needs of complex scenarios.
Adopting the rt-detr model and introducing the triple attention module, a parallel upsampling and downsampling branch structure combined with the gating mechanism, and a parallel expansion convolutional structure, improve the underwater fish image recognition and detection model, improve feature expression ability and accuracy of detection sex.
In complex underwater environments, the accuracy and detection capabilities of underwater fish image recognition are significantly improved, allowing underwater fish detection equipment to achieve high-accuracy detection in complex scenarios.
Smart Images
Figure CN118629070B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image-based recognition and detection, and more specifically, relates to an underwater fish image recognition and detection method. Background Art
[0002] Underwater fish identification is one of the important links in the detection of freshwater and marine fish resources, and is also an important basis for the development and utilization of the above resources. Taking marine resources as an example, due to the different shapes and sizes of marine fish, the complexity of their identification is relatively high. Referring to the patent document with Chinese patent publication number CN111523612A, it discloses a deep-sea fish image classification and recognition method, and specifically discloses the following technical scheme: comprising the following steps: step 1, according to the amount of information on the fin position, the amount of information on the size, the amount of information on the tail shape and the amount of information on the body color of deep-sea fish, a multi-source information database of fish status is established, and the similarity of the fish status is divided into 0~1; step 2, a three-layer neural network based on the BP algorithm is established; step 3, on the basis of the neural network tool, the parameters of each item are set, and the neural network is trained and verified.
[0003] Continuing to refer to the patent document with Chinese patent publication number CN110796197A, it discloses a fish species identification method and system, and specifically discloses the following technical solution: It solves the problem in the prior art that identification can only rely on manual work, which is inefficient, labor-intensive, and has inaccurate identification results. The method includes obtaining an image of a fish to be tested; inputting the image into a pre-trained fish species identification model to extract feature information of the image, and identifying the fish to be tested based on the feature information to obtain a fish species identification result of the fish to be tested; and outputting the fish species identification result.
[0004] However, in the above-mentioned prior art, due to the complex underwater environment and the differences in the postures of the fish themselves, the recognition rate and accuracy of fish are not ideal. Therefore, the applicant believes that there is still room for further technical exploration in the accurate identification of underwater fish, so as to improve the recognition accuracy and the detection capabilities for complex underwater scenes. Summary of the invention
[0005] The purpose of this application is to provide an underwater fish image recognition and detection method, which can improve the recognition and detection accuracy of underwater fish on the basis of existing detection technology and enhance the detection capability in complex scenes.
[0006] To achieve the above objectives, this application is implemented through the following technical solutions:
[0007] The present invention discloses an underwater fish image recognition and detection method, which comprises the following steps:
[0008] S1. Obtain the DeepFish underwater fish public dataset and divide it into training set, validation set, and test set in a ratio of 7:2:1;
[0009] S2, select the rt-detr model as the backbone network, and introduce the triple attention module to improve the last two residual modules in the rt-detr model;
[0010] S3. Use a parallel upsampling branch structure combined with a gating mechanism to improve the first upsampling structure of the neck network of the original RT-DETR model, and use a parallel downsampling branch structure combined with a gating mechanism to improve the second upsampling structure of the neck network of the original RT-DETR model;
[0011] S4, the RepC3 structure of the neck network of the original rt-detr model is improved by combining the parallel dilated convolution structure with different dilation rates and the original convolution structure;
[0012] S5, training the rt-detr model improved in step S4 using the training set in step S1 to obtain an underwater fish image recognition and detection model;
[0013] S6, testing the underwater fish image recognition detection model obtained in step S5 on a validation set, and adjusting the underwater fish image recognition detection model according to the test results;
[0014] S7, testing the underwater fish image recognition detection model adjusted in step S6 using a test set;
[0015] S8, deploying the underwater fish image recognition detection model tested in step S7 in the detection device, and collecting underwater fish images through the underwater image acquisition device;
[0016] S9. The underwater fish images collected by the underwater image acquisition device are imported into the underwater fish image recognition detection model. The underwater fish image recognition detection model outputs the recognition result, and outputs the detection result through the display device or sound device in the detection device.
[0017] As one of the preferred technical solutions of the present application, the triple attention module described in step S2 uses three different branches to process the input features from different dimensions, and then improves the network performance through cross-information fusion.
[0018] As one of the preferred technical solutions of the present application, the triple attention module includes a longitudinal attention module, a deep attention module, and a lateral attention module. For an input feature map of c×w×h consisting of the number of channels c, the feature map width w, and the feature map height h, the processing process is as follows:
[0019] S21, the vertical attention module transforms the input feature map in the width dimension, and the dimension of the transformed feature map becomes w×h×c. A 1×1 convolutional neural network is used to calculate the vertical attention weight, and the calculated weight map is multiplied element by element with the original input feature map, and the reconstructed input feature map dimension c×w×h is output;
[0020] S22, the deep attention module maintains the original dimension of the input feature map c×w×h, uses a 1×1 convolutional neural network to calculate the deep attention weight, multiplies the calculated weight map by the original input feature map element by element, and outputs the reconstructed feature map dimension c×w×h;
[0021] S23, the horizontal attention module transforms the input feature map in the height dimension. The dimension of the transformed feature map is h×w×c. A 1×1 convolutional neural network is used to calculate the horizontal attention weight. The calculated weight map is multiplied element by element with the original input feature map. The dimension of the reconstructed feature map is c×w×h.
[0022] S24. The reconstructed feature maps output by steps S21, S22, and S23 are fused element by element by adding each other to obtain a c×w×h triple attention output feature map.
[0023] As one of the preferred technical solutions of the present application, the parallel upsampling branch structure combined with the gating mechanism and the parallel downsampling branch structure combined with the gating mechanism described in step S3 respectively perform upsampling and downsampling operations on the three attention output feature maps output by step S24, and suppress irrelevant features by combining the gating mechanism to improve the feature expression capability.
[0024] As one of the preferred technical solutions of the present application, in the parallel upsampling branch structure combined with the gating mechanism, according to the number of images n used in each iterative training, a feature map with an input dimension of n×c×w×h is obtained, and an upsampling operation is performed. The steps of the upsampling operation are as follows:
[0025] S301, a feature map with a dimension of n×c×w×h is subjected to global average pooling in the first branch to obtain a feature map with an output of n×c×1×1, and the feature map of n×c×1×1 is processed by a nonlinear activation function combined with a gating mechanism to obtain a first branch output feature map of n×c×1×1;
[0026] S302, the feature map with a dimension of n×c×w×h is processed by a common upsampling structure and a 1×1 convolution in the second branch to obtain a second branch output feature map of n×c / 2×2w×2h with doubled spatial dimensions and half the number of channels;
[0027] S303, the feature map with a dimension of n×c×w×h is processed by the container module in the third branch to obtain a third branch output feature map with a dimension of n×c / 2×2w×2h with doubled spatial dimension and half the number of channels;
[0028] S304, concatenating the second branch output feature map and the third branch output feature map along the channel dimension to obtain an n×c×2w×2h output feature map;
[0029] S305. Multiply the output feature map with a dimension of n×c×2w×2h output in step S304 by the first branch output feature map with a dimension of n×c×1×1 output in step S301 element by element to obtain an up-sampled output feature map with a dimension of n×c×2w×2h after the up-sampling operation.
[0030] As one of the preferred technical solutions of the present application, in the parallel downsampling branch structure combined with the gating mechanism, according to the number of images n used in each iterative training, a feature map with an input dimension of n×c×w×h is obtained, and a downsampling operation is performed. The steps of the downsampling operation are as follows:
[0031] S306, subjecting the feature map with a dimension of n×c×w×h to global average pooling in the first branch to obtain an output feature map of n×c×1×1, and processing the n×c×1×1 feature map with a nonlinear activation function combined with a gating mechanism to obtain an output feature map of the first branch of n×c×1×1;
[0032] S307, the feature map with a dimension of n×c×w×h is subjected to maximum pooling processing in the second branch to obtain a feature map of n×c×w / 2×h / 2 with a spatial dimension halved, and the feature map of n×c×w / 2×h / 2 is further subjected to 1×1 convolution processing to obtain a second branch output feature map of n×c / 2×w / 2×h / 2 with an output channel halved;
[0033] S308, the feature map with a dimension of n×c×w×h is processed by the container module in the third branch to obtain a third branch output feature map with a dimension of n×c / 2×w / 2×h / 2;
[0034] S309, concatenating the feature map output by the second branch and the feature map output by the third branch along the channel dimension to obtain an output feature map of n×c×w / 2×h / 2;
[0035] S310. Multiply the output feature map of dimension n×c×w / 2×h / 2 in step S309 by the first branch output feature map of dimension n×c×1×1 in step S306 element by element to obtain a down-sampled output feature map of dimension n×c×w / 2×h / 2 obtained after the final down-sampling operation.
[0036] As one of the preferred technical solutions of the present application, the improved rt-detr model in step S5 is trained using the training set in step S1, the training parameters are epoch=200, the input image size is 640*640 pixels; 4 groups of training sample iterations are used; the optimizer is AdamW.
[0037] Compared with the prior art, the beneficial effects of this application are:
[0038] The present application can improve the recognition accuracy of underwater fish images in complex underwater environments, so that underwater fish detection equipment can improve detection capabilities in complex scenes in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a principle flow chart of this application.
[0040] Figure 2 It is a principle structure diagram of the triple attention module described in this application.
[0041] Figure 3 It is the improved RepC3 map in this application.
[0042] Figure 4 This is the dilated convolution module diagram in this application.
[0043] Figure 5 This is a diagram of the upsampling structure combined with the gating mechanism in this application.
[0044] Figure 6 This is a diagram of the downsampling structure combined with the gating mechanism in this application.
[0045] Figure 7 is the original image.
[0046] Figure 8 is the detection image of the original rt-detr model.
[0047] Fig. 9 It is the detection image of this application. DETAILED DESCRIPTION
[0048] The technical solution described in this application is further described below in conjunction with the accompanying drawings and embodiments.
[0049] Example 1
[0050] See also Figure 1 , a method for underwater fish image recognition and detection, the method comprising the following steps:
[0051] S1. Obtain the DeepFish underwater fish public dataset and divide it into training set, validation set, and test set in a ratio of 7:2:1;
[0052] S2, select the rt-detr model as the backbone network, and introduce the triple attention module to improve the last two residual modules in the rt-detr model;
[0053] S3. Use a parallel upsampling branch structure combined with a gating mechanism to improve the first upsampling structure of the neck network of the original RT-DETR model, and use a parallel downsampling branch structure combined with a gating mechanism to improve the second upsampling structure of the neck network of the original RT-DETR model;
[0054] S4, the RepC3 structure of the neck network of the original rt-detr model is improved by combining the parallel dilated convolution structure with different dilation rates and the original convolution structure;
[0055] S5, training the rt-detr model improved in step S4 using the training set in step S1 to obtain an underwater fish image recognition and detection model;
[0056] S6, testing the underwater fish image recognition detection model obtained in step S5 on a validation set, and adjusting the underwater fish image recognition detection model according to the test results;
[0057] S7, testing the underwater fish image recognition detection model adjusted in step S6 using a test set;
[0058] S8, deploying the underwater fish image recognition detection model tested in step S7 in the detection device, and collecting underwater fish images through the underwater image acquisition device;
[0059] S9. The underwater fish images collected by the underwater image acquisition device are imported into the underwater fish image recognition detection model. The underwater fish image recognition detection model outputs the recognition result, and outputs the detection result through the display device or sound device in the detection device.
[0060] Example 2
[0061] See also Figures 2 to 6 , a method for underwater fish image recognition and detection, the triple attention module described in step S2 uses three different branches to process input features from different dimensions, and then improves network performance through cross-information fusion. The triple attention module includes a longitudinal attention module, a depth attention module, and a lateral attention module. For an input feature map of c×w×h consisting of a channel number c, a feature map width w, and a feature map height h, the processing process is as follows:
[0062] S21, the vertical attention module transforms the input feature map in the width dimension, and the dimension of the transformed feature map becomes w×h×c. A 1×1 convolutional neural network is used to calculate the vertical attention weight, and the calculated weight map is multiplied element by element with the original input feature map, and the reconstructed input feature map dimension c×w×h is output;
[0063] S22, the deep attention module maintains the original dimension of the input feature map c×w×h, uses a 1×1 convolutional neural network to calculate the deep attention weight, multiplies the calculated weight map by the original input feature map element by element, and outputs the reconstructed feature map dimension c×w×h;
[0064] S23, the horizontal attention module transforms the input feature map in the height dimension. The dimension of the transformed feature map is h×w×c. A 1×1 convolutional neural network is used to calculate the horizontal attention weight. The calculated weight map is multiplied element by element with the original input feature map. The dimension of the reconstructed feature map is c×w×h.
[0065] S24. The reconstructed feature maps output by steps S21, S22, and S23 are fused element by element by adding each other to obtain a c×w×h triple attention output feature map.
[0066] The parallel upsampling branch structure combined with the gating mechanism and the parallel downsampling branch structure combined with the gating mechanism described in step S3 respectively perform upsampling operations and downsampling operations on the three attention output feature maps output by step S24, and suppress irrelevant features by combining the gating mechanism to improve the feature expression capability.
[0067] In the parallel upsampling branch structure combined with the gating mechanism, a feature map with an input dimension of n×c×w×h is obtained according to the number of images n used in each iterative training, and an upsampling operation is performed. The steps of the upsampling operation are as follows:
[0068] S301, a feature map with a dimension of n×c×w×h is subjected to global average pooling in the first branch to obtain a feature map with an output of n×c×1×1, and the feature map of n×c×1×1 is processed by a nonlinear activation function combined with a gating mechanism to obtain a first branch output feature map of n×c×1×1;
[0069] S302, the feature map with a dimension of n×c×w×h is processed by a common upsampling structure and a 1×1 convolution in the second branch to obtain a second branch output feature map of n×c / 2×2w×2h with doubled spatial dimensions and half the number of channels;
[0070] S303, the feature map with a dimension of n×c×w×h is processed by the container module in the third branch to obtain a third branch output feature map with a dimension of n×c / 2×2w×2h with doubled spatial dimension and half the number of channels;
[0071] S304, concatenating the second branch output feature map and the third branch output feature map along the channel dimension to obtain an n×c×2w×2h output feature map;
[0072] S305. Multiply the output feature map with a dimension of n×c×2w×2h output in step S304 by the first branch output feature map with a dimension of n×c×1×1 output in step S301 element by element to obtain an up-sampled output feature map with a dimension of n×c×2w×2h after the up-sampling operation.
[0073] In the parallel downsampling branch structure combined with the gating mechanism, according to the number of images n used in each iterative training, a feature map with an input dimension of n×c×w×h is obtained, and a downsampling operation is performed. The steps of the downsampling operation are as follows:
[0074] S306, subjecting the feature map with a dimension of n×c×w×h to global average pooling in the first branch to obtain an output feature map of n×c×1×1, and processing the n×c×1×1 feature map with a nonlinear activation function combined with a gating mechanism to obtain an output feature map of the first branch of n×c×1×1;
[0075] S307, the feature map with a dimension of n×c×w×h is subjected to maximum pooling processing in the second branch to obtain a feature map of n×c×w / 2×h / 2 with a spatial dimension halved, and the feature map of n×c×w / 2×h / 2 is further subjected to 1×1 convolution processing to obtain a second branch output feature map of n×c / 2×w / 2×h / 2 with an output channel halved;
[0076] S308, the feature map with a dimension of n×c×w×h is processed by the container module in the third branch to obtain a third branch output feature map of n×c / 2×w / 2×h / 2;
[0077] S309, concatenating the feature map output by the second branch and the feature map output by the third branch along the channel dimension to obtain an output feature map of n×c×w / 2×h / 2;
[0078] S310. Multiply the output feature map of dimension n×c×w / 2×h / 2 in step S309 by the first branch output feature map of dimension n×c×1×1 in step S306 element by element to obtain a down-sampled output feature map of dimension n×c×w / 2×h / 2 obtained after the final down-sampling operation.
[0079] In step S5, the improved rt-detr model is trained using the training set in step S1, the training parameter is epoch=200, the input image size is 640*640 pixels; 4 groups of training sample iterations are used; the optimizer is AdamW.
[0080] The remaining detection methods are the same as those described in Example 1 and will not be repeated here to avoid errors.
[0081] See also Figures 7 to 9 Based on Example 2, the underwater fish image recognition and detection method described in this application can achieve the following detection indicators:
[0082]
[0083] It can be seen from this that the present application can achieve underwater fish detection results with high accuracy in complex underwater environments, so as to meet the detection resources of underwater fish and facilitate the detection and utilization of underwater resources.
[0084] Finally, although this specification is described according to implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.
Claims
1. A method for underwater fish image recognition and detection, characterized in that: The method comprises the following steps: S1. Obtain the DeepFish underwater fish public dataset and divide it into training set, validation set, and test set in a ratio of 7:2:1; S2, select the rt-detr model as the backbone network, and introduce the triple attention module to improve the last two residual modules in the rt-detr model; S3. Use a parallel upsampling branch structure combined with a gating mechanism to improve the first upsampling structure of the neck network of the original RT-DETR model, and use a parallel downsampling branch structure combined with a gating mechanism to improve the second upsampling structure of the neck network of the original RT-DETR model; S4, the RepC3 structure of the neck network of the original rt-detr model is improved by combining the parallel dilated convolution structure with different dilation rates and the original convolution structure; S5, training the rt-detr model improved in step S4 using the training set in step S1 to obtain an underwater fish image recognition and detection model; S6, testing the underwater fish image recognition detection model obtained in step S5 on a validation set, and adjusting the underwater fish image recognition detection model according to the test results; S7, testing the underwater fish image recognition detection model adjusted in step S6 using a test set; S8, deploying the underwater fish image recognition detection model tested in step S7 in the detection device, and collecting underwater fish images through the underwater image acquisition device; S9. The underwater fish images collected by the underwater image acquisition device are imported into the underwater fish image recognition detection model. The underwater fish image recognition detection model outputs the recognition result, and outputs the detection result through the display device or sound device in the detection device.
2. The underwater fish image recognition and detection method according to claim 1, characterized in that: The triple attention module described in step S2 uses three different branches to process input features from different dimensions, and then improves network performance through cross-information fusion.
3. The underwater fish image recognition and detection method according to claim 2, characterized in that: The triple attention module includes a vertical attention module, a deep attention module, and a horizontal attention module. For an input feature map of c×w×h consisting of the number of channels c, the width of the feature map w, and the height of the feature map h, the processing process is as follows: S21, the vertical attention module transforms the input feature map in the width dimension, and the dimension of the transformed feature map becomes w×h×c. A 1×1 convolutional neural network is used to calculate the vertical attention weight, and the calculated weight map is multiplied element by element with the original input feature map, and the reconstructed input feature map dimension c×w×h is output; S22, the deep attention module maintains the original dimension of the input feature map c×w×h, uses a 1×1 convolutional neural network to calculate the deep attention weight, multiplies the calculated weight map by the original input feature map element by element, and outputs the reconstructed feature map dimension c×w×h; S23, the horizontal attention module transforms the input feature map in the height dimension. The dimension of the transformed feature map is h×w×c. A 1×1 convolutional neural network is used to calculate the horizontal attention weight. The calculated weight map is multiplied element by element with the original input feature map. The dimension of the reconstructed feature map is c×w×h. S24. The reconstructed feature maps output by steps S21, S22, and S23 are fused element by element by adding each other to obtain a c×w×h triple attention output feature map.
4. The underwater fish image recognition and detection method according to claim 3, characterized in that: The parallel upsampling branch structure combined with the gating mechanism and the parallel downsampling branch structure combined with the gating mechanism described in step S3 respectively perform upsampling operations and downsampling operations on the three attention output feature maps output by step S24, and suppress irrelevant features by combining the gating mechanism to improve the feature expression capability.
5. The underwater fish image recognition and detection method according to claim 4, characterized in that: In the parallel upsampling branch structure combined with the gating mechanism, a feature map with an input dimension of n×c×w×h is obtained according to the number of images n used in each iterative training, and an upsampling operation is performed. The steps of the upsampling operation are as follows: S301, a feature map with a dimension of n×c×w×h is subjected to global average pooling in the first branch to obtain a feature map with an output of n×c×1×1, and the feature map of n×c×1×1 is processed by a nonlinear activation function combined with a gating mechanism to obtain a first branch output feature map of n×c×1×1; S302, the feature map with a dimension of n×c×w×h is processed by a common upsampling structure and a 1×1 convolution in the second branch to obtain a second branch output feature map of n×c / 2×2w×2h with doubled spatial dimensions and half the number of channels; S303, the feature map with a dimension of n×c×w×h is processed by the container module in the third branch to obtain a third branch output feature map with a dimension of n×c / 2×2w×2h with doubled spatial dimension and half the number of channels; S304, concatenating the second branch output feature map and the third branch output feature map along the channel dimension to obtain an n×c×2w×2h output feature map; S305. Multiply the output feature map with a dimension of n×c×2w×2h output in step S304 by the first branch output feature map with a dimension of n×c×1×1 output in step S301 element by element to obtain an up-sampled output feature map with a dimension of n×c×2w×2h after the up-sampling operation.
6. The underwater fish image recognition and detection method according to claim 5, characterized in that: In the parallel downsampling branch structure combined with the gating mechanism, according to the number of images n used in each iterative training, a feature map with an input dimension of n×c×w×h is obtained, and a downsampling operation is performed. The steps of the downsampling operation are as follows: S306, subjecting the feature map with a dimension of n×c×w×h to global average pooling in the first branch to obtain an output feature map of n×c×1×1, and processing the n×c×1×1 feature map with a nonlinear activation function combined with a gating mechanism to obtain an output feature map of the first branch of n×c×1×1; S307, the feature map with a dimension of n×c×w×h is subjected to maximum pooling processing in the second branch to obtain a feature map of n×c×w / 2×h / 2 with a spatial dimension halved, and the feature map of n×c×w / 2×h / 2 is further subjected to 1×1 convolution processing to obtain a second branch output feature map of n×c / 2×w / 2×h / 2 with an output channel halved; S308, the feature map with a dimension of n×c×w×h is processed by the container module in the third branch to obtain a third branch output feature map with a dimension of n×c / 2×w / 2×h / 2; S309, concatenating the feature map output by the second branch and the feature map output by the third branch along the channel dimension to obtain an output feature map of n×c×w / 2×h / 2; S310. Multiply the output feature map of dimension n×c×w / 2×h / 2 in step S309 by the first branch output feature map of dimension n×c×1×1 in step S306 element by element to obtain a down-sampled output feature map of dimension n×c×w / 2×h / 2 obtained after the final down-sampling operation.
7. The underwater fish image recognition and detection method according to claim 1, characterized in that: In step S5, the improved rt-detr model is trained using the training set in step S1, the training parameter is epoch=200, the input image size is 640*640 pixels; 4 groups of training sample iterations are used; the optimizer is AdamW.