Underwater fish target detection method, device and storage medium

Abstract

The present invention discloses an underwater fish target detection method, a device and a storage medium. The backbone network in the underwater fish target detection model is improved. The improved backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities. The local feature map and the global feature map of the underwater image to be detected can be obtained through the fast Fourier transform module, so as to obtain better quality image features and improve the accuracy of target detection. At the same time, the global feature map is extracted in the frequency domain in the fast Fourier transform module, which greatly reduces the calculation complexity and can efficiently extract features, thereby improving the efficiency of target detection.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to an underwater fish target detection method, device and storage medium based on a fast Fourier transform filter. Background Art

[0002] Fish farming occupies a dominant position in aquaculture. In order to ensure the production of fish, it is necessary to estimate the number of farmed fish or monitor their growth status. In addition, in order to prevent the sudden death of fish from seriously affecting the growth of other fish, it is necessary to track the individual target fish with abnormal conditions.

[0003] At present, fish target detection is mainly carried out based on underwater videos or images, using machine learning algorithms to monitor and count the growth status of underwater fish, as well as track individual fish. Among them, in the machine learning algorithm, the backbone network is the most important part of the target detection network. The backbone network is the network responsible for feature extraction in the target detection network. It is generally used for the front end to extract image features of different levels and scales, and generate feature maps for use by the subsequent network.

[0004] At present, most commonly used backbone networks are convolutional neural networks. However, convolutional neural networks can only obtain local information of images and have the disadvantage of being unable to model the global image at the bottom layer. In order to overcome this disadvantage, the prior art often combines convolutional neural networks with self-attention mechanisms as backbone networks, obtains local information of input images through convolutional neural networks, and obtains global information of input images based on self-attention mechanisms, thereby extracting features of input images. However, when the backbone network under this network structure performs feature extraction, the amount of calculation increases sharply with the increase of resolution, and the computing and storage requirements are high, which seriously affects the efficiency of feature extraction during underwater fish target detection. Summary of the invention

[0005] In view of this, the present invention provides an underwater fish target detection method, device and storage medium based on a fast Fourier transform filter, so as to improve the efficiency of underwater fish target detection while improving the accuracy of underwater fish target detection.

[0006] To this end, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention provides an underwater fish target detection method based on a fast Fourier transform filter, the method comprising:

[0008] S1, obtaining an underwater image to be detected;

[0009] S2, inputting the underwater image to be detected into the backbone network of the underwater fish target detection model to obtain a feature extraction result; the backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities; the fast Fourier transform module includes: a local feature extraction module for extracting local features, a Fourier transform filter for extracting global features, and a feature fusion module for fusing local features and global features; in the Fourier transform filter, performing a fast Fourier transform on the input feature map to obtain a frequency domain feature map, multiplying the obtained frequency domain feature map with the kernel parameters of the same size point by point, and then performing a fast Fourier inverse transform on the obtained frequency domain result to obtain a time domain feature map;

[0010] S3. Based on the feature extraction result, an underwater fish target detection result is obtained.

[0011] Furthermore, the underwater image to be detected is input into the backbone network of the underwater fish target detection model to obtain feature extraction results, including:

[0012] The underwater image to be detected is processed using a convolutional neural network with a 1*1 convolution kernel;

[0013] Inputting the processing result into the local feature extraction module to obtain local features;

[0014] Use the convolutional neural network with a 1*1 convolution kernel to perform channel transformation on the processing results;

[0015] The feature map after channel transformation is superimposed on the local feature and input into the Fourier transform filter to obtain the global feature;

[0016] Inputting the local features and the global features into the feature fusion module to obtain a feature fusion result;

[0017] The feature fusion results are processed using a convolutional neural network with a 1*1 convolution kernel to obtain the final feature extraction results.

[0018] Furthermore, the local feature extraction module is a convolutional neural network with a 3x3 convolution kernel.

[0019] Furthermore, the local features and the global features are input into the feature fusion module to obtain a feature fusion result, including:

[0020] The number of feature channels of the local features is transformed by a convolutional neural network with a 1x1 convolution kernel;

[0021] The local features after the number of feature channels is transformed are merged with the global features;

[0022] The fused features are averaged and pooled to adjust the size of the feature map.

[0023] In another aspect, the present invention further provides an underwater fish target detection device based on a fast Fourier transform filter, comprising:

[0024] An image acquisition unit, used for acquiring an underwater image to be detected;

[0025] A feature extraction unit is used to input the underwater image to be detected acquired by the image acquisition unit into the backbone network of the underwater fish target detection model to obtain a feature extraction result; the backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities; the fast Fourier transform module includes: a local feature extraction module for extracting local features, a Fourier transform filter for extracting global features, and a feature fusion module for fusing local features and global features; in the Fourier transform filter, a fast Fourier transform is performed on the input feature map to obtain a frequency domain feature map, the obtained frequency domain feature map is point-by-point multiplied with a kernel parameter of the same size, and then the obtained frequency domain result is subjected to a fast Fourier inverse transform to obtain a time domain feature map;

[0026] The target detection unit is used to obtain underwater fish target detection results based on the feature extraction results.

[0027] Furthermore, the feature extraction unit is specifically used to: process the underwater image to be detected using a convolutional neural network with a 1*1 convolution kernel; input the processing result into the local feature extraction module to obtain local features; perform channel transformation on the processing result using a convolutional neural network with a 1*1 convolution kernel; superimpose the feature map after channel transformation with the local features, and input it into the Fourier transform filter to obtain global features; input the local features and the global features into the feature fusion module to obtain feature fusion results; process the feature fusion results using a convolutional neural network with a 1*1 convolution kernel to obtain the final feature extraction result.

[0028] Furthermore, the local feature extraction module is a convolutional neural network with a 3x3 convolution kernel.

[0029] Furthermore, the feature fusion module is specifically used to: transform the number of feature channels of the local features through a convolutional neural network with a 1x1 convolution kernel; fuse the local features after the transformed feature channel number with the global features; and perform an average pooling operation on the fused features to adjust the size of the feature map.

[0030] On the other hand, the present invention also provides a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, the above-mentioned underwater fish target detection method based on fast Fourier transform filter is implemented.

[0031] Advantages and positive effects of the present invention: In the present invention, the backbone network in the underwater fish target detection model is improved, and the improved backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities. The local feature map and the global feature map of the underwater image to be detected can be obtained through the fast Fourier transform module, so as to obtain better quality image features and improve the accuracy of target detection; at the same time, the global feature map is extracted in the frequency domain in the fast Fourier transform module. Since the feature data multiplication operation in the frequency domain is equivalent to the circular convolution operation of the data in the time domain, and the convolution kernel size is the same as the feature map size, it is equivalent to using the full-size convolution kernel to extract the global feature. Compared with the feature data multiplication operation in the time domain, the calculation complexity is greatly reduced, and the features can be extracted efficiently, thereby improving the efficiency of target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0033] Figure 1 Schematic diagram of the structure of the backbone network of the underwater fish target detection model in an embodiment of the present invention;

[0034] Figure 2 A schematic diagram of the structure of a fast Fourier transform filter in an embodiment of the present invention;

[0035] Figure 3 This is a flow chart of extracting features based on the improved backbone network in an embodiment of the present invention;

[0036] Figure 4 The figure is a schematic diagram of the structure of an underwater fish target detection device based on a fast Fourier transform filter in an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0038] Based on the mainstream ResNet framework, this paper proposes a backbone network for underwater fish target detection based on fast Fourier transform filter. Figure 1 As shown, the ResNet model includes several residual modules, each of which is composed of three convolutional networks of 1*1, 3*3, and 1*1 connected in series. In the present invention, the 3x3 convolutional module used for local feature extraction in the ResNet model is replaced by an FFT_Block module that integrates local and global feature extraction capabilities.

[0039] The structural block diagram of the FFT_Block module is as follows Figure 2 As shown. First, the local features of the input image are extracted using a 3x3 convolution kernel (Conv2) to obtain a local feature map; the input original image is transformed through a 1x1 convolution kernel (Conv1), and then superimposed with the local feature map. The superposition result is input into the Fourier transform filter for global feature extraction to obtain a global feature map. The local feature map is transformed through a 1x1 convolution kernel (Conv3) to transform the number of feature channels, and then fused with the global feature map (point-by-point addition), and finally the average pooling module (Avg_Pooling) is used to adjust the size of the feature map. In the Fourier transform filter, the input feature map (superposition result) is first subjected to a fast Fourier transform (FFT), and the obtained frequency domain feature map is multiplied point by point with the kernel parameter FFT_Kernel of the same size, and then the obtained frequency domain result is subjected to an inverse fast Fourier transform (IFFT) to obtain a time domain feature map.

[0040] According to convolution theory, the multiplication operation of feature data in the frequency domain is equivalent to the circular convolution operation of data in the time domain, and the convolution kernel size is the same as the feature map size, which is equivalent to using the full-size convolution kernel for global feature extraction. Specifically, assuming that the model input image batch is B, and B = 16, the input and output channel dimensions of the existing input feature map are C = 128, and the feature map size is H = W = 56, where H is the image height and W is the image width. The computational complexity of extracting global features using the Fourier transform filter in the FFT_Filter module includes (2H·W·logH)(B·C+C·C)=202100157 operations of converting the input time domain feature map to the frequency domain feature map, B·H·W·C·C=822083584 operations of point-by-point multiplication of the obtained frequency domain feature map and the parameter matrix (the same size kernel parameter FFT_Kernel), and (2H·W·logH)(B·C)=22454272 operations of transforming the obtained feature map back to the time domain. The total computational complexity is (2H·W·logH)(B·C+C·C)+B·H·W·C·C+(2H·W·logH)(B·C)=1.05GFLOPs. When the self-attention mechanism module is used to extract global features, the computational complexity includes B·(H·W) of the Q and K matrix similarity operations. 2 C = 20141047808 operations, B (H W) calculated by softmax 2= 157351936 operations and the similarity calculation between the feature map and the V matrix B·(H·W) 2 C = 20141047808 operations, with a total computational complexity of 2B (H W) 2 ·C+B·(H·W) 2 =40.4GFLOPs. It can be seen that the computational complexity of the model proposed in the present invention is 1 / 40 of that of the self-attention mechanism module. Therefore, the proposed method can realize efficient global feature extraction of the input graph, thereby improving the efficiency of feature extraction of the backbone network.

[0041] The following is a detailed description of the method for underwater fish target detection based on the improved backbone network in an embodiment of the present invention.

[0042] An underwater fish target detection method based on a fast Fourier transform filter in an embodiment of the present invention includes:

[0043] S1, obtaining an underwater image to be detected;

[0044] S2, inputting the underwater image to be detected into the backbone network of the underwater fish target detection model to obtain a feature extraction result;

[0045] The underwater fish target detection model includes a front-end backbone network responsible for feature extraction and a back-end detection network responsible for target detection based on the extracted features. The present invention mainly improves the backbone network, and the network structure of the back-end detection network is not limited here.

[0046] In the embodiment of the present invention, the improved backbone network is composed of a 1*1, a fast Fourier transform module (FFT_Block module) that integrates local and global feature extraction capabilities, and three 1*1 convolutional networks connected in series.

[0047] The FFT_Block module includes: a local feature extraction module for extracting local features, a Fourier transform filter (FFT_Filter) for extracting global features, and a feature fusion module for fusing local features and global features.

[0048] Among them, the local feature extraction module is a convolutional neural network with a 3x3 convolution kernel.

[0049] In FFT_Filter, the input feature map is subjected to fast Fourier transform to obtain the frequency domain feature map, the obtained frequency domain feature map is multiplied point by point with the kernel parameters of the same size, and then the obtained frequency domain result is subjected to inverse fast Fourier transform to obtain the time domain feature map.

[0050] In the feature fusion module, the number of feature channels of the local features is transformed through a convolutional neural network with a 1x1 convolution kernel; the local features after the number of feature channels is transformed are fused with the global features; and the fused features are average pooled to adjust the size of the feature map.

[0051] In the specific implementation, Figure 3 As shown, S2 specifically includes the following steps:

[0052] S201, using a convolutional neural network with a 1*1 convolution kernel to process the underwater image to be detected;

[0053] S202, inputting the processing result into the local feature extraction module to obtain local features;

[0054] S203, using a convolutional neural network with a 1*1 convolution kernel to perform channel transformation on the processing result;

[0055] S204, superimposing the feature map after channel transformation with the local feature, and inputting the result into the Fourier transform filter to obtain a global feature;

[0056] S205, inputting the local features and the global features into the feature fusion module to obtain a feature fusion result;

[0057] S206. Use a convolutional neural network with a 1*1 convolution kernel to process the feature fusion result to obtain the final feature extraction result.

[0058] S3. Based on the feature extraction result, an underwater fish target detection result is obtained.

[0059] In a specific implementation, the obtained feature extraction results can be input into the back-end target detection network in the underwater fish target detection model to obtain the underwater fish target detection results.

[0060] In an embodiment of the present invention, the backbone network in the underwater fish target detection model is improved. The improved backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities. The local feature map and the global feature map of the underwater image to be detected can be obtained through the fast Fourier transform module, so as to obtain better quality image features and improve the accuracy of target detection. At the same time, the global feature map is extracted in the frequency domain in the fast Fourier transform module. Since the feature data multiplication operation in the frequency domain is equivalent to the circular convolution operation of the data in the time domain, and the convolution kernel size is the same as the feature map size, it is equivalent to using the full-size convolution kernel to extract global features. Compared with the feature data multiplication operation in the time domain, the computational complexity is greatly reduced, and the features can be extracted efficiently, thereby improving the efficiency of target detection.

[0061] Corresponding to the underwater fish target detection method based on the fast Fourier transform filter in the present invention, the present invention also provides an underwater fish target detection device based on the fast Fourier transform filter. Figure 4 As shown, it shows a structural schematic diagram of an underwater fish target detection device based on a fast Fourier transform filter in an embodiment of the present invention, the device includes:

[0062] An image acquisition unit 100 is used to acquire an underwater image to be detected;

[0063] The feature extraction unit 200 is used to input the underwater image to be detected acquired by the image acquisition unit into the backbone network of the underwater fish target detection model to obtain a feature extraction result;

[0064] The backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities; the fast Fourier transform module includes: a local feature extraction module for extracting local features, a Fourier transform filter for extracting global features, and a feature fusion module for fusing local features and global features;

[0065] Among them, the local feature extraction module is a convolutional neural network with a 3x3 convolution kernel.

[0066] The Fourier transform filter is specifically used to: perform fast Fourier transform on the input feature map to obtain a frequency domain feature map, multiply the obtained frequency domain feature map point by point with the kernel parameters of the same size, and then perform inverse fast Fourier transform on the obtained frequency domain result to obtain a time domain feature map.

[0067] The feature fusion module is specifically used for: transforming the number of feature channels of the local features through a convolutional neural network with a 1x1 convolution kernel; fusing the local features after the number of feature channels is transformed with the global features; and performing an average pooling operation on the fused features to adjust the size of the feature map.

[0068] The target detection unit 300 is used to obtain underwater fish target detection results based on the feature extraction results.

[0069] Furthermore, the feature extraction unit 200 is specifically used to: process the underwater image to be detected using a convolutional neural network with a 1*1 convolution kernel; input the processing result into the local feature extraction module to obtain local features; perform channel transformation on the processing result using a convolutional neural network with a 1*1 convolution kernel; superimpose the feature map after channel transformation with the local features, and input it into the Fourier transform filter to obtain global features; input the local features and the global features into the feature fusion module to obtain feature fusion results; process the feature fusion results using a convolutional neural network with a 1*1 convolution kernel to obtain the final feature extraction result.

[0070] The underwater fish target detection device based on the fast Fourier transform filter in the embodiment of the present invention uses the improved backbone network to perform feature extraction, and can obtain local feature maps and global feature maps of the underwater image to be detected, thereby obtaining better quality image features and improving the accuracy of target detection; at the same time, the improved backbone network extracts the global feature map in the frequency domain, which greatly reduces the computational complexity and can efficiently extract features, thereby improving the efficiency of target detection.

[0071] For the underwater fish target detection device based on fast Fourier transform filter in the embodiment of the present invention, since it corresponds to the underwater fish target detection method based on fast Fourier transform filter in the above embodiment, the description is relatively simple. Please refer to the description of the above embodiment for relevant similarities, which will not be described in detail here.

[0072] An embodiment of the present invention further discloses a computer-readable storage medium, which stores a computer instruction set. When the computer instruction set is executed by a processor, the underwater fish target detection method based on the fast Fourier transform filter as provided in any of the above embodiments is implemented.

[0073] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0074] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0075] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0076] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0077] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for underwater fish target detection based on fast Fourier transform filter, characterized in that: The method comprises: S1, obtaining an underwater image to be detected; S2, inputting the underwater image to be detected into the backbone network of the underwater fish target detection model to obtain a feature extraction result; the backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities; the fast Fourier transform module includes: a local feature extraction module for extracting local features, a Fourier transform filter for extracting global features, and a feature fusion module for fusing local features and global features; in the Fourier transform filter, performing a fast Fourier transform on the input feature map to obtain a frequency domain feature map, multiplying the obtained frequency domain feature map with the kernel parameters of the same size point by point, and then performing a fast Fourier inverse transform on the obtained frequency domain result to obtain a time domain feature map; S3. Based on the feature extraction result, an underwater fish target detection result is obtained; The underwater image to be detected is input into the backbone network of the underwater fish target detection model to obtain feature extraction results, including: The underwater image to be detected is processed using a convolutional neural network with a 1*1 convolution kernel; Inputting the processing result into the local feature extraction module to obtain local features; Use the convolutional neural network with a 1*1 convolution kernel to perform channel transformation on the processing results; The feature map after channel transformation is superimposed on the local feature and input into the Fourier transform filter to obtain the global feature; Inputting the local features and the global features into the feature fusion module to obtain a feature fusion result; The feature fusion results are processed using a convolutional neural network with a 1*1 convolution kernel to obtain the final feature extraction results.

2. The underwater fish target detection method based on fast Fourier transform filter according to claim 1 is characterized in that: The local feature extraction module is a convolutional neural network with a 3x3 convolution kernel.

3. The underwater fish target detection method based on fast Fourier transform filter according to claim 2 is characterized in that: Inputting the local features and the global features into the feature fusion module to obtain a feature fusion result includes: The number of feature channels of the local features is transformed by a convolutional neural network with a 1x1 convolution kernel; The local features after the number of feature channels is transformed are merged with the global features; The fused features are averaged and pooled to adjust the size of the feature map.

4. An underwater fish target detection device based on a fast Fourier transform filter, characterized in that: The device comprises: An image acquisition unit, used for acquiring an underwater image to be detected; A feature extraction unit is used to input the underwater image to be detected acquired by the image acquisition unit into the backbone network of the underwater fish target detection model to obtain a feature extraction result; the backbone network includes a fast Fourier transform module that integrates local and global feature extraction capabilities; the fast Fourier transform module includes: a local feature extraction module for extracting local features, a Fourier transform filter for extracting global features, and a feature fusion module for fusing local features and global features; in the Fourier transform filter, a fast Fourier transform is performed on the input feature map to obtain a frequency domain feature map, the obtained frequency domain feature map is point-by-point multiplied with a kernel parameter of the same size, and then the obtained frequency domain result is subjected to a fast Fourier inverse transform to obtain a time domain feature map; A target detection unit, used to obtain underwater fish target detection results based on the feature extraction results; Among them, the feature extraction unit is specifically used to: process the underwater image to be detected using a convolutional neural network with a 1*1 convolution kernel; input the processing result into the local feature extraction module to obtain local features; perform channel transformation on the processing result using a convolutional neural network with a 1*1 convolution kernel; superimpose the feature map after channel transformation with the local features, and input it into the Fourier transform filter to obtain global features; input the local features and the global features into the feature fusion module to obtain feature fusion results; process the feature fusion results using a convolutional neural network with a 1*1 convolution kernel to obtain the final feature extraction result.

5. The underwater fish target detection device based on fast Fourier transform filter according to claim 4 is characterized in that: The local feature extraction module is a convolutional neural network with a 3x3 convolution kernel.

6. The underwater fish target detection device based on fast Fourier transform filter according to claim 4 is characterized in that: The feature fusion module is specifically used for: transforming the number of feature channels of the local features through a convolutional neural network with a 1x1 convolution kernel; fusing the local features after the number of feature channels is transformed with the global features; and performing an average pooling operation on the fused features to adjust the size of the feature map.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer instruction set, and when the computer instruction set is executed by the processor, an underwater fish target detection method based on a fast Fourier transform filter as described in any one of claims 1 to 3 is implemented.

Images