Method, device, equipment and medium for identifying feeding behavior of fish school

Abstract

The present invention relates to the field of aquaculture information technology, and discloses a method, device, equipment and medium for identifying the feeding behavior of a school of fish. The present invention obtains a time-series image set of the feeding behavior of a school of fish, and respectively uses a preset enhancement algorithm, a preset optical flow method and a preset Gaussian density model to perform image preprocessing on the time-series image set to obtain corresponding spatial images, optical flow images and density images; according to the spatial image, optical flow image and density image and a preset attention mechanism model, extracts the first feature corresponding to each spatial image, the second feature corresponding to each optical flow image and the third feature corresponding to each density image; according to the first feature, the second feature and the third feature, the first feature, the second feature and the third feature are feature fused through a preset attention mechanism model to determine the corresponding fused feature; according to the fused feature, identifies the feeding behavior of the school of fish; thereby achieving precise feeding control.

Description

Technical Field

[0001] The present invention relates to the field of aquaculture information technology, and in particular to a method, device, equipment and medium for identifying the feeding behavior of a school of fish. Background Art

[0002] In recent years, with the continuous development of aquaculture technology in my country, the output of factory-scale pond fishery has continued to increase, and the precise control of fish feeding amount in aquaculture has received more and more attention.

[0003] At present, in the actual feeding of fish farming, artificial feeding based on experience values ​​is generally adopted, or feeding is carried out by machine, and the feeding control of bait is also manually set according to farming experience. At present, some papers have proposed methods for identifying fish school behavior, but most of them are based on laboratory fish tank type data and are difficult to apply in actual farming cages.

[0004] However, due to the complexity and identification difficulty of the actual breeding application environment, it is difficult to achieve precise feeding control based on the actual breeding conditions of global data. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, device, equipment and medium for identifying the feeding behavior of fish schools, aiming to achieve precise feeding control.

[0006] To achieve the above object, the present invention provides a method for identifying the feeding behavior of a school of fish, the method comprising the following steps:

[0007] Acquire a time-series image set of the feeding behavior of a school of fish, and pre-process the time-series image set using a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model, respectively, to obtain corresponding spatial images, optical flow images, and density images;

[0008] Based on the spatial image, the optical flow image, the density image and a preset attention mechanism model, extracting a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image;

[0009] Based on the first feature, the second feature and the third feature, feature fusion is performed through the preset attention mechanism model to determine a corresponding fusion feature;

[0010] Based on the fusion features, the feeding behavior of the fish school is identified.

[0011] Preferably, the preset attention mechanism model includes a convolution transposition module, or an attention module, or a combination of a convolution transposition module and an attention module.

[0012] Preferably, the step of extracting the first feature corresponding to each spatial image, the second feature corresponding to each optical flow image, and the third feature corresponding to each density image based on the spatial image, the optical flow image, the density image and a preset feature extraction model comprises:

[0013] According to the spatial image, the optical flow image and the density image, convolution and pooling processing are performed on each spatial image, each optical flow image and each density image through the preset attention mechanism model to obtain a processed spatial image, a processed optical flow image and a processed density image;

[0014] The processed spatial image, the processed optical flow image and the processed density image are subjected to residual summation processing for a preset number of times to obtain a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image and a third feature corresponding to each density image.

[0015] Preferably, the step of performing feature fusion based on the first feature, the second feature and the third feature through the preset attention mechanism model to determine the corresponding fusion feature includes:

[0016] According to the first feature, the second feature and the third feature, the internal features of the single image are fused through the preset attention mechanism model to obtain a first internal feature corresponding to the first feature, a second internal feature corresponding to the second feature, and a third internal feature corresponding to the third feature;

[0017] According to the first internal feature, the second internal feature and the third internal feature, feature fusion between multiple images is performed through the preset attention mechanism model to obtain a first associated feature corresponding to the first internal feature, a second associated feature corresponding to the second internal feature, and a third associated feature corresponding to the third internal feature;

[0018] According to the first association feature, the second association feature and the third association feature, feature fusion between different domain features is performed through the preset attention mechanism model to obtain corresponding fusion features.

[0019] Preferably, the preset attention mechanism model includes a convolution transposition module and an attention module, and the step of fusing the internal features of a single image through the preset attention mechanism model according to the first feature, the second feature and the third feature to obtain a first internal feature corresponding to the first feature, a second internal feature corresponding to the second feature and a third internal feature corresponding to the third feature includes:

[0020] Performing a convolution operation on the first feature, the second feature, and the third feature through the convolution transposition module, and performing feature matrix transposition multiplication on the result of the convolution and the first feature, the second feature, and the third feature to obtain a first convolution feature corresponding to the first feature, a second convolution feature corresponding to the second feature, and a third convolution feature corresponding to the third feature;

[0021] Performing a preset number of convolution operations on the first convolution feature, the second convolution feature, and the third convolution feature through the attention module, and performing feature matrix multiplication on the convolution results to obtain a first product feature corresponding to the first convolution feature, a second product feature corresponding to the second convolution feature, and a third product feature corresponding to the third convolution;

[0022] The first product feature, the first product feature and the first product feature are respectively summed with the first convolution feature, the second convolution feature and the third convolution feature to obtain the corresponding first internal feature, the second internal feature and the third internal feature.

[0023] Preferably, the preset attention mechanism model includes an attention module, and the step of performing feature fusion between multiple images through the preset attention mechanism model according to the first internal feature, the second internal feature and the third internal feature to obtain a first associated feature corresponding to the first internal feature, a second associated feature corresponding to the second internal feature and a third associated feature corresponding to the third internal feature includes:

[0024] Performing feature integration processing on the first internal feature, the second internal feature, and the third internal feature respectively to obtain a first integrated feature corresponding to the first internal feature, a second integrated feature corresponding to the second internal feature, and a third integrated feature corresponding to the third internal feature;

[0025] Performing a residual operation on the first integrated feature, the second integrated feature, and the third integrated feature to obtain a first sum feature corresponding to the first integrated feature, a second sum feature corresponding to the second integrated feature, and a third sum feature corresponding to the third integrated feature;

[0026] Performing a pooling operation on the first sum feature, the second sum feature, and the third sum feature to obtain a corresponding pooled first sum feature, a pooled second sum feature, and a pooled third sum feature;

[0027] The attention module performs a preset number of convolution operations on the first and second pooled features, and the third and second pooled features, and performs feature matrix multiplication on the result of the convolution and the first and second pooled features, and the third and second pooled features, respectively, and then sums them to obtain the corresponding first product feature, second product feature, and third product feature;

[0028] Performing a pooling operation on the first product feature, the second product feature, and the third product feature to obtain a pooled first product feature, a pooled second product feature, and a pooled third product feature;

[0029] The attention module performs a preset number of convolution operations on the first pooled product feature, the second pooled product feature, and the third pooled product feature, and performs feature matrix multiplication on the result of the convolution and the first pooled product feature, the second pooled product feature, and the third pooled product feature, respectively, and then sums them up to obtain the corresponding first intent feature, second intent feature, and third intent feature;

[0030] The first intention feature, the second intention feature and the third intention feature are pooled to obtain corresponding first association features, second association features and third association features.

[0031] Preferably, the preset attention mechanism model includes a convolution transposition module and an attention module, and the step of fusing features between different domain features through the preset attention mechanism model according to the first association feature, the second association feature and the third association feature to obtain corresponding fusion features includes:

[0032] Channel-joining the first association feature, the second association feature, and the third association feature to obtain a corresponding three-dimensional matrix feature;

[0033] The convolution operation is performed on the three-dimensional matrix feature by the convolution transposition module, and a feature matrix is ​​multiplied by the convolution result and the three-dimensional matrix feature to obtain a coding feature corresponding to the three-dimensional matrix feature;

[0034] Performing a preset number of convolution operations on the coding feature through the attention module, and multiplying the result of the convolution by a feature matrix to obtain a product coding feature corresponding to the coding feature;

[0035] Feature summation is performed on the product coding feature to obtain a fusion feature corresponding to the product coding feature.

[0036] In addition, to achieve the above-mentioned purpose, the present invention also provides a device for identifying the feeding behavior of a school of fish, the device for identifying the feeding behavior of a school of fish comprising:

[0037] An acquisition module is used to acquire a time-series image set of the feeding behavior of a school of fish, and pre-process the time-series image set using a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model to obtain corresponding spatial images, optical flow images, and density images;

[0038] An extraction module, configured to extract, based on the spatial image, the optical flow image, the density image and a preset attention mechanism model, a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image;

[0039] A fusion module, configured to perform feature fusion based on the first feature, the second feature and the third feature through the preset attention mechanism model to determine a corresponding fusion feature;

[0040] The recognition module is used to recognize the feeding behavior of the fish school based on the fusion features.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a device, which is a device for identifying the feeding behavior of a school of fish. The device for identifying the feeding behavior of a school of fish includes: a memory, a processor, and a program for identifying the feeding behavior of a school of fish stored in the memory and executable on the processor. When the program for identifying the feeding behavior of a school of fish is executed by the processor, the steps of the method for identifying the feeding behavior of a school of fish as described above are implemented.

[0042] In addition, to achieve the above-mentioned purpose, the present invention also provides a medium, which is a computer-readable storage medium, and a program for identifying the feeding behavior of a school of fish is stored on the computer-readable storage medium. When the program for identifying the feeding behavior of a school of fish is executed by a processor, the steps of the method for identifying the feeding behavior of a school of fish as described above are implemented.

[0043] The present invention proposes a method, device, equipment and medium for identifying the feeding behavior of a school of fish; the method for identifying the feeding behavior of a school of fish comprises: obtaining a time-series image set of the feeding behavior of a school of fish, and preprocessing the time-series image set using a preset enhancement algorithm, a preset optical flow method and a preset Gaussian density model to obtain corresponding spatial images, optical flow images and density images; based on the spatial images, the optical flow images, the density images and a preset attention mechanism model, extracting a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image and a third feature corresponding to each density image; based on the first feature, the second feature and the third feature, performing feature fusion through the preset attention mechanism model to determine the corresponding fusion feature; based on the fusion feature, identifying the feeding behavior of the school of fish. The present invention obtains a time-series image set of the feeding behavior of a school of fish, and respectively uses a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model to perform image preprocessing on the time-series image set to obtain corresponding spatial images, optical flow images, and density images; according to the spatial images, optical flow images, density images, and a preset attention mechanism model, a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image are extracted; according to the first feature, the second feature, and the third feature, the first feature, the second feature, and the third feature are feature fused through a preset attention mechanism model to determine the corresponding fused feature; according to the fused feature, the feeding behavior of the school of fish is identified; thereby, precise feeding control is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention;

[0045] Figure 2 It is a flow chart of the first embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0046] Figure 3 A schematic diagram of a preset attention mechanism model of a second embodiment of a method for identifying a school of fish feeding behavior according to the present invention;

[0047] Figure 4 It is a flow chart of a third embodiment of a method for identifying a school of fish feeding behavior according to the present invention;

[0048] Figure 5 A schematic diagram of a sub-flow of the third embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0049] Figure 6 It is a flow chart of a fourth embodiment of a method for identifying a school of fish feeding behavior according to the present invention;

[0050] Figure 7A schematic diagram of a sub-flow of the fourth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0051] Figure 8 It is a flow chart of a fifth embodiment of a method for identifying a school of fish feeding behavior according to the present invention;

[0052] Fig. 9 A schematic diagram of a sub-flow diagram of a fifth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0053] Fig.10 It is a flow chart of a sixth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0054] Fig.11 A schematic diagram of a sub-flow of a sixth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0055] Fig.12 It is a flow chart of the seventh embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0056] Fig.13 A schematic diagram of a sub-flow of the first embodiment of the method for identifying the feeding behavior of a school of fish of the present invention;

[0057] Fig.14 This is a schematic diagram of the functional modules of the first embodiment of the device for identifying the feeding behavior of a school of fish according to the present invention.

[0058] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0060] like Figure 1 As shown, Figure 1 It is a schematic diagram of the device structure of the hardware operating environment involved in the embodiment of the present invention.

[0061] The device in the embodiment of the present invention may be a mobile terminal or a server device.

[0062] like Figure 1As shown, the device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0063] Those skilled in the art will understand that Figure 1 The device structure shown in the figure does not constitute a limitation of the device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0064] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a program for identifying the feeding behavior of a school of fish.

[0065] Among them, the operating system is a program for managing and controlling the identification equipment and software resources of the fish feeding behavior, supporting the operation of the network communication module, the user interface module, the identification program of the fish feeding status and hunger level, and other programs or software; the network communication module is used to manage and control the network interface 1002; the user interface module is used to manage and control the user interface 1003.

[0066] exist Figure 1 In the device for identifying the feeding behavior of a school of fish shown, the device for identifying the feeding behavior of a school of fish calls the identification program for the feeding behavior of a school of fish stored in the memory 1005 through the processor 1001, and executes the operations in each embodiment of the method for identifying the feeding behavior of a school of fish described below.

[0067] Based on the above hardware structure, an embodiment of a method for identifying the feeding behavior of a school of fish of the present invention is proposed.

[0068] Reference Figure 2 , Figure 2 The figure is a flow chart of a first embodiment of a method for identifying a school of fish feeding behavior according to the present invention. The method for identifying a school of fish feeding behavior comprises:

[0069] Step S10, obtaining a time-series image set of the feeding behavior of the school of fish, and preprocessing the time-series image set using a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model, respectively, to obtain corresponding spatial images, optical flow images, and density images;

[0070] Step S20, based on the spatial image, the optical flow image, the density image and a preset attention mechanism model, extracting a first feature corresponding to the spatial image, a second feature corresponding to the optical flow image, and a third feature corresponding to the density image;

[0071] Step S30, based on the first feature, the second feature and the third feature, feature fusion is performed through the preset attention mechanism model to determine a corresponding fusion feature;

[0072] Step S40: identifying the feeding behavior of the fish school based on the fusion features.

[0073] This embodiment obtains a time-series image set of the feeding behavior of a school of fish, and respectively uses a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model to perform image preprocessing on the time-series image set to obtain corresponding spatial images, optical flow images, and density images; according to the spatial images, optical flow images, density images, and a preset attention mechanism model, a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image are extracted; according to the first feature, the second feature, and the third feature, the first feature, the second feature, and the third feature are feature fused by the preset attention mechanism model to determine the corresponding fused feature; according to the fused feature, the feeding behavior of the school of fish is identified; thereby, precise feeding control is achieved.

[0074] The following is a detailed description of each step:

[0075] Step S10, obtaining a time series image set of the feeding behavior of the school of fish, and preprocessing the time series image set using a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model to obtain corresponding spatial images, optical flow images, and density images.

[0076] In this embodiment, the method for identifying fish feeding behavior consists of three main parts, namely, S1, a spatial channel of fish distribution, S2, an optical flow channel of fish swimming speed, and S3, a density channel of fish number; wherein, S1, a spatial channel of fish distribution corresponds to a spatial image, S2, an optical flow channel of fish swimming speed corresponds to an optical flow image, and S3, a density channel of fish number corresponds to a density image.

[0077] A camera is used to continuously shoot the fish breeding cabin to obtain real-time video of the fish feeding behavior; the real-time video is frame segmented and images with clearer fish targets are selected as the time-series image set of the fish feeding behavior; the same time-series image set is preprocessed to obtain the spatial image, optical flow image and density image corresponding to the time-series image set of the fish feeding behavior.

[0078] The spatial image of the fish school is formed by performing image enhancement preprocessing operations on the time series image set using a preset enhancement algorithm.

[0079] Among them, in image analysis, the processing performed on the input image before feature extraction, segmentation and matching. The main purpose of image preprocessing is to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information and simplify data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition. The preprocessing process generally includes digitization, geometric transformation, normalization, smoothing, restoration and enhancement.

[0080] By performing dense optical flow processing on a time series image set of fish feeding behavior, the optical flow image corresponding to the time series image set is obtained; wherein, dense optical flow is an image registration method that performs point-by-point matching on images. Unlike sparse optical flow which only matches a number of feature points on an image, dense optical flow calculates the offset of all points on the image, thereby forming a dense optical flow field.

[0081] Optical flow is a representation of the speed of image movement. When a motion vector is given to each pixel in the image, the entire image forms an optical flow field. By calculating the changes in each pixel in the previous and next frames, the motion vector of each pixel is obtained. Pixels with similar motion vectors constitute the same fish body, so the fish body can be detected and identified.

[0082] In order to obtain the real density image of the school of fish, the head of the fish is used as the detection area to mark the entire fish body; after Gaussian filtering and normalization of the time series image set containing the mark, the filtering result is the density image corresponding to the time series image set. The density image represents the distribution of the school of fish in the feeding behavior scene of the school of fish. The total number of fish in the feeding behavior scene of the school of fish can be obtained by integrating or summing the density map.

[0083] Among them, when the fish school is dense, the occlusion overlap between the fish causes the head area of ​​the fish displayed in the image to become smaller; when the fish school is sparsely distributed, the occlusion overlap between the fish school is less, and the head area of ​​the fish in the image is larger.

[0084] Step S20, based on the spatial image, the optical flow image, the density image and a preset attention mechanism model, extract the first feature corresponding to each spatial image, the second feature corresponding to each optical flow image and the third feature corresponding to each density image.

[0085] In this embodiment, according to the spatial images, optical flow images and density images obtained above, convolution, pooling and residual summation are performed on each spatial image, each optical flow image and each density image through a preset attention mechanism model to extract the internal features of a single spatial image of each spatial image, the internal features of a single spatial image of each optical flow image and the internal features of a single spatial image of each density image; wherein the internal features of a single spatial image of each spatial image, that is, the first features corresponding to each spatial image; the internal features of a single spatial image of each optical flow image, that is, the second features corresponding to each optical flow image; the internal features of a single spatial image of each density image, that is, the third features corresponding to each density image.

[0086] For example, the size of the input spatial image is [3, 640, 360]. First, the input spatial image is subjected to feature extraction through two layers of two-dimensional convolution to obtain the features of [32, 640, 360]. Then, the features are compressed to [32, 320, 180] through a maximum pooling layer. Then, a two-dimensional convolution layer and a maximum pooling layer are used to obtain low-dimensional features of [64, 160, 90]. Finally, high-dimensional features are extracted through three repeated two-dimensional residual blocks to finally form the internal features of a single spatial image of [64, 160, 90], which is the first feature corresponding to each spatial image.

[0087] Step S30, based on the first feature, the second feature and the third feature, feature association is performed through a preset attention mechanism model to determine the corresponding clustering feature.

[0088] In one embodiment, according to the first feature corresponding to each spatial image, the second feature corresponding to each optical flow image, and the third feature corresponding to each density image, the first feature, the second feature, and the third feature are input into a preset attention mechanism model; by using the preset attention mechanism model to perform attention fusion on the first feature, the second feature, and the third feature, the first feature, the second feature, and the third feature are sequentially performed with the feature fusion of the internal features of a single image, the feature fusion between multiple images, and the feature fusion between features in different domains to determine the fusion features corresponding to the spatial image, the optical flow image, and the density image.

[0089] Step S40: identifying the feeding behavior of the fish school based on the clustering features.

[0090] In this embodiment, the feeding process of fish is divided into multiple levels through the fusion features corresponding to the spatial image, optical flow image, and density image, and the feeding state and hunger level of the fish are identified; thereby, the feeding behavior state of the fish in the time series image set can be accurately identified, thereby achieving precise feeding control of the fish.

[0091] This embodiment obtains a time-series image set of the feeding behavior of a school of fish and performs image preprocessing on the time-series image set to obtain corresponding spatial images, optical flow images, and density images; based on the spatial images, optical flow images, and density images, a preset attention mechanism model is used to extract a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image; based on the first feature, the second feature, and the third feature, a preset attention mechanism model is used to perform feature fusion on the first feature, the second feature, and the third feature to determine the corresponding fused feature; based on the fused feature, the feeding behavior of the school of fish is identified; thereby, precise feeding control is achieved.

[0092] Furthermore, based on the first embodiment of the method for identifying the feeding behavior of a school of fish of the present invention, a second embodiment of the method for identifying the feeding behavior of a school of fish of the present invention is proposed.

[0093] The difference between the second embodiment of the method for identifying the feeding behavior of a school of fish and the first embodiment of the method for identifying the feeding behavior of a school of fish is that this embodiment is a refinement of the preset attention mechanism model in step S20, specifically including:

[0094] The preset attention mechanism model includes a convolution transposition module, or an attention module, or a combination of a convolution transposition module and an attention module.

[0095] In this embodiment, the preset attention mechanism model is preferably a convolution transposition module, or an attention module, or a combination of a convolution transposition module and an attention module. According to different task requirements, the preset attention mechanism model can be composed of at least one convolution transposition module, or at least one attention module, or a combination of at least one convolution transposition module and at least one attention module.

[0096] Reference Figure 3 , the process of the preset attention mechanism model: first, the input feature is convolved through a 1*1 convolution layer of the convolution transposition module to stretch the data dimension of the input feature; then the result of the convolution is multiplied by the transposition of the input feature, and finally the data feature after the input feature is compressed can be obtained. The principle is: the input feature is compressed through a 1*1 convolution operation to achieve feature dimension compression, and then the output of the 1*1 convolution layer is used as the selection matrix of the input feature to achieve segmentation of the input feature.

[0097] Furthermore, the output of the convolution transpose module is used as the input of the attention module, and at the same time, it passes through two convolutions to represent the query and key in the attention mechanism model respectively, and the data input to the attention module is directly used as the value in the preset attention mechanism model for subsequent operations. Then the results of the two convolution operations are multiplied and added to the input of the attention module, and finally the attention mechanism feature is obtained, that is, the fusion feature corresponding to the input feature.

[0098] The preset attention mechanism model in the present invention has the performance of feature compression and feature extraction, and can compress the input features while stretching the dimensions of the input features; by using the preset attention mechanism model, the accuracy of fish feeding behavior recognition can be improved.

[0099] Furthermore, based on the first and second embodiments of the method for identifying the feeding behavior of a school of fish of the present invention, a third embodiment of the method for identifying the feeding behavior of a school of fish of the present invention is proposed.

[0100] The third embodiment of the method for identifying the feeding behavior of a school of fish is different from the first and second embodiments of the method for identifying the feeding behavior of a school of fish in that the present embodiment is a refinement of step S20, which extracts the first feature corresponding to each spatial image, the second feature corresponding to each optical flow image, and the third feature corresponding to each density image based on the spatial image, the optical flow image, the density image, and the preset attention mechanism model. Figure 4 , the step specifically includes:

[0101] Step S21, according to the spatial image, the optical flow image and the density image, performing convolution and pooling processing on each spatial image, each optical flow image and each density image through the preset attention mechanism model to obtain a processed spatial image, a processed optical flow image and a processed density image;

[0102] Step S22, performing residual summation processing on the processed spatial image, the processed optical flow image and the processed density image for a preset number of times to obtain a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image and a third feature corresponding to each density image.

[0103] In this embodiment, according to the spatial image, the optical flow image and the density image, convolution and pooling processing are performed on each spatial image, each optical flow image and each density image of the feeding behavior of the fish school through a preset attention mechanism model to obtain a processed spatial image, a processed optical flow image and a processed density image; then, residual summation processing is performed on the processed spatial image, the processed optical flow image and the processed density image for a preset number of times to obtain a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image and a third feature corresponding to each density image; thereby, high-precision feature extraction of the spatial image, the optical flow image and the density image is achieved, thereby improving the accuracy of feeding behavior recognition.

[0104] The following is a detailed description of each step:

[0105] Step S21, according to the spatial image, the optical flow image and the density image, convolution and pooling processing are performed on each spatial image, each optical flow image and each density image through the preset attention mechanism model to obtain a processed spatial image, a processed optical flow image and a processed density image.

[0106] In this embodiment, after obtaining the spatial image, optical flow image and density image of the feeding behavior of the school of fish, the features of each spatial image, optical flow image and density image are extracted through two layers of two-dimensional convolution through a preset attention mechanism model; then the result after the convolution processing is subjected to feature compression through a layer of maximum pooling layer; and then through a layer of two-dimensional convolution layer and a layer of maximum pooling layer, the processed spatial image, the processed optical flow image and the processed density image are obtained.

[0107] Reference Figure 5 For example, if the size of the input spatial image is [3, 640, 360], firstly, the input spatial image is subjected to feature extraction through two layers of two-dimensional convolution to obtain the features of [32, 640, 360]; then, the features are compressed to [32, 320, 180] through the maximum pooling layer; and then, a processed spatial image with low-dimensional features of [64, 160, 90] is obtained through a layer of two-dimensional convolution layer and a layer of maximum pooling layer. Among them, the 3 in [3, 640, 360] represents the RGB three channels of the spatial image, and 640*360 is the pixel size of the spatial image.

[0108] Step S22, performing residual summation processing on the processed spatial image, the processed optical flow image and the processed density image for a preset number of times to obtain a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image and a third feature corresponding to each density image.

[0109] In this embodiment, the processed spatial image, the processed optical flow image, and the processed density image are subjected to high-dimensional feature extraction through three repeated two-dimensional residual blocks, and finally a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image are formed.

[0110] Among them, the two-dimensional residual block consists of a layer of two-dimensional convolution and a summation. That is, the input processed spatial image, processed optical flow image and processed density image are compressed through the convolution layer; then the compressed features and input features are summed through a summation layer to obtain the internal features of a single spatial image, the internal features of a single optical flow image and the internal features of a single density image; among them, the internal features of a single spatial image are the first features corresponding to each spatial image; the internal features of a single optical flow image are the second features corresponding to each optical flow image; the internal features of a single density image are the third features corresponding to each density image.

[0111] Reference Figure 5 For example, the low-dimensional features are used to extract high-dimensional features through three repeated two-dimensional residual blocks, and finally the internal features of a single image of [64, 160, 90] are formed, which is the first feature corresponding to each spatial image.

[0112] In this embodiment, according to the spatial image, the optical flow image and the density image, convolution and pooling processing are performed on each spatial image, each optical flow image and each density image of the feeding behavior of the school of fish through a preset attention mechanism model to obtain a processed spatial image, a processed optical flow image and a processed density image; then, residual summation processing is performed on the processed spatial image, the processed optical flow image and the processed density image for a preset number of times to obtain a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image and a third feature corresponding to each density image; thereby, high-precision feature extraction of the spatial image, the optical flow image and the density image is achieved, thereby improving the accuracy of identifying the feeding behavior of the school of fish.

[0113] Furthermore, based on the first, second and third embodiments of the method for identifying the feeding behavior of a school of fish of the present invention, a fourth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention is proposed.

[0114] The fourth embodiment of the method for identifying the feeding behavior of a school of fish is different from the first, second and third embodiments of the method for identifying the feeding behavior of a school of fish in that the fourth embodiment is to perform feature fusion based on the first feature, the second feature and the third feature through the preset attention mechanism model in step S30 to determine the refinement of the corresponding fusion feature, referring to Figure 6 , the step specifically includes:

[0115] Step S31, according to the first feature, the second feature and the third feature, the internal feature fusion of the single image is performed through the preset attention mechanism model to obtain a first internal feature corresponding to the first feature, a second internal feature corresponding to the second feature, and a third internal feature corresponding to the third feature;

[0116] Step S32, according to the first internal feature, the second internal feature and the third internal feature, feature fusion between multiple images is performed through the preset attention mechanism model to obtain a first associated feature corresponding to the first internal feature, a second associated feature corresponding to the second internal feature, and a third associated feature corresponding to the third internal feature;

[0117] Step S33: Based on the first association feature, the second association feature and the third association feature, feature fusion between different domain features is performed through the preset attention mechanism model to obtain corresponding fusion features.

[0118] This embodiment uses a preset attention mechanism model to perform feature fusion within a single image, feature fusion between multiple images in each channel, and feature fusion between features in different domains on the first feature of each spatial image, the second feature of each optical flow image, and the third feature of each density image to obtain corresponding fusion features; thereby, the anti-interference ability of fish feeding behavior recognition is improved through the complementarity between the three different features.

[0119] The following is a detailed description of each step:

[0120] Step S31, based on the first feature, the second feature and the third feature, the internal features of the single image are fused through the preset attention mechanism model to obtain the first internal feature corresponding to the first feature, the second internal feature corresponding to the second feature, and the third internal feature corresponding to the third feature.

[0121] In this embodiment, the first feature of each spatial image, the second feature of each optical flow image, and the third feature of each density image are input into a preset attention mechanism model. The preset attention mechanism model is composed of a convolution transposition module and an attention module; according to the requirements of different tasks, the preset attention mechanism model can be composed of a convolution transposition module with one or more attention modules, or one or more attention modules.

[0122] In this embodiment, the preset attention mechanism model consists of a convolution transposition module and an attention module; after the first feature, the second feature, and the third feature are fused into a single image through a convolution transposition module and an attention module, a first internal feature corresponding to the first feature, a second internal feature corresponding to the second feature, and a third internal feature corresponding to the third feature are obtained.

[0123] Reference Figure 7 The first feature S12 of the spatial image, the second feature S22 of the optical flow image, and the third feature S32 of the density image are fused through a convolution transposition module and an attention module to obtain the corresponding single-image internal feature association S13 of the spatial image, that is, the first internal feature corresponding to the first feature; the single-image internal feature association S23 of the optical flow image, that is, the second internal feature corresponding to the second feature; and the single-image internal feature association S33 of the density image, that is, the third internal feature corresponding to the third feature.

[0124] Step S32, based on the first internal feature, the second internal feature and the third internal feature, feature fusion between multiple images is performed through the preset attention mechanism model to obtain a first associated feature corresponding to the first internal feature, a second associated feature corresponding to the second internal feature, and a third associated feature corresponding to the third internal feature.

[0125] In this embodiment, the preset attention mechanism model is composed of two convolution transposition modules. The first internal feature, the second internal feature, and the third internal feature are subjected to image integration, convolution and pooling, and two attention modules to perform attention relationship feature fusion processing between frames of the video of the same channel, thereby obtaining a first correlation feature corresponding to the first internal feature, a second correlation feature corresponding to the second internal feature, and a third correlation feature corresponding to the third internal feature.

[0126] Reference Figure 7 The first internal feature S13 of the spatial image, the second internal feature S23 of the optical flow image, and the third internal feature S33 of the density image are subjected to image integration, convolution and pooling, and two attention modules to perform attention relationship feature fusion processing between frames of the video of the same channel, thereby obtaining the corresponding fusion feature S14 between multiple images of the spatial image, that is, the first association feature; the fusion feature S24 between multiple pictures of the optical flow image, that is, the second association feature; and the fusion feature S34 between multiple pictures of the density image, that is, the third association feature.

[0127] Step S33: Based on the first association feature, the second association feature and the third association feature, feature fusion between different domain features is performed through the preset attention mechanism model to obtain corresponding fusion features.

[0128] In this embodiment, the preset attention mechanism model is composed of a convolution transposition module and an attention module. The first associated feature, the second associated feature, and the third associated feature are spliced ​​through channels, and a convolution transposition module and an attention module are used to perform attention acquisition processing between different channels to obtain corresponding fusion features.

[0129] Reference Figure 7 The first associated feature S14 of the spatial image, the second associated feature S24 of the optical flow image, and the third associated feature S34 of the density image are processed through channel splicing, a convolution transposition module, and an attention module to obtain attention between different channels, and the corresponding fusion feature S4 between different images, that is, the fusion feature, is obtained.

[0130] In this embodiment, the first feature of each spatial image, the second feature of each optical flow image, and the third feature of each density image are subjected to feature fusion within a single image, feature fusion between multiple images of each channel, and feature fusion between features of different domains through a preset attention mechanism model to obtain corresponding fusion features; thereby, the anti-interference ability of fish feeding behavior recognition is improved through the complementarity between the three different features.

[0131] Furthermore, based on the first, second, third and fourth embodiments of the method for identifying the feeding behavior of a school of fish of the present invention, a fifth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention is proposed.

[0132] The fifth embodiment of the method for identifying the feeding behavior of a school of fish differs from the first, second, third and fourth embodiments of the method for identifying the feeding behavior of a school of fish in that the present embodiment is to refine the first internal feature corresponding to the first feature, the second internal feature corresponding to the second feature and the third internal feature corresponding to the third feature by using the preset attention mechanism model in step S31 according to the first feature, the second feature and the third feature. Figure 8 , the step specifically includes:

[0133] Step A10, performing a convolution operation on the first feature, the second feature, and the third feature through the convolution transposition module, and performing feature matrix transposition multiplication on the result of the convolution and the first feature, the second feature, and the third feature to obtain a first convolution feature corresponding to the first feature, a second convolution feature corresponding to the second feature, and a third convolution feature corresponding to the third feature;

[0134] Step A20, performing a preset number of convolution operations on the first convolution feature, the second convolution feature, and the third convolution feature through the attention module, and performing feature matrix multiplication on the convolution results to obtain a first product feature corresponding to the first convolution feature, a second product feature corresponding to the second convolution feature, and a third product feature corresponding to the third convolution;

[0135] Step A30, respectively summing the first product feature, the first product feature and the first product feature with the first convolution feature, the second convolution feature and the third convolution feature to obtain the corresponding first internal feature, the second internal feature and the third internal feature.

[0136] In this embodiment, a convolution transposition module is used to convolve the first feature, the second feature, and the third feature, and perform feature matrix transposition and multiplication processing to obtain a first convolution feature corresponding to the first feature, a second convolution feature corresponding to the second feature, and a third convolution feature corresponding to the third feature; the attention module is used to simultaneously perform a preset number of convolutions on the first convolution feature, the second convolution feature, and the third convolution feature, and then multiply them and add them to the first convolution feature, the second convolution feature, and the third convolution feature, respectively, to obtain the corresponding first internal feature, the second internal feature, and the third internal feature; thereby achieving the accuracy of the internal features of a single image.

[0137] The following is a detailed description of each step:

[0138] Step A10, performing a convolution operation on the first feature, the second feature, and the third feature through the convolution transposition module, and performing feature matrix transposition multiplication on the result of the convolution and the first feature, the second feature, and the third feature to obtain a first convolution feature corresponding to the first feature, a second convolution feature corresponding to the second feature, and a third convolution feature corresponding to the third feature.

[0139] In this embodiment, the preset attention mechanism model consists of a convolution transposition module and an attention module; a convolution operation is performed on the first feature, the second feature, and the third feature, and the result of the convolution operation is multiplied by the transposition of the feature matrix with the first feature, the second feature, and the third feature, respectively, to obtain a first convolution feature corresponding to the first feature, a second convolution feature corresponding to the second feature, and a third convolution feature corresponding to the third feature.

[0140] Reference Fig. 9For example, the first feature of the input single spatial image is [64, 160, 90]. After the first feature [64, 160, 90] is subjected to a 1*1 convolution [1*1, 128] operation, the channel is stretched to 128-dimensional features [128, 160, 90]. Then, the feature [128, 160, 90] is multiplied by the transpose of the first feature [64, 160, 90] to obtain the corresponding first convolution feature [128, 64].

[0141] Step A20, performing a preset number of convolution operations on the first convolution feature, the second convolution feature and the third convolution feature through the attention module, and multiplying the convolution results by feature matrices to obtain a first product feature corresponding to the first convolution feature, a second product feature corresponding to the second convolution feature, and a third product feature corresponding to the third convolution.

[0142] In this embodiment, the preset attention mechanism model consists of a convolution transposition module and an attention module; the first convolution feature, the second convolution feature, and the third convolution feature are simultaneously subjected to two different convolutions by the attention module, and the results of the two convolution operations are multiplied by feature matrices to obtain a first product feature corresponding to the first convolution feature, a second product feature corresponding to the second convolution feature, and a third product feature corresponding to the third convolution. The actual preset number of convolution operations needs to be set according to actual conditions.

[0143] Reference Fig. 9 For example, the first convolution feature [128, 64] is simultaneously passed through two 3*3 two-dimensional convolutions with different step sizes. The first two-dimensional convolution is (3*3, s=(1, 1)) and the second two-dimensional matrix is ​​(3*3, s=(2, 1)), and the features with dimensions of [128, 64] and [64, 64] are obtained respectively; then the feature [128, 64] and the feature [64, 64] are multiplied by the feature matrix to obtain the first product feature [128, 64] corresponding to the first convolution feature.

[0144] Step A30, respectively summing the first product feature, the first product feature and the first product feature with the first convolution feature, the second convolution feature and the third convolution feature to obtain the corresponding first internal feature, the second internal feature and the third internal feature.

[0145] In this embodiment, the preset attention mechanism model consists of a convolution transpose module and an attention module; the first product feature, the second product feature, and the third product feature are respectively summed with the first convolution feature, the second convolution feature, and the third convolution feature to obtain the corresponding first internal feature, the second internal feature, and the third internal feature.

[0146] Reference Fig. 9 For example, the first product feature [128, 64] is summed with the first convolution feature [128, 64] to obtain the corresponding single-image internal relationship feature [128, 64] of the spatial image, that is, the first internal feature [128, 64].

[0147] In this embodiment, the convolution transposition module performs convolution and feature transposition multiplication on the first feature, the second feature, and the third feature to obtain a first convolution feature corresponding to the first feature, a second convolution feature corresponding to the second feature, and a third convolution feature corresponding to the third feature; the first convolution feature, the second convolution feature, and the third convolution feature are simultaneously multiplied after a preset number of convolutions through the attention module and added to the first convolution feature, the second convolution feature, and the third convolution feature, respectively, to obtain the corresponding first internal feature, the second internal feature, and the third internal feature; thereby achieving the accuracy of the internal features of a single image.

[0148] Furthermore, based on the first, second, third, fourth and fifth embodiments of the method for identifying the feeding behavior of a school of fish of the present invention, a sixth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention is proposed.

[0149] The sixth embodiment of the method for identifying the feeding behavior of a school of fish differs from the first, second, third, fourth and fifth embodiments of the method for identifying the feeding behavior of a school of fish in that the present embodiment is to refine step S32 by performing feature fusion between multiple images according to the first internal feature, the second internal feature and the third internal feature through the preset attention mechanism model to obtain the first associated feature corresponding to the first internal feature, the second associated feature corresponding to the second internal feature and the third associated feature corresponding to the third internal feature, with reference to Fig.10 , the step specifically includes:

[0150] Step B10, performing feature integration processing on the first internal feature, the second internal feature, and the third internal feature, respectively, to obtain a first integrated feature corresponding to the first internal feature, a second integrated feature corresponding to the second internal feature, and a third integrated feature corresponding to the third internal feature;

[0151] Step B20, performing residual operation on the first integrated feature, the second integrated feature, and the third integrated feature to obtain a first sum feature corresponding to the first integrated feature, a second sum feature corresponding to the second integrated feature, and a third sum feature corresponding to the third integrated feature;

[0152] Step B30, performing a pooling operation on the first sum feature, the second sum feature, and the third sum feature to obtain a corresponding pooled first sum feature, a pooled second sum feature, and a pooled third sum feature;

[0153] Step B40, performing a preset number of convolution operations on the first pooled sum feature, the second pooled sum feature, and the third pooled sum feature through the convolution transposition module, and performing feature matrix multiplication on the result after the convolution and the first pooled sum feature, the second pooled sum feature, and the third pooled sum feature, respectively, and then summing them to obtain the corresponding first product feature, second product feature, and third product feature;

[0154] Step B50, performing a pooling operation on the first product feature, the second product feature, and the third product feature to obtain a pooled first product feature, a pooled second product feature, and a pooled third product feature;

[0155] Step B60, performing a preset number of convolution operations on the first pooled product feature, the second pooled product feature, and the third pooled product feature through the convolution transposition module, and performing feature matrix multiplication on the result of the convolution and the first pooled product feature, the second pooled product feature, and the third pooled product feature, respectively, and then summing them up to obtain the corresponding first intent feature, second intent feature, and third intent feature;

[0156] Step B70: pooling the first intention feature, the second intention feature and the third intention feature to obtain corresponding first association features, second association features and third association features.

[0157] This embodiment integrates the first association feature, the second association feature, and the third association feature for multiple images in the same channel to obtain the corresponding first integrated feature, the second integrated feature, and the third integrated feature; then the first integrated feature, the second integrated feature, and the third integrated feature are subjected to two pooling and a preset attention mechanism model to fuse the features of multiple images in the same channel to obtain the corresponding first associated feature, the second associated feature, and the third associated feature; thereby achieving the accuracy of the association features corresponding to multiple images in the same channel.

[0158] The following is a detailed description of each step:

[0159] Step B10, performing feature integration processing on the first internal feature, the second internal feature and the third internal feature respectively to obtain a first integrated feature corresponding to the first internal feature, a second integrated feature corresponding to the second internal feature and a third integrated feature corresponding to the third internal feature.

[0160] In this embodiment, the set of time-series images within the same time period is integrated, that is, the first internal feature of the spatial image, the second internal feature of the optical flow image, and the third internal feature of the density image within the same time period are integrated for each channel to obtain a first integrated feature corresponding to the first internal feature, a second integrated feature corresponding to the second internal feature, and a third integrated feature corresponding to the third internal feature.

[0161] Reference Fig.11 For example, if the time series image set within the same time period consists of 3000 images, the internal features [128, 64] corresponding to these 3000 spatial images, 3000 optical flow images, and 3000 density images are subjected to feature integration processing within the same channel to obtain the first integrated feature [3000, 128, 64] corresponding to the first internal feature, the second integrated feature [3000, 128, 64] corresponding to the second internal feature, and the third integrated feature [3000, 128, 64] corresponding to the third internal feature.

[0162] Step B20, performing residual operation on the first integrated feature, the second integrated feature and the third integrated feature to obtain a first sum feature corresponding to the first integrated feature, a second sum feature corresponding to the second integrated feature and a third sum feature corresponding to the third integrated feature.

[0163] In this embodiment, the preset attention mechanism model is composed of two attention modules. A residual sum operation is performed on the first integrated feature, the second integrated feature, and the third integrated feature, that is, the first integrated feature, the second integrated feature, and the third integrated feature are summed with the result of the three-dimensional convolution to obtain a first sum feature corresponding to the first integrated feature, a second sum feature corresponding to the second integrated feature, and a third sum feature corresponding to the third integrated feature.

[0164] Reference Fig.11, perform three-dimensional convolution operation on the first integrated feature [3000, 128, 64], the second integrated feature [3000, 128, 64], and the third integrated feature [3000, 128, 64], and sum the result of the three-dimensional convolution with the first integrated feature [3000, 128, 64], the second integrated feature [3000, 128, 64], and the third integrated feature [3000, 128, 64], respectively, to obtain the first sum feature [3000, 128, 64] corresponding to the first integrated feature, the second sum feature [3000, 128, 64] corresponding to the second integrated feature, and the third sum feature [3000, 128, 64] corresponding to the third integrated feature.

[0165] Step B30, performing a pooling operation on the first sum feature, the second sum feature, and the third sum feature to obtain the corresponding pooled first sum feature, the pooled second sum feature, and the pooled third sum feature.

[0166] In this embodiment, the preset attention mechanism model consists of two attention modules. The first sum feature, the second sum feature, and the third sum feature are pooled through a maximum pooling layer to obtain the corresponding pooled first sum feature, the pooled second sum feature, and the pooled third sum feature.

[0167] Reference Fig.11 For example, the first and feature [3000, 128, 64] of the spatial image is pooled through a maximum pooling layer to obtain the corresponding pooled first and feature [3000, 1000].

[0168] Step B40, performing a preset number of convolution operations on the first pooled sum feature, the second pooled sum feature and the third pooled sum feature through the attention module, and multiplying the convolution result with the first pooled sum feature, the second pooled sum feature and the third pooled sum feature by the feature matrix respectively, and summing them to obtain the corresponding first product feature, second product feature and third product feature.

[0169] In this embodiment, the preset attention mechanism model consists of two attention modules. The first pooled sum feature, the second pooled sum feature, and the third pooled sum feature are simultaneously passed through two different convolutional layers through convolution transposition, and the results of the two convolution operations are respectively multiplied by the feature matrix with the first pooled sum feature, the second pooled sum feature, and the third pooled sum feature, and then summed to obtain the corresponding first product feature, second product feature, and third product feature.

[0170] Reference Fig.11For example, through convolution transposition, the pooled first sum feature [3000, 1000] of the spatial image is simultaneously passed through two two-dimensional convolution layers with different step lengths. The first two-dimensional convolution is (3*3, s=(1, 1)), and the second two-dimensional matrix is ​​(6*6, s=(3, 1)), and features with dimensions of [3000, 1000] and [1000, 1000] are obtained respectively; then the feature [3000, 1000] and the feature [1000, 1000] are respectively multiplied by the feature matrix with the first sum feature [3000, 1000] after pooling, and the multiplication results are summed to obtain the first product feature [3000, 1000] corresponding to the first sum feature after pooling.

[0171] Step B50, performing a pooling operation on the first product feature, the second product feature, and the third product feature to obtain a pooled first product feature, a pooled second product feature, and a pooled third product feature.

[0172] In this embodiment, the preset attention mechanism model is composed of two attention modules. The first product feature, the second product feature, and the third product feature are pooled through a maximum pooling layer to obtain a pooled first product feature corresponding to the first product feature, a pooled second product feature corresponding to the second product feature, and a pooled third product feature corresponding to the third product feature.

[0173] Reference Fig.11 For example, the first product feature [3000, 1000] of the spatial image is pooled through a maximum pooling layer to obtain the corresponding first pooled product feature [1000, 500].

[0174] Step B60, performing a preset number of convolution operations on the first pooled product feature, the second pooled product feature and the third pooled product feature through the attention module, and multiplying the convolution result with the first pooled product feature, the second pooled product feature and the third pooled product feature by the feature matrix respectively, and summing them up to obtain the corresponding first intention feature, second intention feature and third intention feature.

[0175] In this embodiment, the preset attention mechanism model consists of two attention modules. The first pooled product feature, the second pooled product feature, and the third pooled product feature are simultaneously passed through two different convolutional layers through the attention module, and the results of the two convolution operations are respectively multiplied by the first pooled product feature, the second pooled product feature, and the third pooled product feature after feature matrix multiplication and summed to obtain the corresponding first intention feature, second intention feature, and third intention feature. The actual preset number of convolution operations needs to be set according to actual conditions.

[0176] Reference Fig.11 For example, the first product feature [1000, 500] of the spatial image after pooling is passed through two two-dimensional convolution layers with different step lengths at the same time. The first two-dimensional convolution is (3*3, s=(1, 1)) and the second two-dimensional matrix is ​​(6*6, s=(3, 1)) to obtain features with dimensions [1000, 500] and [500, 500] respectively; then the feature [1000, 500] and the feature [500, 500] are respectively multiplied by the first product feature [1000, 500] after pooling, and the multiplication results are summed to obtain the corresponding first intent feature [1000, 500].

[0177] Step B70: pooling the first intention feature, the second intention feature and the third intention feature to obtain corresponding first association features, second association features and third association features.

[0178] In this embodiment, the preset attention mechanism model is composed of two attention modules. The first intention feature, the second intention feature, and the third intention feature are pooled through a maximum pooling layer to obtain the corresponding first association feature, the second association feature, and the third association feature.

[0179] Reference Fig.11 For example, the first intention feature [1000, 500] of the spatial image is pooled through a maximum pooling layer to obtain the corresponding video relationship feature [500, 250], that is, the first association feature [500, 250].

[0180] In this embodiment, the first association feature, the second association feature, and the third association feature are integrated with multiple images in the same channel to obtain corresponding first integrated features, second integrated features, and third integrated features; then the first integrated features, the second integrated features, and the third integrated features are subjected to two pooling and a preset attention mechanism model to perform feature fusion processing on multiple images in the same channel to obtain corresponding first associated features, second associated features, and third associated features; thereby achieving the accuracy of the association features corresponding to multiple images in the same channel.

[0181] Furthermore, based on the first, second, third, fourth and fifth embodiments of the method for identifying the feeding behavior of a school of fish of the present invention, a sixth embodiment of the method for identifying the feeding behavior of a school of fish of the present invention is proposed.

[0182] The sixth embodiment of the method for identifying the feeding behavior of a school of fish differs from the first, second, third, fourth and fifth embodiments of the method for identifying the feeding behavior of a school of fish in that the present embodiment is to refine the corresponding fusion features by performing feature fusion between different domain features through the preset attention mechanism model according to the first association feature, the second association feature and the third association feature in step S33. Fig.12 , the step specifically includes:

[0183] Step C10, performing channel splicing on the first association feature, the second association feature and the third association feature to obtain a corresponding three-dimensional matrix feature;

[0184] Step C20, performing a convolution operation on the three-dimensional matrix feature through the convolution transposition module, and performing feature matrix multiplication on the result of the convolution and the three-dimensional matrix feature to obtain a coding feature corresponding to the three-dimensional matrix feature;

[0185] Step C30, performing a preset number of convolution operations on the coding feature through the attention module, and multiplying the result after the convolution by a feature matrix to obtain a product coding feature corresponding to the coding feature;

[0186] Step C40, performing feature summation on the product coding feature to obtain a fusion feature corresponding to the product coding feature.

[0187] This embodiment obtains corresponding three-dimensional matrix features by channel splicing the first associated features, the second associated features, and the third associated features; the three-dimensional matrix features are subjected to feature normalization and fusion processing through a preset attention mechanism model composed of a convolution transposition module and an attention module to obtain fusion features corresponding to the product coding features; thereby achieving the accuracy of the fusion features of the multi-channel image.

[0188] The following is a detailed description of each step:

[0189] Step C10: performing channel splicing on the first association feature, the second association feature and the third association feature to obtain a corresponding three-dimensional matrix feature.

[0190] In this embodiment, the preset attention mechanism model is composed of a convolution transposition module and an attention module. The first associated feature, the second associated feature, and the third associated feature are compressed and concatenated to form a three-dimensional matrix feature corresponding to the first associated feature, the second associated feature, and the third associated feature.

[0191] Reference Fig.13 The first associated feature [500, 250] of the spatial image, the second associated feature [500, 250] of the optical flow image, and the third associated feature [500, 250] of the density image are subjected to channel splicing processing of attention compression and splicing to obtain the corresponding three-dimensional matrix feature [3, 500, 250].

[0192] Step C20, performing a convolution operation on the three-dimensional matrix feature through the convolution transposition module, and performing feature matrix multiplication on the result of the convolution and the three-dimensional matrix feature to obtain a coding feature corresponding to the three-dimensional matrix feature.

[0193] In this embodiment, the preset attention mechanism model consists of a convolution transposition module and an attention module. The convolution transposition module performs a convolution operation on the three-dimensional matrix feature, and multiplies the result of the convolution operation by the transposition of the three-dimensional matrix feature to obtain the encoding feature corresponding to the three-dimensional matrix feature.

[0194] Reference Fig.13 For example, the three-dimensional matrix feature [3, 500, 250] is subjected to a convolution [1*1, 128] operation through the convolution transpose module to obtain the feature [3, 128, 250], and the feature [3, 128, 250] is multiplied by the transpose of the three-dimensional matrix feature [3, 500, 250] to obtain the reorganized encoding feature [128, 250].

[0195] Step C30, performing a preset number of convolution operations on the coding feature through the attention module, and multiplying the result of the convolution by a feature matrix to obtain a product coding feature corresponding to the coding feature.

[0196] In this embodiment, the preset attention mechanism model consists of a convolution transposition module and an attention module. The coding feature is simultaneously passed through two different convolutions by the attention module, and the results of the two convolution operations are multiplied by feature matrices to obtain the product coding feature corresponding to the coding feature. The actual preset number of convolution operations needs to be set according to the actual situation.

[0197] Reference Fig.13For example, the encoded feature [128, 250] is simultaneously passed through two two-dimensional convolutions with different step lengths through the attention module. The first two-dimensional convolution is (3*3, s=(1, 1)) and the second two-dimensional matrix is ​​(1*1, s=(2, 1)), and the features with dimensions of [128, 500] and [500, 500] are obtained respectively; then the feature [128, 500] and the feature [500, 500] are multiplied by the feature matrix to obtain the corresponding product encoded feature [128, 500].

[0198] Step C40, performing feature summation on the product coding feature to obtain a fusion feature corresponding to the product coding feature.

[0199] In this embodiment, the preset attention mechanism model consists of a convolution transposition module and an attention module. The product coding feature and the coding feature are summed to obtain a fusion feature corresponding to the product coding feature.

[0200] Reference Fig.13 For example, the product coding feature [128, 500] is summed with the coding feature [128, 500] to obtain the multi-channel feature [128, 500], that is, the fusion feature corresponding to the product coding feature is obtained.

[0201] In this embodiment, the first associated feature, the second associated feature, and the third associated feature are channel-joined to obtain corresponding three-dimensional matrix features; the three-dimensional matrix features are subjected to feature normalization and fusion processing through a preset attention mechanism model consisting of a convolution transpose module and an attention module to obtain fusion features corresponding to the product coding features; thereby achieving the accuracy of the fusion features of the multi-channel image.

[0202] The present invention also provides a device for identifying the feeding behavior of a school of fish. Fig.14 The device for identifying the feeding behavior of a school of fish of the present invention comprises:

[0203] An acquisition module 10 is used to acquire a time-series image set of the feeding behavior of a school of fish, and pre-process the time-series image set using a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model to obtain corresponding spatial images, optical flow images, and density images;

[0204] An extraction module 20, configured to extract a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image based on the spatial image, the optical flow image, the density image, and a preset attention mechanism model;

[0205] A fusion module 30, configured to perform feature fusion based on the first feature, the second feature and the third feature through the preset attention mechanism model to determine a corresponding fusion feature;

[0206] The identification module 40 is used to identify the feeding behavior of the fish school based on the fusion features.

[0207] In addition, the present invention also provides a medium, which is a computer-readable storage medium, on which a program for identifying fish feeding behavior is stored. When the program for identifying fish feeding behavior is executed by a processor, the steps of the method for identifying fish feeding behavior as described above are implemented.

[0208] Among them, the method implemented when the fish feeding behavior recognition program running on the processor is executed can refer to the various embodiments of the fish feeding behavior recognition method of the present invention, which will not be repeated here.

[0209] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0210] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0211] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0212] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for identifying feeding behavior of a school of fish, characterized in that: The method for identifying the feeding behavior of a school of fish comprises the following steps: Acquire a time-series image set of the feeding behavior of a school of fish, and pre-process the time-series image set using a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model, respectively, to obtain corresponding spatial images, optical flow images, and density images; Based on the spatial image, the optical flow image, the density image and a preset attention mechanism model, extracting a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image; Based on the first feature, the second feature and the third feature, feature fusion is performed through the preset attention mechanism model to determine a corresponding fusion feature; Based on the fusion features, the feeding behavior of the fish school is identified; The step of extracting the first feature corresponding to each spatial image, the second feature corresponding to each optical flow image, and the third feature corresponding to each density image based on the spatial image, the optical flow image, the density image, and a preset feature extraction model comprises: According to the spatial image, the optical flow image and the density image, convolution and pooling processing are performed on each spatial image, each optical flow image and each density image through the preset attention mechanism model to obtain a processed spatial image, a processed optical flow image and a processed density image; Performing a preset number of residual summation processes on the processed spatial image, the processed optical flow image, and the processed density image to obtain a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image; The step of performing feature fusion based on the first feature, the second feature and the third feature through the preset attention mechanism model to determine the corresponding fusion feature includes: According to the first feature, the second feature and the third feature, the internal features of the single image are fused through the preset attention mechanism model to obtain a first internal feature corresponding to the first feature, a second internal feature corresponding to the second feature, and a third internal feature corresponding to the third feature; According to the first internal feature, the second internal feature and the third internal feature, feature fusion between multiple images is performed through the preset attention mechanism model to obtain a first associated feature corresponding to the first internal feature, a second associated feature corresponding to the second internal feature, and a third associated feature corresponding to the third internal feature; According to the first association feature, the second association feature and the third association feature, feature fusion and feature clustering between different domain features are performed through the preset attention mechanism model to obtain corresponding fusion features.

2. The method for identifying the feeding behavior of a school of fish as claimed in claim 1, characterized in that: The preset attention mechanism model includes a convolution transposition module, or an attention module, or a combination of a convolution transposition module and an attention module.

3. The method for identifying the feeding behavior of a school of fish as claimed in claim 2, characterized in that: The preset attention mechanism model includes a convolution transposition module and an attention module, and the step of fusing the internal features of a single image through the preset attention mechanism model according to the first feature, the second feature and the third feature to obtain a first internal feature corresponding to the first feature, a second internal feature corresponding to the second feature and a third internal feature corresponding to the third feature includes: Performing a convolution operation on the first feature, the second feature, and the third feature through the convolution transposition module, and performing feature matrix transposition multiplication on the result of the convolution and the first feature, the second feature, and the third feature to obtain a first convolution feature corresponding to the first feature, a second convolution feature corresponding to the second feature, and a third convolution feature corresponding to the third feature; Performing a preset number of convolution operations on the first convolution feature, the second convolution feature, and the third convolution feature through the attention module, and performing feature matrix multiplication on the convolution results to obtain a first product feature corresponding to the first convolution feature, a second product feature corresponding to the second convolution feature, and a third product feature corresponding to the third convolution feature; The first product feature, the first product feature and the first product feature are respectively summed with the first convolution feature, the second convolution feature and the third convolution feature to obtain the corresponding first internal feature, the second internal feature and the third internal feature.

4. The method for identifying the feeding behavior of a school of fish as claimed in claim 2, characterized in that: The preset attention mechanism model includes an attention module, and the step of performing feature fusion between multiple images through the preset attention mechanism model according to the first internal feature, the second internal feature and the third internal feature to obtain a first associated feature corresponding to the first internal feature, a second associated feature corresponding to the second internal feature and a third associated feature corresponding to the third internal feature includes: Performing feature integration processing on the first internal feature, the second internal feature, and the third internal feature respectively to obtain a first integrated feature corresponding to the first internal feature, a second integrated feature corresponding to the second internal feature, and a third integrated feature corresponding to the third internal feature; Performing a residual operation on the first integrated feature, the second integrated feature, and the third integrated feature to obtain a first sum feature corresponding to the first integrated feature, a second sum feature corresponding to the second integrated feature, and a third sum feature corresponding to the third integrated feature; Performing a pooling operation on the first sum feature, the second sum feature, and the third sum feature to obtain a corresponding pooled first sum feature, a pooled second sum feature, and a pooled third sum feature; The attention module performs a preset number of convolution operations on the first and second pooled features, and the third and second pooled features, and performs feature matrix multiplication on the result of the convolution and the first and second pooled features, and the third and second pooled features, respectively, and then sums them to obtain the corresponding first product feature, second product feature, and third product feature; Performing a pooling operation on the first product feature, the second product feature, and the third product feature to obtain a pooled first product feature, a pooled second product feature, and a pooled third product feature; The attention module performs a preset number of convolution operations on the first pooled product feature, the second pooled product feature, and the third pooled product feature, and performs feature matrix multiplication on the result of the convolution and the first pooled product feature, the second pooled product feature, and the third pooled product feature, respectively, and then sums them up to obtain the corresponding first intent feature, second intent feature, and third intent feature; The first intention feature, the second intention feature and the third intention feature are pooled to obtain corresponding first association features, second association features and third association features.

5. The method for identifying the feeding behavior of a school of fish as claimed in claim 2, characterized in that: The preset attention mechanism model includes a convolution transposition module and an attention module. The step of fusing features between different domain features through the preset attention mechanism model according to the first association feature, the second association feature and the third association feature to obtain corresponding fusion features includes: Channel-joining the first association feature, the second association feature, and the third association feature to obtain a corresponding three-dimensional matrix feature; The convolution operation is performed on the three-dimensional matrix feature by the convolution transposition module, and a feature matrix is ​​multiplied by the convolution result and the three-dimensional matrix feature to obtain a coding feature corresponding to the three-dimensional matrix feature; Performing a preset number of convolution operations on the coding feature through the attention module, and multiplying the result after the convolution by a feature matrix to obtain a product coding feature corresponding to the coding feature; Feature summation is performed on the product coding feature to obtain a fusion feature corresponding to the product coding feature.

6. A device for identifying the feeding behavior of a school of fish, characterized in that: The device for identifying the feeding behavior of a school of fish comprises: An acquisition module is used to acquire a time-series image set of the feeding behavior of a school of fish, and pre-process the time-series image set using a preset enhancement algorithm, a preset optical flow method, and a preset Gaussian density model to obtain corresponding spatial images, optical flow images, and density images; An extraction module, configured to extract, based on the spatial image, the optical flow image, the density image and a preset attention mechanism model, a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image, and a third feature corresponding to each density image; The extraction module is also used to perform convolution and pooling processing on each spatial image, each optical flow image and each density image according to the spatial image, the optical flow image and the density image through the preset attention mechanism model to obtain a processed spatial image, a processed optical flow image and a processed density image; perform residual summation processing on the processed spatial image, the processed optical flow image and the processed density image for a preset number of times to obtain a first feature corresponding to each spatial image, a second feature corresponding to each optical flow image and a third feature corresponding to each density image; A fusion module, configured to perform feature fusion based on the first feature, the second feature and the third feature through a preset attention mechanism model to determine a corresponding fusion feature; The fusion module is also used to perform internal feature fusion of a single image through the preset attention mechanism model according to the first feature, the second feature and the third feature, to obtain a first internal feature corresponding to the first feature, a second internal feature corresponding to the second feature and a third internal feature corresponding to the third feature; perform feature fusion between multiple images through the preset attention mechanism model according to the first internal feature, the second internal feature and the third internal feature, to obtain a first associated feature corresponding to the first internal feature, a second associated feature corresponding to the second internal feature and a third associated feature corresponding to the third internal feature; perform feature fusion and feature clustering between features in different domains through the preset attention mechanism model according to the first associated feature, the second associated feature and the third associated feature, to obtain corresponding fused features; The recognition module is used to recognize the feeding behavior of the fish school based on the fusion features.

7. A device, which is a device for identifying the feeding behavior of a school of fish, characterized in that: The device for identifying the feeding behavior of a school of fish comprises: a memory, a processor, and a program for identifying the feeding behavior of a school of fish stored in the memory and executable on the processor. When the program for identifying the feeding behavior of a school of fish is executed by the processor, the steps of the method for identifying the feeding behavior of a school of fish as described in any one of claims 1 to 5 are implemented.

8. A medium, the medium being a computer-readable storage medium, characterized in that: The computer-readable storage medium stores a fish school feeding behavior recognition program, and when the fish school feeding behavior recognition program is executed by a processor, the steps of the fish school feeding behavior recognition method as described in any one of claims 1 to 5 are implemented.

Images