A fish feeding behavior recognition method based on multi-architecture mixing

Abstract

This invention discloses a multi-architecture hybrid method for recognizing fish feeding behavior, belonging to the field of aquaculture image processing technology. The method includes: acquiring video data of fish feeding processes; preprocessing the video data to obtain a spatiotemporal input tensor; using a multi-architecture hybrid backbone network to extract spatiotemporal features from the spatiotemporal input tensor to obtain an augmented tensor; mapping the augmented tensor to a classifier, outputting the probability distribution of each feeding behavior category, and using the category index corresponding to the highest probability as the recognition result. This invention achieves a unified expression of local burst features and long-range temporal dependencies in fish feeding behavior, reducing computational load while improving the ability to eliminate channel redundancy and model semantic relevance; it also balances computational overhead, enabling high-efficiency and high-precision recognition of fish feeding behavior.

Description

Technical Field

[0001] This invention relates to the field of aquaculture image processing technology, and in particular to a method for recognizing fish feeding behavior based on multi-architecture hybridization. Background Technology

[0002] With the continuous expansion of aquaculture, feeding management faces problems such as reliance on experience, lack of quantitative feedback, and easy feed waste and water quality deterioration. Since the movement patterns and spatiotemporal variations of fish feeding behavior can reflect their hunger state and feeding motivation, identifying feeding behavior is key to achieving on-demand feeding and refined management. However, current production practices largely rely on manual observation, which is highly subjective and difficult to standardize. Furthermore, existing timed and quantitative feeding equipment generally lacks the ability to achieve closed-loop perception of fish feeding status, making it difficult to achieve "on-demand feeding" and resulting in low feed utilization.

[0003] In recent years, the rapid development of artificial intelligence has provided new ideas for the intelligent upgrading of aquaculture. Among them, computer vision technology, with its advantages of non-contact, low cost, and high scalability, has gradually become the mainstream method in the research of fish feeding behavior recognition. Early research mainly relied on manually extracting features such as motion, texture, or shape from video or image sequences and combining them with traditional classifiers for discrimination. However, due to the significant dynamics, complexity, and diversity of fish feeding behavior, hand-crafted features have obvious limitations in capturing the fine-grained spatiotemporal features of fish behavior, severely restricting the generalization ability and recognition accuracy of the model. With the rapid development of deep learning technology, automatic feature extraction methods based on end-to-end learning frameworks have gradually become a research focus. Among them, convolutional neural networks (CNNs) have been widely used in fish behavior recognition tasks due to their excellent spatial feature modeling capabilities. However, CNNs inherently lack a global receptive field and have limitations in processing long-term temporal data, especially when capturing long-term dependencies in behavioral evolution, which limits their application potential in complex behavior recognition tasks. To compensate for the shortcomings of CNNs in temporal modeling, the Transformer architecture has been introduced into the field of fish feeding behavior recognition. This architecture, based on a self-attention mechanism, effectively captures long-range dependencies in behavioral changes through global context modeling, demonstrating excellent performance in various video modeling tasks. However, the computational complexity of the Transformer increases quadratically with sequence length, resulting in significant computational and memory overhead, making it difficult to meet the real-time deployment requirements of edge devices. Furthermore, the Transformer has limited capabilities in modeling local spatial structures, is not sensitive enough to short-term, sudden feeding behaviors, and has limited accuracy in fine-grained behavior recognition. To achieve more efficient long-sequence modeling, recently proposed sequence modeling methods based on linear state-space models (SSMs), such as the Mamba class, have achieved good results in multiple video understanding and sequence processing tasks. Mamba possesses linear computational complexity, effectively models long-term dependencies, and significantly reduces computational burden, making it a highly efficient alternative to the Transformer. Although Mamba has significant advantages in temporal modeling, its spatial modeling capabilities are relatively weak, making it difficult to fully capture the dynamic changes of local features in fish feeding behavior.

[0004] Therefore, a single architecture struggles to balance local details, global dependencies, and computational efficiency. While multi-architecture hybrid strategies that integrate the local spatial modeling advantages of CNNs with the long-range temporal / global context modeling capabilities of SSM / attention have potential advantages, a systematic implementation solution for fish feeding behavior recognition scenarios that addresses real-time performance, lightweight design, and fine-grained representation of complex behaviors is still lacking.

[0005] Therefore, a multi-architecture hybrid method for fish feeding behavior recognition is needed to address the problems of insufficient global receptive field, secondary complexity, and inefficient spatial modeling in existing single-architecture methods such as CNN, Transformer, and Mamba. This would provide a high-efficiency, high-precision, lightweight, and scalable solution for fish feeding behavior recognition. Summary of the Invention

[0006] The purpose of this invention is to propose a method for recognizing fish feeding behavior based on a multi-architecture hybrid approach, comprising:

[0007] Acquire video data of the fish feeding process, and preprocess the video data to obtain the spatiotemporal input tensor;

[0008] An enhanced tensor is obtained by extracting spatiotemporal features from the spatiotemporal input tensor using a multi-architecture hybrid backbone network.

[0009] The augmentation tensor is mapped to a classifier, which outputs the probability distribution of each feeding behavior category. The category index corresponding to the highest probability is used as the recognition result.

[0010] Furthermore, video data of the fish feeding process is acquired, and the specific process of preprocessing the video data is as follows:

[0011] The raw video of the feeding process of fish in the aquaculture pond was obtained using filming equipment;

[0012] The region of interest in the original video is extracted using spatial domain cropping technology, and non-target areas outside the boundaries of the aquaculture pond are removed, while the core observation area of ​​fish activity is retained;

[0013] The region of interest in the video is segmented in a non-overlapping manner according to the number of window frames, so that each segment contains a sequence of several consecutive frames.

[0014] Perform data augmentation operations on the frame sequence;

[0015] Stack consecutive frames to form a spatiotemporal input tensor.

[0016] Furthermore, the window frame count ranges from 8 to 128 frames.

[0017] Furthermore, data augmentation operations include size scaling, pixel normalization, noise reduction, color correction, gamma correction, and histogram equalization.

[0018] Furthermore, the specific process of extracting spatiotemporal features from the spatiotemporal input tensor using a multi-architecture hybrid backbone network to obtain the enhanced tensor is as follows:

[0019] spatiotemporal input tensor Perform two layers of 3D convolutional downsampling to obtain the sampling tensor. ;

[0020] For sampling tensors High-resolution spatial feature extraction is performed to obtain the first extended tensor with doubled channel number. ;

[0021] The first extended tensor Then perform the same high-resolution spatial feature extraction operation to obtain the second extended tensor. ;

[0022] For the second extended tensor Window segmentation, spatial location encoding, and temporal location encoding are performed sequentially. Multiple lightweight hybrid Mamba blocks are then used to extract spatiotemporal features to obtain the enhanced tensor. .

[0023] Furthermore, the spatiotemporal feature extraction using multiple lightweight hybrid Mamba blocks specifically includes:

[0024] The input features are divided into three equal parts along the channel dimension and then processed in parallel by a long-range dependency modeling branch, a local dynamic branch, and a global context branch. The outputs of the three branches are then fused by feature concatenation and linear transformation to obtain the global-local information collaborative features after lightweight hybrid Mamba block processing.

[0025] Furthermore, the long-range dependency modeling branch employs selective scanning or equivalent linear complexity sequence operators based on state-space models; the local dynamics branch uses one-dimensional depthwise separable convolution to enhance local spatiotemporal neighborhood patterns; and the global context branch employs an attention mechanism to obtain contextual relationships across windows or regions.

[0026] The beneficial effects of this invention are as follows:

[0027] 1. This invention achieves a unified expression of local burst features and long-term temporal dependencies of fish feeding behavior by combining a three-dimensional convolutional structure, a spatiotemporal perception mechanism, and a hybrid feature interaction module into a multi-architecture hybrid backbone network, thus balancing accuracy and generalization in complex behavior recognition.

[0028] 2. This invention improves the ability to eliminate channel redundancy and model semantic relevance while reducing computational load through adaptive partial convolution based on channel attention and channel similarity redundancy, thereby enhancing the model's adaptability in complex scenarios.

[0029] 3. This invention achieves unified modeling of long-range dependency modeling, local spatial feature extraction and global context awareness through a lightweight hybrid Mamba module with a three-branch parallel structure, which improves the model's global representation ability of complex feeding behavior, while taking into account computational overhead, and can achieve high-efficiency and high-precision recognition of fish feeding behavior. Attached Figure Description

[0030] Figure 1 This is a flowchart of the fish feeding behavior recognition method based on multi-architecture hybridization according to the present invention;

[0031] Figure 2 For a multi-stage spatiotemporal coding backbone network;

[0032] Figure 3 It is an adaptive partial convolutional module based on channel attention and channel similarity redundancy;

[0033] Figure 4 It is a lightweight hybrid Mamba module with three parallel branches. Detailed Implementation

[0034] This invention proposes a method for recognizing fish feeding behavior based on a multi-architecture hybrid approach. The invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0035] Figure 1 This is a flowchart of the fish feeding behavior recognition method based on a multi-architecture hybrid architecture of the present invention. The main steps include acquiring video data of the fish feeding process, preprocessing the video data to obtain a spatiotemporal input tensor, using a multi-architecture hybrid backbone network to extract spatiotemporal features from the spatiotemporal input tensor to obtain an augmented tensor, mapping the augmented tensor to a classifier, outputting the probability distribution of each feeding behavior category, and using the category index corresponding to the highest probability as the recognition result. Specifically, it includes the following aspects:

[0036] In the data acquisition and preprocessing stage, a multi-stage preprocessing strategy was applied to the acquired video data of fish feeding to optimize data quality. The steps are as follows:

[0037] 1) Use video equipment to acquire video data of the fish feeding process. The video equipment is placed above, to the side above, or in other observable positions in the aquaculture pond / water body; preferably at a top-view or near-top-view angle to reduce obstruction and increase the observable area.

[0038] 2) Regions of Interest (ROI) extraction is performed on the original video to effectively remove non-target areas outside the boundaries of the aquaculture ponds, retain the core observation area of ​​fish activity, and reduce background interference. ROIs are obtained through fixed cropping and geometric rules based on the pond boundaries.

[0039] 3) Divide the ROI video into multiple segments according to the time window. The number of frames T in the window can range from 8 to 128 frames, preferably from 16 to 64 frames. In this embodiment, 32 frames are selected. Adjacent windows can be non-overlapping or partially overlapping, and the overlap ratio can be 0% to 75% to balance real-time performance and sample coverage.

[0040] 4) Perform data augmentation operations on the frame sequence within each segment, including size scaling, pixel normalization, noise reduction, color correction, gamma correction, and histogram equalization.

[0041] 5) Stack the processed consecutive frames to form a spatiotemporal input tensor. ,in, Represents the real number field; This is the batch size, which is the number of samples processed in parallel during a single forward computation. This represents the number of channels, corresponding to the channel dimension of the feature map; The number of consecutively input video frames; The height of the feature map; This represents the width of the feature map.

[0042] In the spatiotemporal feature extraction stage, the spatiotemporal input tensor is... The data is fed into a multi-stage spatiotemporal coding backbone network for feature extraction, such as... Figure 2 As shown; the backbone network is a multi-architecture hybrid structure, including: a spatial downsampling and local feature extraction module with 3D convolution as its core; and an adaptive partial convolution module based on channel attention and channel similarity redundancy, such as... Figure 3 As shown, a lightweight hybrid Mamba module with three parallel branches is used to reduce redundant computation and enhance semantically relevant channel representation. Figure 4 As shown, this is used to integrate long-range dependencies, local dynamics, and global context. Specifically:

[0043] Phase 1: Inputting Tensors into Spacetime A downsampling operation is performed to quickly reduce spatial resolution and computational complexity for initial feature extraction. The downsampling operation consists of two sequential sub-sampling modules. Each sub-sampling module comprises a 3D convolutional layer (Conv3D), a normalization layer, and an activation function. The sampling stride of the 3D convolutional layer is (1,2,2) to maintain temporal resolution, halve spatial resolution, and double the number of channels. After the downsampling operation, a sampled tensor is obtained. , can be represented as:

[0044] ;

[0045] in, , This is the preset number of extended channels; This represents the processing function of a three-dimensional convolutional layer; It is the ReLU activation function; This is the BatchNorm normalization function.

[0046] Phase Two: Processing the Sampling Tensor Perform high-resolution spatial feature extraction to achieve rapid capture of low-level spatial features. The high-resolution spatial feature extraction operation is performed by... It consists of a series of convolutional modules. After processing, a convolutional downsampling with a stride of (1,2,2) is performed to obtain the first expanded tensor. , can be represented as:

[0047] ;

[0048] ;

[0049] in, ; This represents the processing function of the first convolutional module; This represents the processing function for the second convolutional module; Indicates the first The processing functions for each convolutional module. Each convolutional module contains two sequential adaptive partial convolutional modules, along with additional normalization layers, activation functions, regularization layers, and residual connections. For example, the first convolutional module can be represented as:

[0050] ;

[0051] ;

[0052] in, Use the GELU activation function; This represents a random depth regularization operator used to improve the generalization ability of a network; This represents the processing function of the adaptive partial convolution module. Each adaptive partial convolution module consists of an adaptive partial convolution (APConv), a channel-wise convolution (DConv), and a pointwise convolution (PConv), along with a normalization layer and an activation function. The specific representation is as follows:

[0053] ;

[0054] in, D represents the processing function for pointwise convolution; This represents the function for processing channel-wise convolution; This represents the adaptive partial convolution processing function. Its implementation process is as follows: First, the channel attention module generates the weight coefficients for each channel. Then, combined with the channel similarity matrix, it determines the optimal subset of channels to participate in the calculation, thereby achieving differentiated and selective information modeling. The specific implementation is as follows:

[0055] 1) Channel attention: For example, for sampling tensors Global average pooling is performed on each channel in both time and space dimensions to extract a global statistical description for each channel:

[0056]

[0057] in, These represent the indices in the channel dimension, time dimension, height dimension, and width dimension, respectively. h , c ;Sampling tensor Number of channels; The number of consecutively input video frames; The height of the feature map; This represents the width of the feature map.

[0058] Then, a multilayer perceptron (MLP) is used to calculate the channel attention weights. This is used to measure the global importance of different channels.

[0059] ;

[0060] in, , These are learnable parameters; Use the Sigmoid activation function; This is the compression ratio hyperparameter.

[0061] 2) Channel similarity redundancy: The sampling tensor... Each channel is expanded into a vector in the time and space dimensions, defining the channel. and channels The cosine similarity matrix between them is:

[0062]

[0063] ;

[0064] ;

[0065] in, This represents a tensor rearrangement operator used to flatten the time and spatial dimensions of an input tensor into a one-dimensional vector; Indicates the first The spatiotemporal feature vectors corresponding to each channel; Indicates the first The spatiotemporal feature vectors corresponding to each channel are calculated. Other channels average similarity This reflects the redundancy of the channel:

[0066]

[0067] 3) Fusion Scoring and Selection: Attention Weights for Fusion Channels Similarity to channels The importance score for each channel is obtained:

[0068]

[0069] in, This is a weighting adjustment factor; Indicates the degree of non-redundancy of the channel.

[0070] 4) Selective Convolution: Based on Channel Scoring Select the top-rated Each channel constitutes an index set. And extract the corresponding channel sub-tensor from the input feature tensor. Finally, a 3D convolution operation is performed on it to obtain the output of the adaptive partial convolution:

[0071]

[0072]

[0073]

[0074] Phase 3: On the first extended tensor Perform the same high-resolution spatial feature extraction operation as the second-stage structure, and further refine the spatial feature representation while maintaining the temporal resolution to obtain the second extended tensor. The aim is to enhance the spatial discriminative power of high-level semantic features step by step within a multi-scale feature extraction framework, providing a more compact and discriminative feature representation for subsequent temporal modeling and decision-making modules.

[0075] Phase 4: On the second extended tensor Low-resolution temporal feature extraction is performed to achieve accurate modeling and semantic feature aggregation of cross-regional temporal relationships. The low-resolution temporal feature extraction operations include window segmentation, spatial location encoding, temporal location encoding, and... A lightweight hybrid Mamba time-series feature extraction module is implemented as follows:

[0076] 1) Window splitting: Perform dimensional transformation Spatial dimensions are adjusted according to window size. Perform non-overlapping partitioning, each window contains Given a spatial location, the feature representation after window division is as follows:

[0077] ;

[0078] in, This means that the input feature tensor is divided into spatial dimensions according to the window size. After partitioning, the features within each window are flattened to a length of [length missing]. The window-level feature tensor obtained from the sequence. This represents a window partitioning function used to divide the input feature map into blocks of a preset window size in the spatial dimension, while maintaining the relative spatial order of features within the window.

[0079] 2) Spatial location encoding operation: Introducing classification label vectors for spatial location. It is then concatenated to the beginning of the window sequence, and then spatial location encoding is added. The formula is:

[0080] ;

[0081] ;

[0082] in, This represents the window-level feature tensor after spatial location encoding; The concatenation operation represents the concatenation of the spatial classification label into the extended sequence feature tensor formed by concatenating the spatial classification label into the window-level feature sequence; This indicates an operation that concatenates along the sequence dimension while keeping the feature dimension unchanged.

[0083] 3) Timing position encoding operation: ... Rearrange along the dimension to obtain Add timing position coding ,get:

[0084] ;

[0085] in, This represents the feature tensor after fusing temporal location information. Then... Rearranged as follows: And concatenate the time-series classification label vector at the beginning and end of the corresponding sequence in the time-series dimension. ,get:

[0086] ;

[0087] in: This represents a tensor after temporal position encoding.

[0088] 4) To conduct Each lightweight hybrid Mamba module is processed. Each module consists of three parallelized branches: Branch 1: Long-range dependency modeling branch, employing selective scanning based on a state-space model or an equivalent linear complexity sequence operator; Branch 2: Local dynamics branch, using depthwise separable convolution to enhance local spatiotemporal neighborhood patterns; Branch 3: Global context branch, employing an attention mechanism to obtain cross-window / cross-region contextual relationships. The outputs of the three branches are processed through feature concatenation and fusion operations followed by a linear transformation to obtain global-local information collaborative features. For the input... ,through The enhanced tensor is obtained after layer lightweight hybrid Mamba processing. , can be represented as:

[0089] ;

[0090] in, This represents the processing function for the first lightweight hybrid Mamba module; This represents the processing function for the second lightweight hybrid Mamba module; Indicates the first The processing functions for each lightweight hybrid Mamba module. Taking the first lightweight hybrid Mamba module as an example, it can be represented as:

[0091]

[0092]

[0093]

[0094]

[0095]

[0096] in, It is a linear mapping operator used to reduce the channel dimension of input features from... Mapped to . This represents a one-dimensional convolution operator used for local feature extraction in the sequence or time dimension; This represents the feature branch obtained after performing long-range dependency modeling on the input features using the selective scanning operator, which is used to capture global dynamic relationships across time or across sequences; This represents the feature branch obtained by modeling the local spatiotemporal neighborhood through one-dimensional convolution, which is used to capture short-range dynamic change patterns; This represents the feature branch obtained by aggregating global context information through an attention mechanism, used to model the global correlation between different time steps or spatial locations; This represents a selective scanning operator used to recursively or in parallel update sequence features based on a state-space model, modeling long-range dependencies with linear complexity. This represents an attention computation operator based on query, key, and value, used to perform weighted aggregation of global information according to the similarity relationship between features; It refers to the number of heads to focus on. It is the SiLU activation function.

[0097] In the result generation phase, from the augmentation tensor Extract the feature vector corresponding to the classification label. The probability distribution of each feeding behavior category is output through a classifier mapping. The category index corresponding to the highest probability is used as the final recognition result. , can be represented as:

[0098] ;

[0099] ;

[0100]

[0101] in, This indicates the index corresponding to the element with the largest value in the input.

[0102] The method of this invention was validated on a video dataset containing four behavioral categories: "normal cruising," "food searching," "weak feeding," and "strong feeding." The model achieved accuracy of 93.94%, precision of 93.16%, recall of 93.50%, and F1-score of 93.27%. The model exhibits low parameter count and computational complexity (FLOPs), at 9.71M and 94.97G respectively, demonstrating a good efficiency-performance balance.

[0103] In summary, the technical solution of this invention achieves a unified expression of local burst features and long-term temporal dependencies in fish feeding behavior, reducing computational load while improving the ability to eliminate channel redundancy and model semantic relevance; at the same time, it can take into account computational overhead, enabling high-efficiency and high-precision recognition of fish feeding behavior.

Claims

1. A method for recognizing fish feeding behavior based on multi-architecture hybrid approach, characterized in that, include: Acquire video data of the fish feeding process, and preprocess the video data to obtain the spatiotemporal input tensor; An enhanced tensor is obtained by extracting spatiotemporal features from the spatiotemporal input tensor using a multi-architecture hybrid backbone network. The augmentation tensor is mapped to a classifier, which outputs the probability distribution of each feeding behavior category. The category index corresponding to the highest probability is used as the recognition result.

2. The method for recognizing fish feeding behavior based on multi-architecture hybridization according to claim 1, characterized in that, The specific process of acquiring video data of fish feeding behavior and preprocessing the video data is as follows: The raw video of the feeding process of fish in the aquaculture pond was obtained using filming equipment; The region of interest in the original video is extracted using spatial domain cropping technology, and non-target areas outside the boundaries of the aquaculture pond are removed, while the core observation area of ​​fish activity is retained; The region of interest in the video is segmented in a non-overlapping manner according to the number of window frames, so that each segment contains a sequence of several consecutive frames. Perform data augmentation on the frame sequence; Stack consecutive frames to form a spatiotemporal input tensor.

3. The method for recognizing fish feeding behavior based on multi-architecture hybridization according to claim 2, characterized in that, The number of window frames ranges from 8 to 128 frames.

4. The method for fish feeding behavior recognition based on multi-architecture hybridization according to claim 2 or 3, characterized in that, The data augmentation operations include size scaling, pixel normalization, noise reduction, color correction, gamma correction, and histogram equalization.

5. The method for recognizing fish feeding behavior based on multi-architecture hybridization according to claim 1, characterized in that, The specific process of extracting spatiotemporal features from spatiotemporal input tensors using a multi-architecture hybrid backbone network to obtain enhanced tensors is as follows: spatiotemporal input tensor Perform two layers of 3D convolutional downsampling to obtain the sampling tensor. ; For sampling tensors High-resolution spatial feature extraction is performed to obtain the first extended tensor with doubled channel number. ; The first extended tensor Then perform the same high-resolution spatial feature extraction operation to obtain the second extended tensor. ; For the second extended tensor Window segmentation, spatial location encoding, and temporal location encoding are performed sequentially. Multiple lightweight hybrid Mamba blocks are then used to extract spatiotemporal features to obtain the enhanced tensor. .

6. The method for recognizing fish feeding behavior based on multi-architecture hybridization according to claim 5, characterized in that, Spatiotemporal feature extraction using multiple lightweight hybrid Mamba blocks specifically includes: The input features are divided into three equal parts along the channel dimension and then processed in parallel by a long-range dependency modeling branch, a local dynamic branch, and a global context branch. The outputs of the three branches are then fused by feature concatenation and then subjected to a linear transformation to obtain the global-local information collaborative features after lightweight hybrid Mamba block processing.

7. The method for recognizing fish feeding behavior based on multi-architecture hybridization according to claim 6, characterized in that, The long-range dependency modeling branch adopts selective scanning or equivalent linear complexity sequence operators based on state-space models; The local dynamic branch employs one-dimensional depthwise separable convolution to enhance the local spatiotemporal neighborhood pattern. The global context branch uses an attention mechanism to obtain contextual relationships across windows or regions.

Images