Fish school feeding behavior identification method and device

Abstract

The invention provides a fish shoal feeding behavior identification method and device, and relates to the technical field of artificial intelligence, and the method comprises the steps: obtaining synchronously collected fish shoal feeding video data and water surface fluctuation signal data, and a pre-trained fish shoal feeding behavior identification model; wherein the water surface fluctuation signal data is a multi-axis motion sensing signal acquired by a multi-axis sensor; and inputting the fish school ingestion video data and the water surface fluctuation signal data into the pre-trained fish school ingestion behavior recognition model to obtain a fish school ingestion behavior recognition result. Through multi-modal feature fusion, behavior features in a video and physical features in a fluctuation signal are comprehensively analyzed, so that the ingestion intensity is more accurately judged. A multi-task learning strategy is used in the training process of the model, the synergistic effect of visual features and fluctuation signal features is optimized, subtle differences of ingestion intensity can be captured, and misjudgment caused by insufficient single-mode data is avoided.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and device for identifying the feeding behavior of a school of fish. Background Art

[0002] Proper feeding is a crucial component of aquaculture, not only impacting the healthy growth of fish but also determining production efficiency and costs. Underfeeding impairs normal fish growth, while overfeeding leads to feed waste and water pollution, increasing the risk of disease. Currently, most fish feeding methods rely on manual experience and timed, quantitative, and timed control. Precise feeding tailored to fish appetite has become a pressing challenge. Research has shown that behavioral information, such as the intensity and amplitude of changes in fish feeding activity, can directly reflect their appetite. Therefore, real-time monitoring and identification of fish feeding behavior, and the development of optimal feeding strategies based on this information, are key to reducing production costs. In recent years, machine vision has been widely used in aquaculture, providing an effective means for real-time, automated, and non-contact research.

[0003] Vision-based methods for identifying fish feeding behavior are suitable for most aquaculture environments, but they fail in complex environments such as strong light reflections, turbid water, or at night. Furthermore, they can be affected by factors such as water flow, fish movement, and external interference, leading to misjudgment of feeding intensity, such as misidentifying vigorous swimming as intense feeding. Summary of the Invention

[0004] The present invention provides a method and device for identifying the feeding behavior of a school of fish, which is used to solve the problems of identification failure caused by environmental interference and misjudgment of feeding intensity in the prior art. The technical solutions proposed by the present invention are as follows: In a first aspect, the present invention provides a method for identifying feeding behavior of a school of fish, comprising: Acquire synchronously collected fish feeding video data and water surface fluctuation signal data, and a pre-trained fish feeding behavior recognition model; wherein the water surface fluctuation signal data is a multi-axis motion sensing signal collected by a multi-axis sensor; Inputting the fish feeding video data and the water surface fluctuation signal data into the pre-trained fish feeding behavior recognition model to obtain a fish feeding behavior recognition result; The fish feeding behavior recognition model is trained in the following way: Obtain a multimodal dataset of fish feeding, including synchronously collected feeding video data, motion sensor data, and true labels, where the true labels are the feeding intensity classification levels; Dividing the fish feeding multimodal dataset into a training set and a validation set according to a preset ratio; A pre-built multimodal fusion neural network model is trained based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model.

[0005] Optionally, the multimodal fusion neural network model includes a visual feature extraction network, a motion feature extraction network, a cross-modal feature fusion network and a classification layer; The pre-built multimodal fusion neural network model is trained based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model, including: Based on the visual feature extraction network, the spatiotemporal features of the feeding video data in the training set are extracted to generate original visual features; Extracting original motion features of motion sensor data in the training set based on motion feature extraction network; Calculate cross-modal consistency loss based on original visual features and original motion features; Based on the cross-modal feature fusion network, the original visual features and the original motion features are cross-modally fused to obtain the globally aligned fusion features; Inputting the globally aligned fusion features into a classification layer to obtain a prediction result, and calculating a fusion classification loss based on the prediction result and the true label; Calculating a total loss based on the cross-modal consistency loss and the fusion classification loss; Back propagation is performed according to the total loss to update the model parameters of the multimodal fusion neural network model to obtain an optimized model; Use the validation set to evaluate the performance of the optimized model until the model converges or reaches the preset training rounds.

[0006] Optionally, based on the original visual features and the original motion features, a cross-modal consistency loss is calculated, including: Calculating a mean square error loss for feature distribution alignment using the original visual features and the original motion features; Calculate the cosine similarity loss of feature direction consistency using the original visual features and the original motion features; A weighted sum is performed on the mean square error loss and the cosine similarity loss to obtain the cross-modal consistency loss.

[0007] Optionally, the visual feature extraction network adopts a three-dimensional sliding window transformer architecture, and the visual feature extraction network includes a three-dimensional block partitioning module, a linear embedding layer, a multi-layer three-dimensional sliding window transformer block and a downsampling module; The method of extracting spatiotemporal features of the feeding video data in the training set based on the visual feature extraction network to generate original visual features includes: The feeding video data is divided into video segments according to a fixed time window, and the motion sensor data is synchronously divided into time series signal segments of the same time window; Performing size normalization processing on each frame image in the video clip to obtain preprocessed video data; Dividing the pre-processed video data into three-dimensional blocks using a three-dimensional block division module; The pixel values ​​of each 3D block are mapped to a high-dimensional feature representation through a linear embedding layer, and the output visual feature tensor is divided into non-overlapping 3D windows to obtain multiple window feature blocks; The multi-layer three-dimensional sliding window transformer block is used to perform attention calculation on the window feature block, and the feature representation output by the three-dimensional sliding window transformer block of the current layer is subjected to neighborhood merging and downsampling processing through the downsampling module, and then input into the three-dimensional sliding window transformer block of the next layer; the feature representation output by the three-dimensional sliding window transformer block of each layer is subjected to global average pooling and linear projection processing to obtain the original visual feature.

[0008] Optionally, each layer of the 3D sliding window transformer block includes a 3D window multi-head attention and a 3D sliding window multi-head self-attention. In each layer of the 3D sliding window transformer block, the input feature block is processed in the following manner: Through the three-dimensional window multi-head self-attention, the local spatiotemporal attention calculation is performed on the input feature block, and the feature representation with local spatiotemporal correlation is output; Through three-dimensional sliding window multi-head self-attention, cross-window feature interaction is performed on the feature representation with local spatiotemporal correlation, and a feature representation with cross-window perception capability is output.

[0009] Optionally, the cross-modal feature fusion network includes n cross-modal attention fusion modules, m Transformer layers and a cross transformer layer, where nm=1; Based on the cross-modal feature fusion network, the original visual features and original motion features are cross-modally fused to obtain globally aligned fusion features, including: The first cross-modal attention fusion module takes the original visual features and the original motion features as input and generates aggregated features through a cross-modal attention mechanism; The subsequent cross-modal attention fusion module takes the original visual features and the enhanced motion features output by the previous Transformer layer as input to update the aggregated features; The fused features output by the last cross-modal attention fusion module are input into the cross transformer layer for cross-modal global alignment to obtain the globally aligned fused features.

[0010] In a second aspect, the present invention further provides a device for identifying feeding behavior of a school of fish, comprising the following modules: a data acquisition module for acquiring synchronously collected fish feeding video data and water surface fluctuation signal data, and a pre-trained fish feeding behavior recognition model; wherein the water surface fluctuation signal data is a multi-axis motion sensing signal collected by a multi-axis sensor; wherein the fish feeding behavior recognition model is trained in the following manner: acquiring a fish feeding multimodal dataset, including synchronously collected feeding video data, motion sensing data, and true labels, where the true labels are feeding intensity classification levels; dividing the fish feeding multimodal dataset into a training set and a validation set according to a preset ratio; and training a pre-constructed multimodal fusion neural network model based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model; The behavior recognition module is used to input the fish feeding video data and the water surface fluctuation signal data into the pre-trained fish feeding behavior recognition model to obtain a fish feeding behavior recognition result.

[0011] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the method for identifying fish feeding behavior as described in the first aspect above is implemented.

[0012] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for identifying fish feeding behavior as described in the first aspect above.

[0013] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for identifying fish feeding behavior as described in the first aspect above.

[0014] Based on the above technical solution, the present invention has the following beneficial effects compared with the prior art: The method and device for identifying fish feeding behavior provided by this invention systematically address the existing issues of identification failure and feeding intensity misjudgment caused by environmental interference through multimodal data fusion and synchronous data acquisition. To address environmental interference, this method leverages the complementarity of video data and water surface fluctuation signal data, reducing reliance on a single modality through multimodal fusion. For example, when light interference is severe, the model can rely more on the fluctuation signal data, which is less affected by light. When water flow disturbances are significant, the model can use behavioral patterns in the video data to aid in identification. Furthermore, synchronous data acquisition ensures strict temporal alignment of behavioral and physical characteristics, avoiding misjudgments caused by temporal misalignment. Fish feeding video data provides behavioral characteristics of the school, while water surface fluctuation signal data reflects the physical characteristics of the school feeding. The combination of these two provides a more comprehensive description of the school's feeding behavior. Through multimodal feature fusion, this method comprehensively analyzes both behavioral characteristics in the video and physical characteristics in the fluctuation signal, resulting in a more accurate assessment of feeding intensity. During training, the model uses a multi-task learning strategy to optimize the synergy between visual and fluctuating signal features, enabling it to capture subtle differences in feeding intensity and avoid misjudgments caused by insufficient single-modal data. By fusing multimodal features, it can capture more subtle behavioral characteristics, thereby improving the accuracy of feeding behavior recognition.

[0015] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 3 is a flow chart of a method for identifying fish feeding behavior provided in an embodiment of the present invention.

[0019] Figure 2 Schematic diagram of the structure of the fish feeding behavior recognition system provided in an embodiment of the present invention.

[0020] Figure 3is a label classification diagram of different feeding behavior intensities in an embodiment of the present invention.

[0021] Figure 4 Schematic diagram of the structure of the fish feeding behavior recognition model in an embodiment of the present invention.

[0022] Figure 5 1 is a flow chart of visual feature extraction and a schematic diagram of the structure of a three-dimensional sliding window converter block in an embodiment of the present invention.

[0023] Figure 6 Schematic diagram of the structure of the cross-modal feature fusion network in an embodiment of the present invention.

[0024] Figure 7a and Figure 7b 1 is a diagram showing the recognition results of two different long signals by the fish feeding behavior recognition model in an embodiment of the present invention.

[0025] Figure 8 Schematic diagram of the structure of the fish feeding behavior recognition device provided in an embodiment of the present invention.

[0026] Figure 9 It is a structural schematic diagram of the electronic device provided by the present invention.

[0027] Reference numerals: 1. Multi-axis sensor; 2. Foam plastic; 3. Stainless steel screen tube; 4. Breeding pond; 5. Light source; 6. Image acquisition device; 7. Calculation processor. DETAILED DESCRIPTION

[0028] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0029] This paper proposes a method for identifying fish feeding behavior, specifically a model for identifying fish feeding behavior. This model fuses features from video data of fish schools with water surface fluctuation signals collected by sensors. Based on the fused features, the model identifies the feeding behavior of fish schools, enabling accurate feeding and delivery. The proposed model effectively processes the fused features and achieves an accuracy of 94.29% for identifying four different feeding intensities of fish, a 3.56% improvement over the baseline model. This model could provide technical support for the subsequent development of intelligent bait-casting machines.

[0030] Reference Figure 1 As shown, the fish feeding behavior recognition method includes the following: S110, obtaining synchronously collected fish feeding video data and water surface fluctuation signal data, and a pre-trained fish feeding behavior recognition model; wherein the water surface fluctuation signal data is a multi-axis motion sensing signal collected by a multi-axis sensor.

[0031] Reference Figure 2 The figure shows the fish feeding behavior recognition system provided by the present invention, which consists of a multi-axis sensor 1, foam plastic 2, a stainless steel screen 3, a culture tank 4, a light source 5, an image acquisition device 6, and a processor 7. Image acquisition device 6 (such as an industrial-grade IP camera) is installed 3 meters above the culture tank 4 in the recirculating aquaculture system to ensure coverage of the entire fish activity area. Image acquisition device 6 has a high frame rate (≥30 fps) and low-light adaptability to capture subtle feeding movements (such as rapid tail wagging and pecking). Light source 5 is located directly above the culture tank 4.

[0032] A multi-axis sensor 1 (such as a six-axis inertial sensor) is encapsulated in a rectangular foam plastic 2, which floats on the surface of the aquaculture pond 4 to isolate the sensor from direct interference from water currents. A counterweight (such as a lead block) is installed inside the foam plastic 2, adjusting its center of gravity to ensure rapid stabilization when the fish stop feeding and reduce inertial drift. An isolation zone is established in the center of the aquaculture pond, and a stainless steel screen 3 (pore size 1-2 cm) is used to form a circular fence to limit the movement of the foam plastic 2 and prevent it from drifting with water currents or disturbances caused by feeding. The sensor encapsulation and isolation zone design reduces direct interference from water currents and fish, and the filtering algorithm further improves signal quality, ensuring that the fluctuation signal more accurately reflects feeding behavior.

[0033] The multi-axis sensor 1 is connected to the processor 7 via a Bluetooth module, transmitting multi-axis motion sensing signals in real time. Both the image acquisition device 6 and the multi-axis sensor 1 are connected to the processor 7. A timestamp generator is used to unify the time base, ensuring strict alignment of the timestamps of the video frames and sensor data. In the data acquisition program, a timestamp is added to each video frame and each sensor data packet, and the timestamps are used to align the data of the two modalities in subsequent processing.

[0034] Image acquisition device 6 captures real-time dynamic images of the fish feeding process, generating feeding video data and capturing behavioral characteristics of the fish (such as swimming trajectories, aggregation levels, feeding movements, etc.). Multi-axis sensor 1 is reset to zero before signal acquisition, and the signal acquisition frequency is set to 50 Hz. Multi-axis sensor 1 collects multi-axis motion sensing signals from the water surface, such as triaxial acceleration (reflecting vertical and horizontal impact forces on the water surface) and triaxial angular velocity (reflecting micro-vibrations of the water surface), generating water surface fluctuation signal data. The water surface fluctuation signal data reflects micro-fluctuations caused by the feeding behavior of the fish (such as eddies generated by the fish swimming and the instantaneous impact forces generated during feeding). Its amplitude and frequency are positively correlated with the feeding intensity. Ensure strict temporal synchronization between the fish feeding video data and the water surface fluctuation signal data so that subsequent models can accurately correlate information from the two modalities. The video and sensor signals are segmented at the same time step (e.g., 10 frames per second of video + 10 sets of sensor data) to generate time-aligned data segments. For example, a 1-minute video is segmented into 600 segments (each containing 10 frames of video and 10 sets of sensor data) and saved as timestamp-feature pairs. Invalid areas (such as the edge of the fish tank) are cropped, and the resolution is adjusted to a uniform size (e.g., 256×256 pixels). The video data is captured at 2.56-second time steps and saved in MP4 format.

[0035] The angle data of each axis in the collected multi-axis motion sensor signal is optimized. The absolute value of the amplitude change between two adjacent sampling points of the angle data of each axis is taken to replace the angle data of each axis, so that it has the same change trend as the acceleration angular velocity data of each axis. The multi-axis motion sensor signal is recorded at a rate of 50Hz and then sampled in a window of 2.56 seconds. The processed data is saved in TXT format.

[0036] Video data provides behavioral patterns of fish schools (such as swimming trajectories and aggregation levels), but is easily affected by light. Fluctuation signal data reflects the physical impact of feeding and is less affected by light. Combining the two provides a comprehensive description of feeding behavior. Strict temporal alignment ensures that the model can correlate behavioral and physical characteristics at the same point in time. For example, the synchronization of a school's feeding movements (video) and the instantaneous peak impact force (sensor) provides strong evidence of feeding intensity.

[0037] Through the above process, we achieved efficient collection, strict synchronization, and standardized preprocessing of fish feeding video data and water surface fluctuation signal data. This solution provides a reliable data foundation for the multimodal fusion neural network model, enabling it to comprehensively utilize behavioral and physical characteristics to significantly improve the robustness and accuracy of feeding behavior recognition.

[0038] S120 , inputting the fish feeding video data and the water surface fluctuation signal data into the pre-trained fish feeding behavior recognition model to obtain a fish feeding behavior recognition result.

[0039] Load a pre-trained fish feeding behavior recognition model. This model, trained using a multimodal fusion neural network, can simultaneously process video data and water surface fluctuation signal data. Input the simultaneously collected fish feeding video data and water surface fluctuation signal data into the pre-trained fish feeding behavior recognition model. Using a multimodal fusion mechanism, the model comprehensively analyzes the behavioral characteristics in the video and the physical characteristics in the fluctuation signal, outputting the fish feeding behavior recognition results (e.g., feeding intensity level).

[0040] The above-mentioned fish feeding behavior recognition model is trained in the following way: A multimodal fish feeding dataset is obtained, comprising simultaneously collected feeding video data, motion sensor data, and true labels, where the true labels are feeding intensity classification levels. The multimodal fish feeding dataset is divided into a training set and a validation set according to a preset ratio. A pre-built multimodal fusion neural network model is trained based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model.

[0041] Compare the real-time feeding behavior of fish and video data to produce real labels, such as Figure 3 As shown in the figure, the four-category classification criteria of "Strong", "Medium", "Weak" and "None" based on the experience of feeding behavior recognition correspond to strong, medium, weak and none feeding intensity of fish schools respectively.

[0042] Specifically, a large multimodal dataset of fish feeding patterns was collected, including simultaneously acquired feeding video data, motion sensor data (i.e., water surface fluctuation signal data), and corresponding true labels (feeding intensity classification levels). The dataset was divided into training and validation sets according to a preset ratio (e.g., 80% training set, 20% validation set) for model training and performance evaluation. A multimodal fusion neural network model was constructed and trained using the training set. The model parameters were optimized using a backpropagation algorithm to minimize the loss between the predicted results and the true labels. The validation set was used to evaluate model performance, and the model structure or hyperparameters were adjusted to prevent overfitting. After training, a pretrained fish feeding behavior recognition model was obtained, which can be used to identify fish feeding behavior in real time.

[0043] This paper uses the PyTorch deep learning framework and Python to build a fish feeding behavior recognition model. The model is trained on an NVIDIA GTX 3090Ti GPU. The model batch size is set to 16, the number of iterations is set to 200, and the learning rate is set to 1e-5. The network parameters are optimized using the Adam optimizer with a regularization dropout of 0.3. The training environment is CUDA 12.1 and CUDNN 8.8.1.

[0044] The method for identifying fish feeding behavior provided by this invention systematically addresses the existing problems of identification failure and feeding intensity misjudgment caused by environmental interference through multimodal data fusion and synchronous acquisition. To address environmental interference, this method leverages the complementarity of video data and water surface fluctuation signal data, reducing reliance on a single modality through multimodal fusion. For example, when light interference is severe, the model can rely more on the fluctuation signal data, which is less affected by light. Conversely, when water flow disturbances are significant, the model can use behavioral patterns in the video data to aid in identification. Furthermore, synchronous data acquisition ensures strict temporal alignment of behavioral and physical characteristics, avoiding misjudgments caused by temporal misalignment. Fish feeding video data provides behavioral characteristics of the school, while water surface fluctuation signal data reflects the physical characteristics of the school feeding. The combination of these two provides a more comprehensive description of the school's feeding behavior. Through multimodal feature fusion, this method comprehensively analyzes both behavioral characteristics in the video and physical characteristics in the fluctuation signal, resulting in a more accurate assessment of feeding intensity. During training, the model uses a multi-task learning strategy to optimize the synergy between visual and fluctuating signal features, enabling it to capture subtle differences in feeding intensity and avoid misjudgments caused by insufficient single-modal data. By fusing multimodal features, it can capture more subtle behavioral characteristics, thereby improving the accuracy of feeding behavior recognition.

[0045] In some embodiments, two loss functions are used to learn the fish feeding intensity recognition task. The multimodal fusion neural network model includes a visual feature extraction network, a motion feature extraction network, a cross-modal feature fusion network and a classification layer; The pre-built multimodal fusion neural network model is trained based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model, including: S210 , extracting spatiotemporal features of the feeding video data in the training set based on the visual feature extraction network to generate original visual features; and extracting original motion features of the motion sensor data in the training set based on the motion feature extraction network.

[0046] The visual feature extraction network, based on a convolutional neural network (CNN), 3D-CNN, or 3D sliding window transformer architecture, extracts spatiotemporal features from feeding video data. These spatiotemporal features combine spatial and temporal features. Spatial features capture static information such as the fish school's morphology (e.g., outline, density), and position distribution. Temporal features capture dynamic information such as the school's movement trajectory and feeding frequency. The visual feature extraction network outputs raw visual features (e.g., high-dimensional feature vectors) that reflect the school's behavioral patterns.

[0047] The motion feature extraction network, based on a temporal convolutional network (TCN) or Transformer architecture, extracts multi-scale temporal features from multi-axis motion sensor signals. Using convolution kernels or attention windows of varying sizes, it captures variations in the fluctuation signal at different time scales (such as high-frequency impact forces and low-frequency periodic fluctuations), reflecting the physical impact of feeding fish on the water surface. The motion feature extraction network outputs raw motion features (such as high-dimensional feature vectors) that reflect the physical characteristics of the fish's feeding behavior.

[0048] S220. Calculate the cross-modal consistency loss (ConsistencyLoss) based on the original visual features and the original motion features.

[0049] The purpose of calculating cross-modal consistency loss is to align visual features and motion features in the semantic space and reduce the semantic gap between modalities. The original visual features can be calculated through contrastive learning (such as contrastive loss function), mean square error loss function or similarity measurement (such as cosine similarity). and original motion features (i.e. low-scale features ), refer to Figure 4 If the two patterns correspond to the same feeding behavior (e.g., high-intensity feeding), the feature distance is shortened; if they correspond to different behaviors, the feature distance is increased. This enhances consistency between modalities and avoids misjudgments due to modal differences.

[0050] S230. Based on the cross-modal feature fusion network, the original visual features and the original motion features are cross-modally fused to obtain a globally aligned fusion feature.

[0051] The cross-modal feature fusion network deeply fuses visual and motion features to generate globally aligned fused features. For example, cross-modal attention is used to dynamically adjust the weights of the two modal features. For example, during periods of high feeding intensity, more attention is paid to motion features (fluctuation signals); during complex behavioral patterns, more attention is paid to visual features. The feature vectors of the two modalities are concatenated or weighted summed to generate a fused feature. The output is a globally aligned fused feature (e.g., a higher-dimensional feature vector) that comprehensively reflects the behavioral and physical characteristics of the fish school.

[0052] S240: Input the globally aligned fusion features into a classification layer to obtain a prediction result, and calculate a fusion classification loss (Cross Entroy Loss) based on the prediction result and the true label.

[0053] The learning of the fused features is supervised by fusion classification loss, enabling accurate prediction of feeding intensity and optimizing the fused features' ability to express feeding intensity. The globally aligned fused features are mapped to feeding intensity levels (e.g., low, medium, high) through a classification layer. A classification loss function (e.g., cross-entropy loss) is used to calculate the difference between the predicted results and the true labels.

[0054] S250: Calculate a total loss based on the cross-modal consistency loss and the fusion classification loss, perform backpropagation based on the total loss, and update model parameters of the multimodal fusion neural network model to obtain an optimized model.

[0055] The total loss is obtained by directly adding or weighted summing the cross-modal consistency loss and the fusion classification loss. The model parameters (such as the visual feature extraction network, motion feature extraction network, and attention module weights) are updated based on the total loss to optimize the feature extraction and fusion process.

[0056] S260. Use the validation set to evaluate the performance of the optimized model until the model converges or reaches the preset training rounds.

[0057] Calculate the recognition accuracy, F1 score, and other indicators on the validation set to monitor model performance. Repeat steps S210-S260 above. When the model's performance on the validation set no longer improves or reaches the preset training rounds, stop training.

[0058] This method extracts spatiotemporal features and multi-scale temporal features from video and motion sensor data, respectively, to comprehensively reflect feeding behavior. Visual and motion features complement each other in feature space, reducing reliance on a single modality. For example, under light interference, visual features may be distorted, but motion features can still reflect feeding intensity. Under water disturbance, motion features may contain noise, but visual features can assist in determining behavioral patterns. Using consistency loss, the two modalities are aligned in semantic space, reducing semantic ambiguity between the modalities and improving the model's robustness to environmental interference. The motion feature extraction network uses multi-scale temporal analysis to capture the variations in the fluctuating signal at different time scales, improving its ability to distinguish feeding intensity. A cross-modal attention mechanism dynamically adjusts the weights of the two modalities based on the characteristics of feeding behavior, enabling the model to more flexibly utilize information from different modalities. For example, when feeding intensity is low, visual features may be more reliable and thus be given a higher weight by the model; when feeding intensity is high, motion features may be more sensitive and thus be given a higher weight by the model. Through feature fusion, the model can integrate behavioral and physical features, avoiding misjudgments caused by insufficient data from a single modality. For example, during light feeding, fish may be observed to slowly gather in the video, but the fluctuations in the motion features are small. During intense feeding, the fish gather more densely and move more frequently, and the fluctuations in the motion features increase significantly. The fused classification loss directly supervises the model's prediction of feeding intensity, optimizing the fused features' ability to express feeding intensity. By jointly optimizing the model using a cross-modal consistency loss and a fused classification loss, both robustness and accuracy are improved.

[0059] In some embodiments, cross-modal consistency loss is intended to force visual features (from video data) and motion features (from multi-axis sensor data) to remain aligned in the semantic space, thereby reducing semantic ambiguity between modalities and improving the robustness and accuracy of the multimodal fusion model. The present invention introduces the mean square error loss (MSE Loss) and cosine similarity loss (CS Loss) in the consistency loss, thereby combining the contrast learning ideas of absolute value and direction to align visual features with motion features, making the features of the two closer in space. The mean square error loss (MSE Loss) of feature distribution alignment quantifies the difference in numerical distribution between visual features and motion features, forcing the two to be close in feature space. The cosine similarity loss of feature direction consistency quantifies the directional similarity between visual features and motion features, forcing the two to point in the same direction in the semantic space.

[0060] The step S220 described above of calculating the cross-modal consistency loss based on the original visual features and the original motion features includes: S2201: Calculate the mean square error loss of feature distribution alignment using the original visual features and the original motion features.

[0061] The mean square error loss measures the square difference between motion features and visual features in Euclidean space to ensure that they are as close as possible in absolute value. Specifically, it is: Where, N is the sample size, and Represent the original motion features and original visual features of the i-th sample, is the L2 norm (Euclidean distance).

[0062] It represents the square distance between the original motion features and the original visual features in the Euclidean space, and the overall mean of all samples is taken as the loss.

[0063] The mean squared error (MSE) loss makes the numerical distribution of visual and motion features more consistent, avoiding feature shifts caused by modality differences. For example, if a dimension in the visual feature represents "feeding intensity," the corresponding dimension in the motion feature should also represent a similar meaning. The MSE loss will force the two dimensions to have similar values.

[0064] S2202: Calculate the cosine similarity loss of feature direction consistency using the original visual features and the original motion features.

[0065] Cosine similarity loss (CS Loss) focuses on the directional similarity of motion features and visual features in the vector space of multi-axis sensor signals, measures the angular difference between features, and ignores the difference in modulus. Specifically: Where, and are the L2 norms of the original motion features and original visual features, respectively. The theoretical range of cosine similarity is [-1, 1], but in forward tasks it is typically [0, 1]. By minimizing this, the directions of the two modal features are more consistent.

[0066] The cosine similarity loss for feature directional consistency will make the directions of visual features and motion features more consistent, ensuring semantic consistency even if there are scaling differences in their values. For example, if both visual features and motion features represent "feeding intensity", even if their value ranges are different (such as 0-1 for visual features and 0-100 for motion features), the cosine similarity loss can still ensure that they point to the same semantic meaning in the direction.

[0067] S2203 . Perform a weighted summation on the mean square error loss and the cosine similarity loss to obtain the cross-modal consistency loss CLoss.

[0068] in, and is a learnable weight parameter used to balance the two losses. and , which can balance the importance of feature distribution alignment and direction consistency.

[0069] The present invention forces visual features and motion features to be numerically close by calculating the mean square error loss, thus avoiding feature offsets caused by modal differences. By calculating the cosine similarity loss, visual features and motion features are forced to be consistent in direction, ensuring semantic alignment even if the numerical ranges are different. The two losses complement each other, significantly reducing semantic ambiguity between modalities and improving the robustness of the multimodal fusion model. Visual features may be distorted by changes in lighting, but motion features are not affected by lighting. The consistency loss forces the model to extract consistent semantic information from the two modalities, reducing its reliance on a single modality. Motion features may generate noise due to water flow, but visual features can assist in judging behavioral patterns. The consistency loss ensures that the model remains consistent even when the modal quality degrades. In the subsequent cross-modal feature fusion network, the aligned visual features and motion features can be more effectively fused to generate more discriminative fused features. For example, in the task of predicting feeding intensity, the aligned features can more accurately reflect the physical and behavioral characteristics of feeding behavior, improving classification accuracy. Consistency loss, as a regularization term, limits the overfitting of the model on a single modality and forces the model to learn shared semantic information across modalities, thereby improving generalization capabilities.

[0070] In some embodiments, the present invention introduces a 3D SwinTransformer architecture to extract features from videos. The visual feature extraction network adopts a 3D SwinTransformer architecture, and the visual feature extraction network includes a 3D block partitioning module, a linear embedding layer, a multi-layer 3D SwinTransformer block (i.e. Figure 5 Video SwinTransformer Blocks×L1) and downsampling modules in ; The visual feature extraction network extracts the spatiotemporal features of the feeding video data in the training set, generates original visual features, and refers to Figure 4 and Figure 5 As shown, including: S310 , dividing the feeding video data into video segments according to a fixed time window (eg, 2.56 seconds), and synchronously dividing the motion sensing data into time series signal segments Timeseries of the same time window.

[0071] S320: Perform size normalization processing on each frame of image in the video clip to obtain pre-processed video data.

[0072] Each frame in the video clip is resized to obtain preprocessed video data. For example, 32 frames are used as the basic unit for visual feature extraction, and each frame is scaled to 224×224.

[0073] S330: Divide the pre-processed video data into three-dimensional blocks using a three-dimensional block division module.

[0074] 3D Swin Transformer first treats the input image frame sequence Input Video (i.e. the pre-processed video data) as a 3D data volume. Then, using the 3D block partitioning module, the 3D video block partitioning (VideoPatchPartition) operation divides this 3D data volume into multiple non-overlapping 3D patches (3D Patches), each of which contains pixel information within a spatiotemporal neighborhood. Specifically, the pre-processed video data is divided into many patches containing time dimension and spatial dimension information. × × ×3 3D blocks ( is the time step, × is the spatial resolution, 3 is the RGB channel), the segmented video data is divided into × × 3D patches, including A small chunk of time, A small block of height, For example, if the input video clip is 64 frames × 224 × 224 × 3, it is divided into 8 × 56 × 56 3D blocks (block size 8 × 4 × 4 × 3).

[0075] S340. Map the pixel values ​​of each three-dimensional block to a high-dimensional feature representation through a linear embedding layer, divide the output visual feature tensor into non-overlapping three-dimensional windows, and obtain multiple window feature blocks.

[0076] A linear embedding layer is used to map the pixel values ​​of each 3D block into a high-dimensional feature representation, called a visual feature vector. (such as 96 dimensions), output visual feature tensor A∈ (F×H×W×C) , Represents a real vector space. The visual feature tensor is further divided into non-overlapping three-dimensional windows, each with a size of P×M×M (e.g. 8×7×7), to obtain window feature blocks. Therefore, these three-dimensional blocks are divided into × × Table 1 shows the meaning of the above parameters: Table 1

[0077] S350. Use the multi-layer three-dimensional sliding window transformer block to perform attention calculation on the window feature block, perform neighborhood merging and downsampling processing on the feature representation output by the three-dimensional sliding window transformer block of the current layer through the downsampling module, and input it into the three-dimensional sliding window transformer block of the next layer; perform global average pooling and linear projection processing on the feature representation output by the three-dimensional sliding window transformer block of each layer to obtain the original visual feature.

[0078] Processing is performed using each 3D sliding window transformer layer in a multi-layer 3D sliding window transformer block. In the first stage, 3D Window Multi-Head Self-Attention (3DW-MSA) is computed within the window to capture local spatiotemporal features. Building on W-MSA, this invention introduces a window sliding mechanism in SW-MSA, linking features from different windows with each slide to capture broader contextual information while maintaining low computational cost. 3D Shift Window-Multi Head Self-Attention (3DSW-MSA) enables cross-window interaction and enhances global perception. In the second stage, the output features undergo neighborhood merging (merging 2×2×2 neighborhoods) and dimensionality reduction in the first downsampling module to generate mid-scale features. Repeated downsampling and sliding window transformer processing gradually expands the receptive field and extracts high-level semantic features. The multi-scale features output by each layer undergo global average pooling to compress the spatiotemporal dimensions and generate a global feature vector with a shape of 1×1×1×D. A linear projection layer projects the global feature vector into the final raw visual features. ∈ (such as D=1024), which is used for subsequent cross-modal fusion or classification tasks.

[0079] This invention converts video data into a high-dimensional feature representation through three-dimensional block partitioning and linear embedding, preserving spatiotemporal information. A sliding window transformer hierarchically extracts local and global spatiotemporal dependencies. Downsampling and global pooling generate compact raw visual features, facilitating subsequent multimodal fusion. 3DW-MSA captures local feeding behaviors (such as fish mouth opening and closing), while 3DSW-MSA models group interactions (such as the trajectories of fish competing for food). Self-attention is calculated within a local window, significantly reducing computational complexity. Multi-stage downsampling gradually expands the receptive field, capturing spatiotemporal dependencies from local to global perspectives. 3DSW-MSA captures the temporal dynamics of fish schooling behavior (such as swimming trajectories and feeding frequency). The generated raw visual features are dimensionally aligned with the original motion features, facilitating subsequent cross-modal consistency constraints and feature fusion, providing high-quality visual feature input for the cross-modal fusion module. The 3DSwinTransformer is designed to more effectively capture temporal and spatial features in videos. Compared to the 2DSwinTransformer, it extends the temporal dimension, thereby enhancing the expressiveness and discriminability of features.

[0080] In some embodiments, each layer of the 3D sliding window transformer block includes a 3D window multi-head attention (3DW-MSA) and a 3D sliding window multi-head self-attention (3DSW-MSA). In each layer of the 3D sliding window transformer block, the input feature block is processed in the following manner: Through three-dimensional window multi-head self-attention, local spatiotemporal attention calculation is performed on the input feature block, and a feature representation with local spatiotemporal correlation is output; through three-dimensional sliding window multi-head self-attention, cross-window feature interaction is performed on the feature representation with local spatiotemporal correlation, and a feature representation with cross-window perception capability is output.

[0081] The multi-head attention is calculated as follows: in, 、 、 , superscript D Represents the dimension, represents a real vector space. , PM 2 It is the total number of patches contained in a single 3D window.

[0082] It is l Layer a The attention head is Attention weights for three-dimensional blocks; It is l Layer aThe attention head is p The attention weights of the three-dimensional blocks. L is the network layer number, a is the attention head number, p is the index of the 3D block currently being processed, p ′ is the target 3D block index for which the correlation is calculated, It is l Tier a In the attention head p The query vector of a 3D block, It is l Tier a In the attention head p ′ key vectors of target 3D blocks, It is l Tier a In the attention head p The relative position encoding of the three-dimensional blocks, It is l Tier a In the attention head p A vector of values ​​for a three-dimensional block, D is the feature dimension of each attention head, It is p The three-dimensional block p ′ attention weights of target 3D blocks, Represents a normalization operation.

[0083] Reference Figure 5 As shown in Figure 1, the 3D sliding window transformer block includes: two normalization layers (LayerNorm, LN), a 3D window multi-head self-attention (3DW-MSA) calculation layer, a first residual connection layer, a 3D sliding window multi-head self-attention (3DSW-MSA) calculation layer, a second residual connection layer and a feedforward neural network layer. The two normalization layers are respectively called the first normalization layer and the second normalization layer.

[0084] The input data of the first layer of the three-dimensional sliding window transformer block is the window feature block (size P×M×M×C) after linear embedding (LinearEmbedding), which flattens the three-dimensional feature block into a two-dimensional sequence (length PM², dimension C).

[0085] ① The first layer normalization process performs layer normalization on the input features to stabilize the feature distribution, accelerate convergence, and output the normalized features X'.

[0086] ② The 3DW-MSA calculation layer calculates local attention in a P×M×M window and uses a multi-head mechanism to split the features into h heads (each head has dimension d k=C / h), calculate the attention weights in parallel, and inject the spatiotemporal position information to output the local enhanced feature X''.

[0087] ③The first residual connection layer obtains the feature X''' through X+X'', retaining the original feature information and alleviating the gradient disappearance.

[0088] ④The second normalization layer performs layer normalization on X''' and outputs the normalized feature Y.

[0089] ⑤ The 3DSW-MSA computation layer first performs a window shift on the normalized feature Y, shifting it by ⌊P / 2⌋ frames along the temporal axis and ⌊M / 2⌋ pixels along the spatial axis (height / width), and outputs the shifted feature map. Cyclic shift is then performed, cyclically filling the feature blocks that have moved out of the boundary of the shifted feature map (for example, moving the last frame on the temporal axis to the beginning of the first frame), resulting in a cyclically shifted feature map. Cross-window interaction is then used to establish feature associations between adjacent windows. Specifically, the cyclically shifted feature map is input and multi-head attention is performed at the new window position. Each head calculates the interaction weights of the query Q, key K, and value V. A 3D relative position encoding is then injected, expanding the receptive field from P×M×M to 2P×2M×2M. The cross-window interaction feature is then output. The cross-window interaction feature is then reversely cyclically shifted by the original offset to restore the original coordinates of the feature map, outputting the spatially aligned feature Y'.

[0090] ⑥The second residual connection layer obtains the feature Y'' that integrates local and global features through X'''+Y'.

[0091] The feedforward neural network (FFN) consists of a two-layer multilayer perceptron (MLP). The first MLP quadruples the dimension of the input feature Y'', and the second MLP compresses the feature dimension back to its original size, outputting the final window feature representation Z. The first MLP uses the GELU activation function, while the second MLP performs a pure linear transformation.

[0092] In some embodiments, for the multimodal fish feeding behavior classification task, the goal is to align the features of the two modalities to maximize their interactive information, while using this interactive information to guide the learning of unimodal features. After modality encoding, a cross-modal feature fusion network with an aggregation block is further used to learn a refined multimodal representation that incorporates modality consistency and modality-specific information, guided by motion features and also guides the enhancement of motion features.

[0093] The cross-modal feature fusion network includes n cross-modal attention fusion (CAF) modules, m Transformer layers and a cross transformer layer (Cross Transformer Layer), where nm=1.

[0094] Based on the cross-modal feature fusion network, the original visual features and original motion features are cross-modally fused to obtain globally aligned fusion features, including: The first cross-modal attention fusion module takes the original visual features and the original motion features as input and generates aggregated features through a cross-modal attention mechanism; The subsequent cross-modal attention fusion module takes the original visual features and the enhanced motion features output by the previous Transformer layer as input to update the aggregated features; The fused features output by the last cross-modal attention fusion module are input into the cross transformer layer for cross-modal global alignment to obtain the globally aligned fused features.

[0095] The cross-modal feature fusion network is designed to learn multi-axis sensor signal features at different scales and adaptively learn multi-modal features from videos under the guidance of multi-scale multi-axis sensor signal features. The present invention takes the cross-modal feature fusion network (including three CAF modules, two Transformer layers and one cross transformer layer) as an example for explanation. Figure 4 As shown. The original motion feature is defined as the low-scale feature , mid-scale features and high-scale features are learned in two Transformer layers (i.e. and ), which is the enhanced motion feature mentioned above. The two Transformer layers are called the first Transformer layer and the second Transformer layer. Figure 3 The horizontal axis (Time) represents the time series, corresponding to the continuous time period of the fish's feeding behavior (such as a complete feeding cycle). The vertical axis (Acceleration X) represents the X-axis acceleration value, reflecting the intensity of the water surface fluctuations captured by the six-axis inertial sensor in the X-axis direction. The change in its value is directly related to the intensity of feeding behavior.

[0096] The above motion feature extraction network adopts the Transformer architecture. First, the continuous water surface fluctuation signal data is segmented according to a certain time window to form multiple motion sensor data segments. The segmented motion sensor data segments are projected through a linear layer to convert them into feature vectors suitable for Transformer input, called motion feature vectors. This step maps the raw motion sensor data into a high-dimensional feature space.

[0097] The Transformer architecture includes a Transformer layer, which is called the Transformer layer of the modality embedding stage. Input the Transformer layer, capture global temporal dependencies through the multi-head self-attention mechanism, and output low-scale features , this feature retains the high-frequency details of the original signal (such as instantaneous fluctuations).

[0098] The first Transformer layer is As input, it captures medium-periodic motion patterns (such as feeding action clips) through deeper temporal modeling and outputs medium-scale features The second Transformer layer is Perform high-order temporal modeling to output high-scale features that represent long-term temporal behaviors (such as a complete feeding cycle) , forming a hierarchical temporal representation system. This multi-scale design enables the model to simultaneously capture the instantaneous changes and long-term patterns of feeding behavior, providing fine-grained motion guidance signals for subsequent cross-modal fusion. Low-scale features Maintain the original sampling rate to capture transient motion details; medium-scale features Obtained by 2x downsampling, used to analyze motion patterns over medium time spans; high-scale features Extracted through 4x downsampling, it is used to identify global behavioral patterns. This multi-scale partitioning approach provides a comprehensive representation of motion information for subsequent feature fusion. Unlike the Transformer layer in the modality embedding stage, the Transformer layer in this stage directly models temporal features, without the need for indirect modeling by mapping temporal features to predefined tokens: in, is the characteristic of the current scale, is the characteristic of the previous scale, Indicates the i Layer related parameters. Indicates a size of T ×d A real matrix of ; T It is the length in the time dimension, that is, the number of time steps contained in the motion feature sequence; d It is the dimension of motion characteristics.

[0099] The CAF module mainly uses the symmetric cross-modal attention mechanism to connect the intrinsic correlation between the elements in the two input feature sequences, realizing the exchange of information between the two modalities. It uses multi-axis sensor signals to guide video information. First, initialize a multimodal aggregation module , the low-scale features and visual features Fusion, updated by multi-head attention and cross-modal attention calculations to obtain the relationship between motion features and video modalities , get the attention weight of motion features to visual features : in, represents the normalization operation, and are all learnable parameters. is the dimension of each attention head. Similarly, the self-attention weight of the visual feature can be obtained : in, is a learnable parameter, so It can be updated by weighted cross-modal features and weighted self-attention features as follows: in , Indicates the j Aggregate features of the layers, and These are all learnable parameters.

[0100] Reference Figure 6 As shown in the figure, the Cross Attention Fusion (CAF) module includes multiple layer normalization modules, linear projection modules, multi-head cross attention modules, layer normalization modules, multi-head self attention modules and feed-forward networks. The processing of the CAF module includes: Visual features of the input and motion characteristics Layer normalization is performed separately to obtain normalized visual features and motion features.

[0101] The normalized visual features are linearly projected to generate key (K) and value (V) matrices, and the normalized motion features are linearly projected to generate query (Q) matrix.

[0102] Use Q, K, and V as the input of the multi-head cross attention module, calculate the attention weight α of motion features to visual features through multi-head cross attention, and output weighted visual features.

[0103] The weighted visual features are subjected to layer normalization processing to obtain normalized weighted visual features.

[0104] The normalized motion features are passed through the multi-head self-attention module to calculate the self-attention weight β , output weighted motion features; The results of cross attention and self-attention (i.e. the normalized weighted visual features and weighted motion features) are layer-normalized to obtain normalized features, and the normalized features are nonlinearly transformed through the feed-forward network to generate the aggregated features of the current CAF module. , realizing the conversion from motion-related features to vision-related features .

[0105] The cross-modal feature fusion network of the present invention realizes the effective fusion of motion features and visual features through a multi-level structure. The processing flow is as follows: First, the two Transformer layers perform deep processing on medium and high scale features respectively. The first Transformer layer directly processes low scale features. , the self-attention mechanism is used to model the temporal relationship of motion within a medium time range, and the processed medium-scale features are output The second Transformer layer processes the mesoscale features in the same way. , capturing longer-term motion patterns and outputting processed high-scale features '.

[0106] Then, refer to Figure 4 As shown in the figure, the three cross-modal attention fusion modules fuse features of different scales in turn. The first cross-modal attention fusion module combines the above low-scale features With the original visual features Combined with the attention mechanism guided by motion features, the key feeding areas in the video are highlighted, and the The second cross-modal attention fusion module combines the mid-scale features processed by Transformer With the original visual features Fusion, establish cross-modal associations with medium time span, and obtain The third cross-modal attention fusion module integrates the processed high-scale features and original visual features , capturing the macro behavior pattern, and obtaining Each cross-modal attention fusion module adopts a residual connection structure to ensure that motion feature information is not lost during the fusion process.

[0107] Finally, the cross transformer layer is the high-scale feature output of the last transformer layer. And the aggregated features output by the last CAF module ( ) as input, and realize the refined alignment of temporal and visual features through the cross-modal attention mechanism, and output feature representation with rich spatiotemporal information, namely the above-mentioned global alignment fusion feature.

[0108] The present invention integrates the fish feeding behavior recognition model of fish school feeding video and water surface fluctuation characteristics. By fusing video features with water surface fluctuation characteristics fed back by a multi-axis sensor, it overcomes the shortcomings of a single modality. At the same time, it introduces efficient CAF and uses motion features of low, medium and high scales to guide video features to obtain aggregation features of low, medium and high scales, further improving the efficiency and accuracy of feeding behavior recognition, realizing the precise feeding control of fish schools, reducing feed waste, and lowering the requirements for the actual breeding site environment.

[0109] Reference Figure 7a and Figure 7b The figure shows the effectiveness of the proposed fish feeding behavior recognition model for long-term fish feeding behavior recognition. By comparing the prediction results with the actual labels, it can be seen that the four-level feeding intensity classification (Strong / Medium / Weak / None) output by the model is highly consistent with the actual situation, especially in areas with sudden changes in feeding intensity (such as the rapid transition from None to Strong), showing excellent real-time response capabilities.

[0110] The following describes the fish feeding behavior recognition device provided by the present invention. The fish feeding behavior recognition device described below and the fish feeding behavior recognition method described above can be referenced to each other.

[0111] The fish feeding behavior recognition device provided by the present invention refers to Figure 8 As shown, including: Data acquisition module 410 is used to acquire synchronously collected fish feeding video data and water surface fluctuation signal data, as well as a pre-trained fish feeding behavior recognition model; wherein the water surface fluctuation signal data is a multi-axis motion sensing signal acquired by a multi-axis sensor; wherein the fish feeding behavior recognition model is trained by: acquiring a multi-modal data set of fish feeding, including synchronously collected feeding video data, motion sensing data, and true labels, where the true labels are feeding intensity classification levels; dividing the multi-modal data set of fish feeding into a training set and a validation set according to a preset ratio; and training a pre-constructed multi-modal fusion neural network model based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model; The behavior recognition module 420 is used to input the fish feeding video data and the water surface fluctuation signal data into the pre-trained fish feeding behavior recognition model to obtain a fish feeding behavior recognition result.

[0112] Figure 9 An example of a physical structure diagram of an electronic device is shown below. Figure 9 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, the communications interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call logic instructions in the memory 530 to execute the method for identifying the feeding behavior of a school of fish.

[0113] In addition, the logic instructions in the aforementioned memory 530 can be implemented in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0114] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the fish feeding behavior identification method provided by the above methods.

[0115] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the fish feeding behavior identification method provided by the above methods.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0117] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for identifying feeding behavior of a school of fish, characterized in that: include: Acquire synchronously collected fish feeding video data and water surface fluctuation signal data, and a pre-trained fish feeding behavior recognition model; wherein the water surface fluctuation signal data is a multi-axis motion sensing signal collected by a multi-axis sensor; Inputting the fish feeding video data and the water surface fluctuation signal data into the pre-trained fish feeding behavior recognition model to obtain a fish feeding behavior recognition result; The fish feeding behavior recognition model is trained in the following way: Obtain a multimodal dataset of fish feeding, including synchronously collected feeding video data, motion sensor data, and true labels, where the true labels are the feeding intensity classification levels; Dividing the fish feeding multimodal dataset into a training set and a validation set according to a preset ratio; A pre-built multimodal fusion neural network model is trained based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model.

2. The method for identifying the feeding behavior of a school of fish according to claim 1, wherein: The multimodal fusion neural network model includes a visual feature extraction network, a motion feature extraction network, a cross-modal feature fusion network and a classification layer; The pre-built multimodal fusion neural network model is trained based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model, including: Based on the visual feature extraction network, the spatiotemporal features of the feeding video data in the training set are extracted to generate original visual features; Extracting original motion features of motion sensor data in the training set based on motion feature extraction network; Calculate cross-modal consistency loss based on original visual features and original motion features; Based on the cross-modal feature fusion network, the original visual features and the original motion features are cross-modally fused to obtain the globally aligned fusion features; Inputting the globally aligned fusion features into a classification layer to obtain a prediction result, and calculating a fusion classification loss based on the prediction result and the true label; Calculating a total loss based on the cross-modal consistency loss and the fusion classification loss; Back propagation is performed according to the total loss to update the model parameters of the multimodal fusion neural network model to obtain an optimized model; Use the validation set to evaluate the performance of the optimized model until the model converges or reaches the preset training rounds.

3. The method for identifying the feeding behavior of a school of fish according to claim 2, wherein: Based on the original visual features and original motion features, the cross-modal consistency loss is calculated, including: Calculating a mean square error loss for feature distribution alignment using the original visual features and the original motion features; Calculate the cosine similarity loss of feature direction consistency using the original visual features and the original motion features; A weighted sum is performed on the mean square error loss and the cosine similarity loss to obtain the cross-modal consistency loss.

4. The method for identifying fish feeding behavior according to claim 2, wherein: The visual feature extraction network adopts a three-dimensional sliding window transformer architecture, which includes a three-dimensional block partitioning module, a linear embedding layer, a multi-layer three-dimensional sliding window transformer block and a downsampling module; The method of extracting spatiotemporal features of the feeding video data in the training set based on the visual feature extraction network to generate original visual features includes: The feeding video data is divided into video segments according to a fixed time window, and the motion sensor data is synchronously divided into time series signal segments of the same time window; Performing size normalization processing on each frame image in the video clip to obtain preprocessed video data; Dividing the pre-processed video data into three-dimensional blocks using a three-dimensional block division module; The pixel values ​​of each 3D block are mapped to a high-dimensional feature representation through a linear embedding layer, and the output visual feature tensor is divided into non-overlapping 3D windows to obtain multiple window feature blocks; The multi-layer three-dimensional sliding window transformer block is used to perform attention calculation on the window feature block, and the feature representation output by the three-dimensional sliding window transformer block of the current layer is subjected to neighborhood merging and downsampling processing through the downsampling module, and then input into the three-dimensional sliding window transformer block of the next layer; the feature representation output by the three-dimensional sliding window transformer block of each layer is subjected to global average pooling and linear projection processing to obtain the original visual feature.

5. The method for identifying the feeding behavior of a school of fish according to claim 4, wherein: Each layer of the 3D sliding window transformer block contains 3D window multi-head attention and 3D sliding window multi-head self-attention. In each layer of the 3D sliding window transformer block, the input feature block is processed in the following way: Through the three-dimensional window multi-head self-attention, the local spatiotemporal attention calculation is performed on the input feature block, and the feature representation with local spatiotemporal correlation is output; Through three-dimensional sliding window multi-head self-attention, cross-window feature interaction is performed on the feature representation with local spatiotemporal correlation, and a feature representation with cross-window perception capability is output.

6. The method for identifying the feeding behavior of a school of fish according to claim 2, wherein: The cross-modal feature fusion network includes n cross-modal attention fusion modules, m Transformer layers and a cross transformer layer, where nm=1; Based on the cross-modal feature fusion network, the original visual features and original motion features are cross-modally fused to obtain globally aligned fusion features, including: The first cross-modal attention fusion module takes the original visual features and the original motion features as input and generates aggregated features through a cross-modal attention mechanism; The subsequent cross-modal attention fusion module takes the original visual features and the enhanced motion features output by the previous Transformer layer as input to update the aggregated features; The fused features output by the last cross-modal attention fusion module are input into the cross transformer layer for cross-modal global alignment to obtain the globally aligned fused features.

7. A device for identifying fish feeding behavior, characterized in that: include: a data acquisition module for acquiring synchronously collected fish feeding video data and water surface fluctuation signal data, and a pre-trained fish feeding behavior recognition model; wherein the water surface fluctuation signal data is a multi-axis motion sensing signal collected by a multi-axis sensor; wherein the fish feeding behavior recognition model is trained in the following manner: acquiring a fish feeding multimodal dataset, including synchronously collected feeding video data, motion sensing data, and true labels, where the true labels are feeding intensity classification levels; dividing the fish feeding multimodal dataset into a training set and a validation set according to a preset ratio; and training a pre-constructed multimodal fusion neural network model based on the training set and the validation set to obtain the pre-trained fish feeding behavior recognition model; The behavior recognition module is used to input the fish feeding video data and the water surface fluctuation signal data into the pre-trained fish feeding behavior recognition model to obtain a fish feeding behavior recognition result.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method for identifying the feeding behavior of a school of fish as claimed in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying the feeding behavior of a school of fish as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method for identifying the feeding behavior of a school of fish as claimed in any one of claims 1 to 6 is implemented.

Images