Multi-mode fish school feeding intensity detection method, system, equipment, medium and product

Abstract

The invention provides a multi-mode fish school ingestion intensity detection method, system and device, a medium and a product, and relates to the technical field of aquaculture, the method comprises the following steps: obtaining fish school ingestion target video data and fish school ingestion target audio data in a time period corresponding to the fish school ingestion target video data; inputting the fish school ingestion target video data and the fish school ingestion target audio data into a multi-mode ingestion intensity detection model to obtain fish school ingestion intensity information output by the multi-mode ingestion intensity detection model; wherein the multi-modal ingestion intensity detection model is obtained by training a convolutional neural network according to fish shoal ingestion sample video data and fish shoal ingestion sample audio data marked with a fish shoal ingestion behavior intensity label; the convolutional neural network comprises a Res2Net network, a RepViT network, a feature fusion residual connection module and a full connection layer. According to the invention, more accurate fish school ingestion intensity information can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of aquaculture, and particularly relates to a multi-modal fish school feeding intensity detection method, system, device, medium and product. Background Art

[0002] In aquaculture, reasonable feeding is crucial for fish growth, production efficiency and cost control. Traditional aquaculture methods often rely on manual experience for feeding, making it difficult to accurately grasp the actual needs of fish, which easily leads to underfeeding or overfeeding. Underfeeding will affect the normal growth of fish, while overfeeding will cause feed waste and water quality deterioration, thereby increasing the risk of fish diseases.

[0003] At present, the research on fish feeding behavior recognition mainly conducts single-modal feeding behavior recognition based on technologies such as acoustics, images and sensors. However, the feeding behavior of fish is complex and variable, and existing feeding behavior recognition methods are easily affected by factors such as water flow, fish movement and external interference, resulting in signal overlap and interference, making it difficult to accurately distinguish different feeding intensities and behaviors, and leading to low accuracy of feeding behavior recognition.

[0004] Therefore, there is an urgent need for a multi-modal fish school feeding intensity detection method, system, device, medium and product to solve the above problems. Summary of the Invention

[0005] In view of the problems existing in the prior art, the present invention provides a multi-modal fish school feeding intensity detection method, system, device, medium and product.

[0006] The present invention provides a multi-modal fish school feeding intensity detection method, including: Obtaining fish school feeding target video data and fish school feeding target audio data corresponding to the time period of the fish school feeding target video data; Inputting the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain fish school feeding intensity information output by the multi-modal feeding intensity detection model; wherein, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module and a fully connected layer.

[0007] According to the multi-modal fish school feeding intensity detection method provided by the present invention, the multi-modal feeding intensity detection model is trained through the following steps: Obtain the feeding history data of the fish school, where the feeding history data of the fish school includes the feeding sample video data generated in the historical period, and the feeding sample audio data of the fish school corresponding to the historical period of the feeding sample video data; Based on a preset time window, perform time-axis alignment cutting on the feeding history data of the fish school to obtain a fish school feeding behavior video data set and a fish school feeding behavior audio data set; Based on the feeding behavior intensity categories in the fish school feeding behavior video data set or the fish school feeding behavior audio data set, mark the corresponding fish school feeding behavior intensity labels for the sample data in the fish school feeding behavior video data set and the fish school feeding behavior audio data set to obtain a training sample set; According to the training sample set, train the convolutional neural network to obtain the multi-modal feeding intensity detection model.

[0008] According to a multi-modal fish school feeding intensity detection method provided by the present invention, a coordinate attention mechanism is set in the Res2Net network to extract features from the feeding sample audio data of the fish school to obtain sound sample features; The RepViT network is used to extract features from the feeding sample video data of the fish school to obtain visual sample features; The feature fusion residual connection module is used to perform linear transformation and residual fusion on the visual sample features and the sound sample features to obtain multi-modal fusion sample features; The fully connected layer is used to obtain the corresponding fish school feeding behavior intensity prediction probability according to the multi-modal fusion sample features, and adjust the model parameters of the convolutional neural network according to the loss value calculated from the fish school feeding behavior intensity prediction probability.

[0009] According to a multi-modal fish school feeding intensity detection method provided by the present invention, the Res2Net network is a hierarchical residual connection structure, and multiple convolutional branches are provided in the residual block of the Res2Net network, where, except for the first convolutional branch, the output of the current convolutional branch is used as the input of the next convolutional branch.

[0010] According to a multi-modal fish school feeding intensity detection method provided by the present invention, the feature fusion residual connection module is specifically used for: Perform a linear transformation on the visual sample features to obtain linearly transformed visual sample features; Perform a linear transformation on the sound sample features to obtain linearly transformed sound sample features; Fuse the linearly transformed visual sample features and the linearly transformed sound sample features to obtain preliminary linearly fused features; Perform residual linear transformation on the visual sample features and the sound sample features respectively to obtain the visual sample features after residual linear transformation and the sound sample features after residual linear transformation; Fuse the preliminary linear fusion features, the visual sample features after residual linear transformation, and the sound sample features after residual linear transformation to obtain the multi-modal fusion sample features.

[0011] According to a multi-modal fish school feeding intensity detection method provided by the present invention, after inputting the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain the fish school feeding intensity information output by the multi-modal feeding intensity detection model, the method further includes: Adjust the current fish school bait feeding strategy according to the fish school feeding intensity information.

[0012] The present invention also provides a multi-modal fish school feeding intensity detection system, including: A multi-modal data acquisition module for acquiring fish school feeding target video data and fish school feeding target audio data corresponding to a time period of the fish school feeding target video data; A fish school feeding intensity detection module for inputting the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain the fish school feeding intensity information output by the multi-modal feeding intensity detection model; wherein, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module, and a fully connected layer.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the multi-modal fish school feeding intensity detection method as described in any one of the above.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the multi-modal fish school feeding intensity detection method as described in any one of the above.

[0015] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, it implements the multi-modal fish school feeding intensity detection method as described in any one of the above.

[0016] The multi-modal fish school feeding intensity detection method, system, device, medium and product provided by the present invention obtain the video of the fish school feeding target and the corresponding audio data in a time period, and then input them into a convolutional neural network with a Res2Net network, a RepViT network, a feature fusion residual connection module and a fully connected layer, and which has been trained with sample multi-modal data, so as to obtain more accurate fish school feeding intensity information. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0018] Figure 1 It is a schematic flowchart of the multi-modal fish school feeding intensity detection method provided by the present invention; Figure 2 It is a schematic diagram of the overall structure of the multi-modal feeding intensity detection model provided by the present invention; Figure 3 It is a schematic diagram of the structure of the coordinate attention mechanism provided by the present invention; Figure 4 It is a schematic diagram of the structure of the feature fusion residual connection module provided by the present invention; Figure 5 It is a schematic diagram of the structure of the multi-modal fish school feeding intensity detection system provided by the present invention; Figure 6 It is a schematic diagram of the fish school feeding behavior acquisition scenario in the aquaculture area provided by the present invention; Figure 7 It is a schematic diagram of the detection results corresponding to different feeding behavior intensity labels provided by the present invention; Figure 8 It is a schematic diagram of the structure of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

[0020] Currently, the research on fish feeding behavior recognition mainly relies on technologies such as acoustics, images, and sensors. However, fish feeding behavior is complex and variable, and is easily affected by water flow, fish movement, and external interference, resulting in signal overlap and interference, making it difficult to accurately distinguish different feeding intensities and behaviors, which is specifically reflected in the following aspects: The image-based recognition method, although able to quantify the feeding behavior of fish schools, is limited by the image quality and has obvious limitations. For example, turbid water and poor light will cause a decrease in image contrast. The rapid swimming of fish schools may cause out-of-focus or ghosting, and bait occlusion may also lead to misjudgment. These problems limit the application effect of machine vision methods in actual aquaculture. In addition, the neural network model is restricted by the quality of image preprocessing and also has problems such as complex model structure, difficult training, and deployment.

[0021] Acoustic technology is not affected by light conditions and can identify feeding behavior by analyzing the changes in acoustic signals generated during fish feeding. Different behaviors of fish, such as eating and breeding, are accompanied by different vocalization characteristics. During feeding, actions such as the fish's rapid tail slapping, collisions, biting, and swallowing will cause changes in acoustic signals, which can effectively reflect the feeding intensity information of the fish school. However, acoustic technology also has problems such as high equipment cost and sensitivity to noise, which may affect the accuracy of recognition.

[0022] Methods such as acceleration sensors, such as implanted accelerometers, although able to monitor the acceleration changes of fish, have potential harm to the fish body and are difficult to be widely applied in actual aquaculture. Moreover, the signals collected by acceleration sensors are easily interfered by factors such as water surface fluctuations and fish movement. Especially when fry or low-density fish schools are feeding, the signal intensity changes are not obvious, making it difficult to accurately analyze the feeding intensity.

[0023] Existing single-modal methods in fish feeding behavior recognition all have certain limitations and are difficult to meet the requirements of precise feeding. For example, image-based methods are difficult to cope with complex lighting and water quality conditions, acoustic technology is sensitive to noise, and sensor methods are restricted by equipment cost and fish health, and cannot provide richer and more reliable information, resulting in lower accuracy and reliability of feeding behavior recognition.

[0024] In view of the problems existing in the above-mentioned prior art, the present invention provides a technology for fusing images and sounds. By analyzing the changes in image and sound signal data, the feeding behavior of fish schools is accurately analyzed, so as to achieve intelligent feeding. Among them, the present invention provides a Reinforced Cross-modal Uni-Modal Teacher (abbreviated as RC-UMT) model based on the fusion of image features and sound features, that is, a multi-modal feeding intensity detection model. By fusing the features extracted from images and audio through residual connection, the recognition accuracy of fish feeding behavior is improved, providing reliable technical support for precise feeding. In addition, the recognition accuracy of the RC-UMT model in the present invention for four different feeding intensities (strong, medium, weak, none) of fish can reach 93%, which is higher than that of a single audio or video modality and 7% higher than the basic model. It can reduce the breeding cost, reduce water pollution, and promote the intelligent development of fishery breeding.

[0025] Figure 1 It is a schematic flow chart of the multi-modal fish school feeding intensity detection method provided by the present invention, as Figure 1 shown, the present invention provides a multi-modal fish school feeding intensity detection method, including: Step 101, obtain the fish school feeding target video data and the fish school feeding target audio data corresponding to the time period of the fish school feeding target video data.

[0026] In the present invention, through a specific video acquisition device (such as a camera), shooting is carried out in the environment where the fish school feeds, so as to obtain video data that can reflect the feeding situation of the fish school. These video data record various behavioral performances of the fish school during the feeding process, such as the swimming trajectory of the fish school, feeding actions, etc., and are an important basis for subsequent analysis of the feeding behavior of the fish school.

[0027] At the same time, an audio acquisition device (such as an underwater microphone) is used to synchronously acquire the fish school feeding audio data in the same time period as the above video data. Various sounds will be generated by fish during the feeding process, such as the sound of biting bait, body collision, etc. These audio data contain the acoustic characteristics of the fish school feeding behavior and can reflect the feeding situation of the fish school from another angle. It should be noted that the fish school feeding audio data obtained in the present invention is spectrogram data.

[0028] Step 102: Input the fish school feeding target video data and the fish school feeding target audio data into the multi-modal feeding intensity detection model to obtain the fish school feeding intensity information output by the multi-modal feeding intensity detection model. Among them, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels. The convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module, and a fully connected layer.

[0029] In the present invention, the obtained fish school feeding target video data and the corresponding period audio data are input into the pre-constructed multi-modal feeding intensity detection model together. The multi-modal feeding intensity detection model is a model that comprehensively utilizes video and audio modality information for processing, and can detect the feeding intensity of the fish school more comprehensively and accurately. Through the processing and analysis of the multi-modal feeding intensity detection model, information about the feeding intensity of the fish school will be output. These information can intuitively reflect the feeding activity level of the fish school in the current time period. For example, different levels such as strong, medium, weak, and no feeding intensity, providing a reference basis for subsequent aquaculture management (such as precise feeding).

[0030] Specifically, in the present invention, the multi-modal feeding intensity detection model is obtained by training with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels. During training, a large amount of fish school feeding sample data needs to be collected, and each sample is labeled with the feeding behavior intensity (for example, by professional personnel judging the feeding intensity according to video and audio features and attaching corresponding labels), and then the convolutional neural network is trained with these labeled sample data so that the network can learn the mapping relationship between video and audio features and feeding intensity. Moreover, the convolutional network structure of the present invention is improved. Among them, the Res2Net network is an improved residual network. By dividing the feature map into multiple subgroups and performing hierarchical connections between the subgroups, it can more effectively extract the multi-scale features of the spectrogram corresponding to the audio data, thereby better capturing the detailed information of the fish school feeding behavior in the audio data.

[0031] The RepViT network combines the feature extraction ability of the vision Transformer and the local perception characteristics of the convolutional neural network. By optimizing the model structure through the reparameterization technique, it can extract features more efficiently when processing image data, which helps to improve the recognition ability of the fish school feeding behavior in the video.

[0032] The role of the feature fusion residual connection module is to fuse the features extracted from video data and audio data. Since video and audio respectively reflect the feeding behavior of fish schools from different perspectives, effectively fusing the features they extract can make full use of the information of the two modalities and improve the accuracy of detection. The residual connection helps to solve the problem of gradient disappearance during the training of deep networks and ensures the effective transmission of features during the fusion process. The fully connected layer is located at the end of the convolutional neural network and is used to integrate and map the features extracted by the previous network, and finally output the prediction result of the feeding intensity of the fish school. In the present invention, the fully connected layer converts the features into specific feeding intensity categories or values through operations such as weighted summation of the previously extracted features, and completes the detection task of the model.

[0033] The multi-modal fish school feeding intensity detection method provided by the present invention obtains the fish school feeding target video and the corresponding audio data in a period, and then inputs them into a convolutional neural network with a Res2Net network, a RepViT network, a feature fusion residual connection module and a fully connected layer, and which has been trained with sample multi-modal data, so as to obtain more accurate fish school feeding intensity information.

[0034] Based on the above embodiments, the multi-modal feeding intensity detection model is trained through the following steps: Obtain the historical data of fish school feeding. Among them, the historical data of fish school feeding includes the fish school feeding sample video data generated in the historical period, and the fish school feeding sample audio data corresponding to the historical period of the fish school feeding sample video data; Based on a preset time window, perform time-axis alignment cutting on the historical data of fish school feeding to obtain a fish school feeding behavior video data set and a fish school feeding behavior audio data set; Based on the fish school feeding behavior intensity categories in the fish school feeding behavior video data set or the fish school feeding behavior audio data set, label the sample data in the fish school feeding behavior video data set and the fish school feeding behavior audio data set with the corresponding fish school feeding behavior intensity labels to obtain a training sample set; According to the training sample set, train the convolutional neural network to obtain the multi-modal feeding intensity detection model.

[0035] In the present invention, it is necessary to collect feeding-related data generated by the fish school during a historical period, specifically including video data of fish school feeding samples and corresponding audio data of fish school feeding samples. The video data is captured by devices such as cameras, recording the visual behaviors of the fish school during the feeding process. For example, by using a camera 3 meters above the ground above the fish tank in the recirculating aquaculture system to capture the swimming, foraging, and bait swallowing actions of the fish school during feeding. In the present invention, frame extraction is performed on the audio data of the fish school feeding samples, and then the frame-extracted audio data is converted into a spectrogram, and the spectrogram is further processed by data augmentation and normalization to better extract audio features in subsequent steps.

[0036] The audio data can be collected by audio devices such as underwater microphones. For example, a hydrophone is suspended and placed in the recirculating aquaculture pond, and connected to a computer through a network cable. The obtained audio data of the fish school feeding contains various sounds generated during the fish school feeding, such as the sounds made by the fish biting the bait and the body collision. Among them, when collecting the audio data, the data collection and saving frequency is set to once every 2 minutes, and the time of the audio device and the computer is synchronized.

[0037] Furthermore, a specific time length (i.e., a preset time window) is set as the time window. For example, every 10 seconds, 30 seconds, etc. is a time window. The size of the preset time window can be adjusted according to actual needs and the characteristics of the data, and it determines the duration of the sample data obtained by subsequent cutting. Taking the preset time window as the unit, the obtained historical data of the fish school feeding is cut. During the cutting process, it is necessary to ensure that the video data and the audio data are aligned on the time axis, that is, each cut video sample and the corresponding audio sample are synchronized in time, so as to ensure that during subsequent processing, the video and audio information can be accurately corresponded to jointly reflect the feeding behavior of the fish school during a specific time period.

[0038] After cutting, a video dataset of the fish school feeding behavior and an audio dataset of the fish school feeding behavior are obtained. The sample data in these two datasets are all in units of the preset time window, and each sample contains the feeding behavior information of the fish school during this time period. Then, the feeding behavior intensity of the fish school is classified. For example, the feeding intensity is divided into four categories: strong, medium, weak, and none. Among them, the basis for classification can be features such as the feeding action frequency, amplitude of the fish school, and the intensity and frequency distribution of the audio signal.

[0039] Further, based on the fish school feeding behavior intensity categories in the fish school feeding behavior video dataset or the fish school feeding behavior audio dataset, each sample data in the fish school feeding behavior video dataset and the fish school feeding behavior audio dataset is labeled. For example, if a certain video sample and the corresponding audio sample show frequent and large-scale feeding actions of the fish school, and the audio signal intensity is high, then this sample is labeled with a "strong" feeding behavior intensity label. Through such a labeling process, each sample data has a corresponding feeding behavior intensity label, thus obtaining a training sample set. In the present invention, a part of the training sample set is divided into a test set (for example, 80% of the samples in the training sample set are used for training, and 20% of the samples are used for testing) to evaluate and verify the performance of the model trained based on this training sample set.

[0040] Further, the obtained training sample set is input into a convolutional neural network for training. During the training process, the convolutional neural network continuously learns the features in the video data and audio data, as well as the mapping relationship between these features and the fish school feeding behavior intensity labels. By adjusting the parameters of the network, the convolutional neural network can more accurately predict the corresponding fish school feeding behavior intensity according to the input video and audio data. After a certain number of iterative trainings, by optimizing the loss function (such as the cross-entropy loss function) through an algorithm, when the performance of the model reaches the expected requirements (for example, the accuracy rate on the test set reaches a certain level), the training process ends, and a multi-modal feeding intensity detection model is obtained. When the multi-modal feeding intensity detection model receives new fish school feeding video data and audio data, it can output the corresponding feeding behavior intensity information, providing decision-making support for aquaculture management such as precise feeding of fish. In the present invention, the model batch size is set to 16, the number of iterative cycles is set to 1000, the learning rate is set to 0.001, the network parameters are optimized through the Adam optimizer, and the acceleration environment during training is CUDA 11.7 and CUDNN 8.5.0.

[0041] Based on the above embodiments, a coordinate attention mechanism is set in the Res2Net network for feature extraction of the fish school feeding sample audio data to obtain sound sample features; The RepViT network is used for feature extraction of the fish school feeding sample video data to obtain visual sample features; The feature fusion residual connection module is used for linearly transforming and residually fusing the visual sample features and the sound sample features to obtain multi-modal fusion sample features; The fully connected layer is used to obtain the corresponding fish school feeding behavior intensity prediction probability according to the multi-modal fusion sample features, and adjust the model parameters of the convolutional neural network according to the loss value calculated based on the fish school feeding behavior intensity prediction probability.

[0042] In the present invention, Res2Net can more effectively extract multi-scale features of images by dividing the feature map into multiple subgroups and performing hierarchical connections between the subgroups. When processing the audio data of fish school feeding samples, in order to further improve the extraction effect of audio data features, a Coordinate Attention (CA) mechanism is set in the Res2Net network. The CA mechanism can perform weighted processing on the spatial coordinates of the input feature map to capture the features in different frequency ranges of the audio signal. In the audio spectrogram, the features in different frequency ranges and time periods may have different importance for reflecting the fish school feeding behavior. The CA mechanism can automatically learn these importance differences, and by assigning different weights to different positions of the feature map, the convolutional neural network can focus on the key information closely related to the fish school feeding behavior in the audio, such as the sound changes in a specific frequency range, so as to more accurately extract the sound sample features. These sound sample features contain rich information of the audio signal during fish school feeding, such as the sound intensity, frequency distribution, etc., providing important audio feature representations for subsequent multi-modal fusion and classification.

[0043] The RepViT network can perform frame-by-frame analysis on video frames to capture the global features and local features in the images. Among them, the global features can reflect macroscopic information such as the overall distribution and swimming direction of the fish school, while the local features can focus on the action details of individuals in the fish school, such as the opening and closing of the fish's mouth and the body swing. Through this multi-level feature extraction, the RepViT network can further mine the semantic information related to the fish school feeding behavior in the video data to obtain visual sample features. These visual sample features intuitively show the visual performance of the fish school during the feeding process, providing a visual basis for the model to understand the fish school feeding behavior.

[0044] Furthermore, after obtaining the visual sample features and sound sample features, since these two types of features come from different modalities, their dimensions and representation forms may be different. In order to enable them to effectively interact and integrate in the subsequent fusion process, the present invention performs linear transformation on the visual sample features and sound sample features through a feature fusion residual connection module, and adds a residual connection during feature fusion, effectively avoiding information loss, alleviating the problem of gradient disappearance, reducing the number of parameters, and improving the recognition accuracy of the model.

[0045] After linear transformation and residual fusion, the visual sample features and audio sample features are effectively integrated to obtain multi-modal fusion sample features. This multi-modal fusion sample feature contains information on both the visual and audio aspects of the fish school feeding behavior, can more comprehensively reflect the feeding state of the fish school, and provides a richer feature representation for subsequent classification prediction. Finally, the fully connected layer integrates and maps the multi-modal fusion sample features to output the predicted probability of the fish school feeding behavior intensity. During the training process, according to the predicted probability of the fish school feeding behavior intensity output by the fully connected layer and the true label of the sample (i.e., the actual category of the fish school feeding behavior intensity), the loss value can be calculated. The loss value reflects the degree of difference between the prediction result of the convolutional neural network and the true result. During the model training process, through the backpropagation algorithm, the loss value is propagated backward along the network layer, and the parameters of the convolutional neural network (including the Res2Net network, RepViT network, and feature fusion residual connection module, etc.) are adjusted according to the size of the loss value and the gradient information. The purpose of the adjustment is to make the prediction result of the convolutional neural network closer to the true result, reduce the loss value, and thus improve the recognition accuracy of the convolutional neural network. Through continuous iterative training and parameter adjustment, the convolutional neural network can gradually learn the accurate mapping relationship between the visual and audio features and the fish school feeding behavior intensity, and achieve effective classification prediction of the fish school feeding behavior intensity.

[0046] Figure 2 For the overall structural schematic diagram of the multi-modal feeding intensity detection model provided by the present invention, reference can be made to Figure 2 As shown, the audio encoder uses a Res2Net network combined with a coordinate attention mechanism (CA) to extract features from the input fish school feeding sample audio data. The hierarchical residual connection structure of Res2Net helps to extract multi-scale features, while the CA mechanism can capture the position information in the feature map, enabling the model to focus on key regions.

[0047] The visual encoder uses a RepViT network to extract features from the input fish school feeding sample video data. RepViT combines the powerful feature extraction ability of the vision Transformer and the local perception characteristics of the convolutional neural network, and can effectively capture the features in the video data.

[0048] In the present invention, the RepViT network is built based on MobileNetV3 and adopts a MetaFormer structure similar to ViT. Among them, the 1×1 convolution and 3×3 depthwise separable convolutions in the RepViT block are effectively coordinated through the structural reparameterization technique, enhancing the model's learning ability during training and reducing the computational cost during inference. At the macro architecture level, the early convolution at the front end can improve the stability and performance of processing high-resolution video frames, the deep separable downsampling layer reduces information loss, and the simple classifier reduces latency. At the micro level, the 3×3 convolution ensures inference efficiency, and the cross-block placement strategy of the Squeeze-and-Excitation (SE) layer enhances the attention to key features.

[0049] In the present invention, the RepViT network reparameterizes the ViT model, reducing the computational complexity and the number of model parameters while maintaining the Transformer's modeling ability, which helps to improve the efficiency of video feature extraction. Especially in multi-modal tasks, it can better capture long-range dependencies and global features. By performing low-rank approximation on the matrix operations in the self-attention mechanism, the RepViT network significantly reduces the computational complexity and simultaneously reduces memory consumption.

[0050] Furthermore, the residual linear layer in the cross-modal encoder performs further linear transformation on the features extracted by the visual encoder and the audio encoder, and retains the original feature information through the residual connection, which helps the transmission of information and the flow of gradients. Among them, the linear layer is used to perform linear transformation on the features, adjusting the dimension and distribution of the features. Then, the features of the visual and audio modalities are fused through "Fused" to comprehensively utilize the information of the two modalities to more comprehensively describe the feeding behavior of fish.

[0051] In the present invention, gradient modulation is performed on the fused features to adjust the gradient update amplitude of different modality features during the training process to balance the contributions of different modalities to model training and avoid a certain modality dominating the training process.

[0052] During the training process, the model parameters are updated through classification and loss calculation. Among them, the softmax operation is respectively performed on the features of the visual modality (logits-a) and the audio modality (logits-v) to obtain the probability distribution of each category. Then, the cross entropy loss is used to measure the difference between the model prediction result and the true label. At the same time, the discrepancy ratio can also be calculated to evaluate the consistency between the prediction results of the two modalities. In the present invention, the feeding intensity of fish is divided into four categories: strong, medium, weak, and none. By training the model to learn the mapping relationship between the features and these categories, the model can finally accurately predict the feeding intensity of fish.

[0053] Based on the above embodiments, the Res2Net network is a hierarchical residual connection structure, and multiple convolutional branches are arranged in the residual block of the Res2Net network. Among them, except for the first convolutional branch, the output of the current convolutional branch is used as the input of the next convolutional branch.

[0054] In the present invention, the Res2Net network adopts a hierarchical residual connection structure, which enhances the network's ability to extract multi-scale features. Information with different frequency components has different importance for characterizing fish feeding behavior. The Res2Net network can capture multi-scale features in the audio signal at a fine-grained level, so as to better extract audio features related to fish feeding behavior.

[0055] In the present invention, multiple convolutional branches are introduced into the residual block of the Res2Net network to increase the receptive field of the network, so as to better capture features of different scales in the audio data. Specifically, the Res2Net network utilizes the idea of a feature pyramid to decouple the 3×3 convolution in the Bottleneck of the Res2Net network at multiple scales. In the present invention, after the 1×1 convolution, the feature map is divided into 4 sub-feature maps with the same spatial size but 1 / 4 of the number of channels, corresponding to 4 convolutional branches (the total number of convolutional branches is 4). Each convolutional branch has a corresponding sub-feature map. Except for the first convolutional branch, the second to fourth convolutional branches have corresponding 3×3 convolutions. Each time a 3×3 convolution is performed, a feature extraction is carried out, thereby changing the receptive field of the branch. And each output will be used as the next sub-feature map. s is 4), and each convolutional branch has a corresponding sub-feature map . Except (i.e., the first convolutional branch), to (i.e., the second to fourth convolutional branches) have corresponding 3×3 convolutions. Each time a 3×3 convolution is performed, a feature extraction is carried out, thereby changing the receptive field of the branch. And each output will be used as the next sub-feature map The input. Each convolutional branch and the sub-feature map The corresponding relationship between F can be expressed by the following formula: ; Through the above convolutional branch structure, each convolutional operation can receive information from different subsets. Based on the combined effect, the output features contain rich receptive field sizes of different scales, effectively improving the network's ability to represent multi-scale features.

[0056] Furthermore, in the present invention, the core of the coordinate attention mechanism is to encode position information into channel attention through pooling operations. Figure 3 Schematic diagram of the structure of the coordinate attention mechanism provided by the present invention, which can be referred to Figure 3 As shown, the input feature map , has a dimension of C×W×H. Subsequently, 2 pooling kernels ( h , 1), (1, w ) are used to perform pooling on different channels of the input feature map along the horizontal direction (i.e., X-direction average pooling X AvgPool2d) and the vertical direction (i.e., Y-direction average pooling Y AvgPool) respectively. Among them, the feature pooling process of the c channel with a height of h is as follows: ; Among them, is the feature value on the h layer and the c channel; W is the width of the feature map; is the index along the width direction. The feature pooling process of the c channel with a width of w is as follows: ; Among them, is the feature value on the w layer and the c channel; H is the height of the feature map; is the index along the height direction.

[0057] The operations in the above embodiments can obtain the horizontal and vertical perception feature maps, realizing the embedding of coordinate information. Concatenate the above feature maps (Concat), and then use the 1×1 convolutional transformation function F1 to perform a transformation operation on it: ; Among them, f is the output feature; is a non-linear activation function; [ ] is a concatenation operation; F 1 is a convolution transformation function; and are feature maps after horizontal average pooling and vertical average pooling respectively. f is split (Split) along the spatial dimension into 2 independent tensors and ; Subsequently, 2 1×1 convolution transformation functions and are used to perform transformation operations on them to make them consistent with the input dimension, obtaining the attention weights: ; Among them, is the attention weight of the independent tensor ; is the Sigmoid activation function.

[0058] ; Among them, is the attention weight of the independent tensor ;

[0059] Finally, the input feature map is multiplied by the attention weights generated by the coordinate attention mechanism to obtain the final output of the coordinate attention mechanism: ; Among them, is the final output pixel point; and are the indices along the width and height directions respectively; is the attention weight of the independent tensor in the height direction; is the attention weight of the independent tensor in the width direction.

[0060] In the present invention, through the cascade of multiple convolutional branches, the convolution operation of each branch can gradually expand the receptive field. In the residual block of Res2Net, since the subsequent branches receive the output of the previous branches, it is equivalent to processing the input features at different scales, so that the network can capture a larger range of context information. Moreover, the combination of multiple convolutional branches enables the network to learn richer and more complex feature representations. By fusing the outputs of different branches, this enhanced feature representation ability helps the model better distinguish different fish feeding behavior intensities and improve the generalization ability and environmental adaptability of the model.

[0061] Based on the above embodiments, the feature fusion residual connection module is specifically used for: Perform a linear transformation on the visual sample features to obtain the linearly transformed visual sample features; Perform a linear transformation on the sound sample features to obtain the linearly transformed sound sample features; Fuse the linearly transformed visual sample features and the linearly transformed sound sample features to obtain a preliminary linearly fused feature; Perform a residual linear transformation on the visual sample features and the sound sample features respectively to obtain the visually sample features after residual linear transformation and the sound sample features after residual linear transformation; Fuse the preliminary linearly fused feature, the visually sample features after residual linear transformation, and the sound sample features after residual linear transformation to obtain the multi-modal fused sample features.

[0062] In the present invention, to improve the feature fusion efficiency, by performing a linear transformation on the input features (i.e., visual sample features and sound sample features), and then adding the linearly transformed features, a preliminary linearly fused feature is obtained. At the same time, a residual connection is introduced to map the original features to the output space, effectively avoiding information loss and alleviating the problem of gradient disappearance.

[0063] Figure 4 For the structural schematic diagram of the feature fusion residual connection module provided by the present invention, reference can be made to Figure 4 As shown, in the present invention, on the basis of performing a preliminary linear transformation and fusion on the input features and , a residual path is introduced in parallel. For the residual linear transformation of the input feature , its action mechanism follows the standard residual linear transformation formula: ; where is the original input feature (i.e., visual sample feature or sound sample feature), W represents the weight matrix unique to this layer, b is the corresponding bias vector. When the input feature passes through the residual linear transformation, according to the above residual linear transformation formula, a feature tensor that is re-encoded and mines potential feature information is output, while retaining the dimensional characteristics suitable for subsequent fusion steps. Similarly, for the input feature , the above-mentioned similar residual linear transformation operation is performed, so that the input feature combines its own weight matrix and bias vector, and outputs the transformed feature according to the linear transformation rule, ensuring that its dimension matches the subsequent steps.

[0064] In the present invention, the extracted visual features and sound features are passed through a convolutional module to obtain feature vectors of a unified length of 512. Then, the feature vectors obtained by convolution are fused, and the fused feature vectors are then passed through a fully connected layer to obtain the potential information between corresponding modalities. Finally, the final prediction value is obtained through the fully connected layer, and the prediction value is limited between 0 and 3, corresponding to feeding intensities: strong, medium, weak, and none, respectively.

[0065] The multi-modal feeding intensity detection model provided by the present invention has an identification accuracy rate of 93%, which is 7% higher than the original UMT (Unified Multi-modal Transformers) model, and the number of parameters is reduced by 25.41%. Compared with the single audio and video modalities, it is improved by 7% and 6.5% respectively, providing strong technical support for precise feeding, helping to reduce breeding costs and water pollution, and providing technical support for the subsequent development of intelligent feeding machines. Moreover, by fusing video and audio modalities and combining an improved feature extraction network and feature fusion strategy, the deficiencies of single modalities in fish feeding behavior recognition are made up for, the identification accuracy rate of the model for fish feeding behavior is improved, and the influence of environmental factors on the identification results is reduced.

[0066] Based on the above embodiments, after inputting the fish group feeding target video data and the fish group feeding target audio data into the multi-modal feeding intensity detection model and obtaining the fish group feeding intensity information output by the multi-modal feeding intensity detection model, the method further includes: Adjusting the current fish group bait feeding strategy according to the fish group feeding intensity information.

[0067] In the present invention, after obtaining the fish group feeding intensity information, if the fish group feeding intensity is strong, it indicates that the current fish group has a high demand for bait, and the bait may be more in line with the taste and nutritional needs of the fish group. At this time, it can be considered to appropriately increase the feeding amount of the bait to meet the needs of the fish group for rapid growth. However, caution should also be exercised when increasing the feeding amount to avoid water quality deterioration caused by overfeeding. At the same time, the current bait type and feeding frequency can be maintained because the existing feeding method has been able to stimulate a strong feeding desire in the fish group.

[0068] When the fish group feeding intensity is at a medium level, it indicates that the current feeding situation can basically meet the needs of the fish group, but there is still some room for optimization. The current feeding amount can be maintained, or the nutritional composition of the bait can be fine-tuned according to the growth stage and breeding objectives of the fish group, such as increasing or decreasing the content of certain proteins and vitamins, to promote the healthier growth of the fish group. In addition, the feeding frequency can also be appropriately adjusted to make the supply of bait more uniform.

[0069] If the feeding intensity of the fish school is weak, it may mean that the palatability of the bait is poor, the feeding amount is too much or too little, the feeding time is inappropriate, or factors such as water quality affect the appetite of the fish school. At this time, various adjustment measures need to be taken, such as reducing the feeding amount of the bait to avoid water pollution caused by leftover bait; changing the type of bait and selecting a bait that is more suitable for the taste and digestion and absorption of the fish school; adjusting the feeding time and choosing a period when the fish school is more active for feeding; checking the water quality conditions and taking measures to improve the water quality if necessary.

[0070] If no feeding behavior of the fish school is detected, it is necessary to check the quality of the bait and the feeding method to confirm whether there are problems such as bait spoilage and improper feeding position. At the same time, the water quality should be comprehensively detected to check whether there are situations such as excessive harmful substances and too low dissolved oxygen. According to the inspection results, corresponding solutions should be taken, such as replacing fresh bait, adjusting the feeding position, and improving the water quality.

[0071] In the present invention, an intelligent control system can be used to input the fish school feeding intensity information into a preset algorithm model, and the intelligent control system automatically calculates the optimal bait feeding strategy and controls the feeding equipment to make corresponding adjustments. This method can achieve precise feeding, reduce the interference of human factors, and improve the breeding efficiency and scientificity. For example, the intelligent control system can dynamically adjust the feeding amount according to the real-time monitored feeding intensity data to ensure that the bait supply is always in the best state.

[0072] The multi-modal fish school feeding intensity detection system provided by the present invention will be described below. The multi-modal fish school feeding intensity detection system described below can be mutually corresponding and referred to the multi-modal fish school feeding intensity detection method described above.

[0073] Figure 5 For the structural schematic diagram of the multi-modal fish school feeding intensity detection system provided by the present invention, reference can be made to Figure 5 As shown, the present invention provides a multi-modal fish school feeding intensity detection system, including a multi-modal data acquisition module 501 and a fish school feeding intensity detection module 502. Among them, the multi-modal data acquisition module 501 is used to obtain the fish school feeding target video data and the fish school feeding target audio data corresponding to the time period of the fish school feeding target video data; the fish school feeding intensity detection module 502 is used to input the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain the fish school feeding intensity information output by the multi-modal feeding intensity detection model; among them, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module, and a fully connected layer.

[0074] In the present invention, the multimodal data acquisition module 501 captures videos in the environment where fish are feeding through specific video capture devices (such as cameras), thereby obtaining video data that can reflect the feeding situation of the fish school. These video data record various behavioral performances of the fish school during the feeding process, such as the swimming trajectories and feeding actions of the fish school, and are important bases for subsequent analysis of the feeding behavior of the fish school.

[0075] Meanwhile, the multimodal data acquisition module 501 uses an audio acquisition device (such as an underwater microphone) to synchronously acquire audio data of the fish school feeding during the same time period as the above-mentioned video data. Fishes produce various sounds during the feeding process, such as the sounds of biting bait and body collisions. These audio data contain the acoustic characteristics of the feeding behavior of the fish school and can reflect the feeding situation of the fish school from another perspective. It should be noted that the audio data of the fish school feeding obtained in the present invention is spectrogram data. Figure 6 FIG. is a schematic diagram of the acquisition scenario of the feeding behavior of the fish school in the aquaculture area provided by the present invention. The process of acquiring the feeding behavior of the fish school through the video acquisition device and the audio acquisition device can be referred to Figure 6 as shown.

[0076] Furthermore, the fish school feeding intensity detection module 502 inputs the obtained fish school feeding target video data and the corresponding period of audio data into a pre-constructed multimodal feeding intensity detection model. The multimodal feeding intensity detection model is a model that comprehensively utilizes two modalities of video and audio information for processing, and can detect the feeding intensity of the fish school more comprehensively and accurately. After being processed and analyzed by the multimodal feeding intensity detection model, information about the feeding intensity of the fish school will be output. These information can intuitively reflect the feeding activity level of the fish school during the current time period. For example, different levels such as strong, medium, weak, and none of the feeding intensity provide a reference basis for subsequent aquaculture management (such as precise feeding).

[0077] In the present invention, it can be referred to Figure 6 as shown, the video acquisition device (camera) and the audio acquisition device (hydrophone) are connected to a computer through a network cable or a wireless network. Among them, the video acquisition device and the audio acquisition device can collect data in real time under the control of the computer. The video acquisition device collects the feeding changes on the water surface during the fish school feeding in real time; the audio acquisition device collects the sound changes in the water during the fish school feeding in real time and transmits the signal to the computer through a network cable. In one embodiment, a light source (lighting device) is used to supplement light for the video acquisition device to improve the image quality of the video data. Figure 7 FIG. is a schematic diagram of the detection results corresponding to different feeding behavior intensity labels provided by the present invention. The detection results of the feeding behavior intensity of the fish school obtained in real time according to the data collected by the above devices and based on the trained model can be referred to Figure 7 as shown.

[0078] The multi-modal fish school feeding intensity detection system provided by the present invention obtains the fish school feeding target video and the audio data in the corresponding period, and then inputs them into a convolutional neural network with a Res2Net network, a RepViT network, a feature fusion residual connection module and a fully connected layer, and which has been trained with sample multi-modal data, so as to obtain more accurate fish school feeding intensity information.

[0079] The system provided by the embodiments of the present invention is used to execute the above method embodiments. For the specific process and detailed content, please refer to the above embodiments and will not be elaborated here.

[0080] Figure 8 The structural schematic diagram of the electronic device provided by the present invention is as Figure 8 shown. The electronic device may include: a processor 801, a communication interface 802, a memory 803, and a communication bus 804. Among them, the processor 801, the communication interface 802, and the memory 803 complete mutual communication through the communication bus 804. The processor 801 can call the logic instructions in the memory 803 to execute the multi-modal fish school feeding intensity detection method, and the method includes: obtaining the fish school feeding target video data and the fish school feeding target audio data in the corresponding period of the fish school feeding target video data; inputting the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain the fish school feeding intensity information output by the multi-modal feeding intensity detection model; wherein, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module and a fully connected layer.

[0081] In addition, when the logical instructions in the aforementioned memory 803 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0082] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multi-modal fish school feeding intensity detection method provided by the above-mentioned various methods. The method includes: obtaining fish school feeding target video data and the fish school feeding target audio data corresponding to the time period of the fish school feeding target video data; inputting the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain the fish school feeding intensity information output by the multi-modal feeding intensity detection model; wherein, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module, and a fully connected layer.

[0083] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the multi-modal fish school feeding intensity detection method provided in the above-mentioned various embodiments. The method includes: obtaining video data of the fish school feeding target and audio data of the fish school feeding target corresponding to the time period of the video data of the fish school feeding target; inputting the video data of the fish school feeding target and the audio data of the fish school feeding target into a multi-modal feeding intensity detection model to obtain fish school feeding intensity information output by the multi-modal feeding intensity detection model; wherein, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module, and a fully connected layer.

[0084] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative effort.

[0085] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A multi-modal fish school feeding intensity detection method, characterized in that, Including: Obtaining fish school feeding target video data and fish school feeding target audio data corresponding to the time period of the fish school feeding target video data; Inputting the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain fish school feeding intensity information output by the multi-modal feeding intensity detection model; wherein, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module, and a fully connected layer.

2. The multimodal fish school feeding intensity detection method according to claim 1, wherein The multi-modal feeding intensity detection model is obtained through the following steps: Obtaining fish school feeding historical data, wherein the fish school feeding historical data includes the fish school feeding sample video data generated in a historical time period and the fish school feeding sample audio data corresponding to the historical time period of the fish school feeding sample video data; Based on a preset time window, performing time-axis alignment cutting on the fish school feeding historical data to obtain a fish school feeding behavior video data set and a fish school feeding behavior audio data set; Based on the fish school feeding behavior intensity categories in the fish school feeding behavior video data set or the fish school feeding behavior audio data set, marking corresponding fish school feeding behavior intensity labels for the sample data in the fish school feeding behavior video data set and the fish school feeding behavior audio data set to obtain a training sample set; According to the training sample set, training the convolutional neural network to obtain the multi-modal feeding intensity detection model.

3. The multi-modal fish school feeding intensity detection method according to claim 2, wherein, A coordinate attention mechanism is set in the Res2Net network for extracting features from the fish school feeding sample audio data to obtain sound sample features; The RepViT network is used for extracting features from the fish school feeding sample video data to obtain visual sample features; The feature fusion residual connection module is used for linearly transforming and residually fusing the visual sample features and the sound sample features to obtain multi-modal fusion sample features; The fully connected layer is used for obtaining corresponding fish school feeding behavior intensity prediction probabilities according to the multi-modal fusion sample features, and adjusting the model parameters of the convolutional neural network according to the loss value calculated based on the fish school feeding behavior intensity prediction probabilities.

4. The multimodal fish school feeding intensity detection method according to claim 3, wherein The Res2Net network is a hierarchical residual connection structure, and multiple convolutional branches are set in the residual block of the Res2Net network. Among them, except for the first convolutional branch, the output of the current convolutional branch is used as the input of the next convolutional branch.

5. The multimodal fish school feeding intensity detection method according to claim 3, characterized in that The feature fusion residual connection module specifically is used for: Performing a linear transformation on the visual sample features to obtain linearly transformed visual sample features; Performing a linear transformation on the sound sample features to obtain linearly transformed sound sample features; Fusing the linearly transformed visual sample features and the linearly transformed sound sample features to obtain preliminary linearly fused features; Perform residual linear transformation on the visual sample features and the audio sample features respectively to obtain the visually sample features after residual linear transformation and the audio sample features after residual linear transformation; Fuse the preliminary linear fusion features, the visually sample features after residual linear transformation, and the audio sample features after residual linear transformation to obtain the multi-modal fusion sample features.

6. The multi-modal fish school feeding intensity detection method according to claim 1, wherein After inputting the fish school feeding target video data and the fish school feeding target audio data into the multi-modal feeding intensity detection model to obtain the fish school feeding intensity information output by the multi-modal feeding intensity detection model, the method further includes: Adjust the current fish school bait feeding strategy according to the fish school feeding intensity information.

7. A multi-modal fish school feeding intensity detection system, characterized in that, Including: A multi-modal data acquisition module for acquiring fish school feeding target video data and fish school feeding target audio data corresponding to the time period of the fish school feeding target video data; A fish school feeding intensity detection module for inputting the fish school feeding target video data and the fish school feeding target audio data into a multi-modal feeding intensity detection model to obtain fish school feeding intensity information output by the multi-modal feeding intensity detection model; wherein, the multi-modal feeding intensity detection model is obtained by training a convolutional neural network with fish school feeding sample video data and fish school feeding sample audio data marked with fish school feeding behavior intensity labels; the convolutional neural network includes a Res2Net network, a RepViT network, a feature fusion residual connection module, and a fully connected layer.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the multi-modal fish school feeding intensity detection method according to any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-modal fish school feeding intensity detection method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-modal fish school feeding intensity detection method according to any one of claims 1 to 6.

Images