A method and system for evaluating fish feeding intensity by target detection and acoustic detection

Abstract

The present application relates to the technical field of aquaculture monitoring, and discloses a fish school feeding intensity evaluation method and system based on target detection and acoustic detection, which adopts an improved lightweight double-flow network model to process image and sound data of fish schools after feeding, and outputs visual features and acoustic features; a multi-scale causal feature sublimation method is used to transform the visual features, and a time-series causal filtering method is used to transform the acoustic features, so as to obtain visual and acoustic high-level semantics; spatial correlation information of the visual high-level semantics and time correlation information of the acoustic high-level semantics are extracted, and then the two are fused to obtain a fusion feature vector for dynamic evaluation of the fish school feeding intensity; and finally, the evaluation result is verified through a time-series consistency verification mechanism to obtain the feeding intensity level; the present application realizes accurate and real-time evaluation of the fish school feeding intensity in a deep water net cage by deeply mining complementary information of multi-modal data, and provides a reliable basis for scientific breeding management.

Description

Technical Field

[0001] This invention relates to the field of aquaculture monitoring technology, and in particular to a method and system for evaluating the feeding intensity of fish groups using target detection and acoustic detection. Background Technology

[0002] In deep-sea cage aquaculture, accurately understanding the feeding status of fish schools is crucial for optimizing aquaculture management, improving feed utilization, and ensuring healthy fish growth. Traditional methods for assessing fish feeding intensity have several limitations: monitoring technologies based on single sensors, such as relying solely on underwater cameras for visual monitoring, while capable of acquiring some behavioral information about fish schools, suffer from severely degraded image quality in complex environments such as turbid water and insufficient light, leading to inaccurate target detection and an inability to effectively identify the feeding behavior and intensity of fish schools; while purely acoustic monitoring can only determine the presence of fish based on sound signals, failing to comprehensively reflect the feeding status of the fish schools; furthermore, existing fusion schemes (such as video + audio) often employ post-feature fusion (such as stitching or weighted averaging), failing to fully explore the deep correlations and complementary information between different modalities, resulting in low accuracy and reliability in assessing feeding intensity.

[0003] Therefore, how to accurately and in real time assess the feeding intensity of fish schools has become a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a method and system for evaluating the feeding intensity of fish schools using target detection and acoustic detection, solving the problem of how to accurately and in real time assess the feeding intensity of fish schools.

[0005] To address the aforementioned technical problems, this invention provides a method for evaluating fish feeding intensity using target detection and acoustic detection, comprising:

[0006] Simultaneously acquire real-time image and sound data of fish in deep-sea cages after feeding;

[0007] The real-time image data and the real-time sound data are input into an improved lightweight two-stream network model for feature extraction, and visual and acoustic features are output.

[0008] The visual features are transformed into high-level visual semantics through a multi-scale causal feature sublimation method, and the acoustic features are transformed into high-level acoustic semantics through a temporal causal filtering method.

[0009] Extract the spatial association information of the visual high-level semantics and the temporal association information of the acoustic high-level semantics, and fuse the spatial association information and the temporal association information to obtain a fused feature vector;

[0010] The feeding intensity of fish swarms is dynamically evaluated based on the fused feature vectors, and the evaluation results are verified through a temporal consistency check mechanism to obtain the feeding intensity level.

[0011] Compared with the prior art, the beneficial effects of the embodiments of the present invention are as follows:

[0012] By employing multi-source data acquisition and comprehensively utilizing video streams and acoustic signals, the feeding status of fish schools can be fully reflected, improving the accuracy of evaluation. An improved lightweight dual-stream network model is used for feature extraction, effectively reducing model complexity and computational load while ensuring accuracy. This allows the model to output visual and acoustic features accurately in real time, meeting the needs of timely monitoring of fish school status in actual production. Multi-scale causal feature enhancement and temporal causal filtering methods are used to semantically transform and fuse features, further mining the deeper information hidden behind image and sound data, elevating low-level features to high-level representations with clear semantics. Dynamic evaluation of fish feeding intensity based on fused feature vectors can reflect changes in fish feeding status in real time, providing timely and accurate decision-making basis for aquaculture personnel. Simultaneously, a temporal consistency verification mechanism is used to verify the evaluation results, eliminating erroneous evaluations caused by data noise or instantaneous model errors, further improving the reliability and stability of the evaluation results and ensuring the scientific and effective nature of aquaculture decisions. Attached Figure Description

[0013] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a flowchart of a method for evaluating the feeding intensity of fish groups using target detection and acoustic detection, provided in a certain embodiment of the present invention;

[0015] Figure 2 This is a structural diagram of a fish feeding intensity evaluation system based on target detection and acoustic detection, provided in a certain embodiment of the present invention;

[0016] Figure label:

[0017] The module includes: 10. Data acquisition module; 20. Feature extraction module; 30. Feature transformation module; 40. Feature fusion module; and 50. Evaluation and verification module. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings and examples. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0020] The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art will understand the specific meaning of these terms in this invention according to the specific circumstances. It should be noted in the description of this invention that, unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by those skilled in the art. The terminology used in this specification is for the purpose of describing specific embodiments only and is not intended to limit the invention; those skilled in the art will understand the specific meaning of these terms in this invention according to the specific circumstances.

[0021] In one embodiment, such as Figure 1 As shown, the first aspect of the present invention provides a method for evaluating the feeding intensity of fish schools based on target detection and acoustic detection, comprising:

[0022] S1. Simultaneously acquire real-time image and sound data of fish in deep-water cages after feeding;

[0023] Specifically, 4K HDR underwater cameras (120° wide-angle) are deployed on the sidewalls (30° tilt) and top of the deep-water cage to capture video streams after feeding at 30fps as real-time image data. The equipment has a low-light sensitivity of 0.001 Lux, enabling clear capture of the spatial distribution of fish (e.g., their aggregation range in the feeding area), individual movement (e.g., swimming direction, whether they rush towards the bait), and group dynamics (e.g., whether they form feeding swarms) in deep-water, low-light environments. A directional, high-sensitivity hydrophone array (Resonance) is deployed 1m below the feeding area. The TC4013 (frequency response 1Hz-180kHz) collects the sounds of fish feeding and colliding in the 1kHz-20kHz frequency band after baiting (such as the sound of fish mouths opening and closing and hitting the bait, transient sounds of fish bodies colliding, etc.) in sync with real-time image data as real-time sound data. It should be noted that other types and deployment methods can be used for the camera and sound acquisition equipment, but their deployment location must ensure that it can cover the core area of ​​baiting and the surrounding swimming range, avoid blind spots, and be able to collect the sounds of fish feeding.

[0024] Next, addressing the fogging issue caused by scattering from suspended particles in underwater videos, image enhancement-based defogging algorithms (such as Retinex algorithm and wavelet transform), image restoration-based defogging algorithms (such as dark channel prior), or deep learning-based defogging algorithms (such as ESA algorithm and DenseAttention) are used to preprocess real-time image data, yielding processed image data. Then, an adaptive notch filter (ANF) is used to denoise the real-time audio data, suppressing specific frequency noise (such as the fixed frequency noise of the net cage pump). The ANF can monitor noise frequencies in real time and dynamically adjust filtering parameters, filtering out interference while preserving details of feeding collision sounds in the 1kHz-20kHz frequency band (such as the time-frequency characteristics of transient impact sounds), thus obtaining processed audio data. It should be noted that the above processing methods for image and audio data are only one example; other preprocessing procedures can also be used to achieve the corresponding defogging or noise reduction objectives.

[0025] Finally, the image processing data and sound processing data are spatiotemporally aligned using the Dynamic Time Warping (DTW) algorithm and the Time Difference of Arrival (TDA) algorithm to facilitate subsequent feature extraction. The core of the DTW algorithm is to find the optimal time correspondence by elastically stretching or compressing two time series, thus solving the timestamp misalignment problem between visual and acoustic streams caused by acquisition device response delays and differences in data sampling frequencies. The specific implementation steps are as follows:

[0026] First, time series features are extracted. For visual streams (i.e., image processing data), key time markers are extracted from video frames, such as frames where fish show obvious feeding behavior or frames where fish density changes abruptly, to form a visual time series. For acoustic streams (i.e., acoustic processing data), key time markers are extracted from audio signals, such as moments when feeding collision sounds burst out densely or when the acoustic event rate exceeds a threshold, to form an acoustic time series.

[0027] Next, the absolute value of the time difference for each element in the visual time series and the acoustic time series is calculated to construct a distance matrix.

[0028] Then, based on the distance matrix, a dynamic programming algorithm is used to find the optimal path from a point in the matrix to any other point. During the path planning process, the path constraint is that the path can only move to the right, down, or down to the right from the current position (to ensure the monotonicity of the time series and prevent reverse alignment). The constraint is to find the correspondence that minimizes the overall time deviation between the two sequences as the cumulative distance minimization constraint, and then search for the optimal alignment path.

[0029] Subsequently, based on the optimal alignment path, the acoustic stream timestamps are mapped to the visual stream time axis (or vice versa), so that each frame of visual data corresponds to a segment of synchronized acoustic data, thereby achieving timestamp calibration; for example, the acoustic signal is divided into segments corresponding to video frames one by one according to the alignment path, ensuring that subsequent feature extraction is synchronized in the time dimension.

[0030] The Time Difference of Arrival (TDOA) algorithm calculates the time difference between the arrival of a sound source signal at different hydrophones, combines this with the spatial coordinates of the hydrophone array to locate the three-dimensional position of the sound source, and then establishes a mapping with the spatial coordinates of the visual target to achieve cross-modal alignment. The specific steps are as follows:

[0031] First, the hydrophone array is deployed and its coordinates are calibrated. Three hydrophones are deployed below the feeding area to form a planar / three-dimensional array (such as an equilateral triangle distribution). The three-dimensional coordinates of each hydrophone are calibrated using a laser rangefinder, and a spatial coordinate system is established with a fixed point at the bottom of the net cage (such as the center point, the corner of the net cage, etc.) as the origin.

[0032] Then, TDOA calculation is performed. The arrival times of the same feeding collision sound event are extracted from different hydrophones based on the sound processing data, and the time difference is calculated according to the order of arrival.

[0033] Next, the sound source is located in three dimensions. The product of the sound wave propagation speed in water and the time difference is used as the distance difference. A spatial positioning equation is constructed based on the principle that the difference between the spatial distance between the sound source coordinates and the spatial coordinates of the two hydrophones is equal to the corresponding distance difference. Solving this equation yields the three-dimensional coordinates of the sound source, which is the spatial location of the feeding collision sound.

[0034] Subsequently, distortion correction was performed on the video frames captured by the underwater camera (based on the camera's intrinsic parameter matrix). The pixel coordinates of the fish target were converted into three-dimensional coordinates in the space coordinate system of the net cage through perspective transformation. That is, the camera's extrinsic parameters (rotation matrix, translation vector) were calibrated using reference objects of known size inside the net cage (such as the diameter of the feeding tube), establishing a mapping relationship between pixel coordinates and real space coordinates. For the center point of the detected fish bounding box, its corresponding three-dimensional coordinates were calculated using the extrinsic parameter matrix. Of course, other existing technologies can also be used for coordinate conversion from two-dimensional to three-dimensional, and the specific implementation process will not be detailed here.

[0035] Finally, the Euclidean distance between the three-dimensional coordinates of the sound source and the three-dimensional coordinates of each visual target is calculated, and the visual target with the smallest distance is taken as the fish associated with the sound source (if the calculated Euclidean distance is less than the preset threshold, such as 30cm, it is determined to be a target in the same spatial position), thus realizing the spatial mapping between the sound source and the fish.

[0036] This completes the spatiotemporal alignment of sound and visual data, resulting in sound-aligned data and visual-aligned data. A dynamic update mechanism can be used for the alignment of sound and vision, that is, the spatiotemporal alignment process is repeated every 100ms to adapt to the changes in the position of the sound source and visual target caused by the swimming of fish, so as to ensure the real-time performance and accuracy of cross-modal fusion.

[0037] This invention uses Time-Depth Wave (DTW) to ensure that the acoustic events used for TDOA localization and the video frames used for visual coordinate extraction are in the same time window, avoiding spatial matching errors caused by temporal misalignment (such as matching the fish body of the previous moment with the sound source of the current moment). It also uses spatial mapping to improve the temporal alignment accuracy. That is, if the spatial matching degree between the sound source and the visual target at a certain moment is extremely high (the distance is close to 0), then that moment is used as an "anchor point" to optimize the alignment path of DTW, thereby improving the stability of long sequence temporal alignment.

[0038] S2. Input the real-time image data and the real-time sound data into an improved lightweight two-stream network model for feature extraction, and output visual features and acoustic features; wherein, the improved lightweight two-stream network model includes a pruned YOLO-MS network; the pruned YOLO-MS network includes a feature extraction layer, a feature fusion layer and a detection output layer.

[0039] In one embodiment, step S2 includes:

[0040] The feature extraction layer extracts appearance and motion features from the real-time image data to obtain several multi-scale fusion feature maps.

[0041] The feature fusion layer is used to perform upsampling fusion processing and downsampling fusion processing on each of the multi-scale fusion feature maps to obtain several enhanced multi-scale feature maps;

[0042] The enhanced multi-scale feature maps are input into the detection output layer for processing, and the number of fish, orientation angle, and instantaneous speed are output as the visual features.

[0043] Specifically, this invention employs a pruned YOLO-MS network to extract features from real-time image data (which can also be considered as image data after preprocessing and spatiotemporal alignment) to obtain visual features. The pruned YOLO-MS network is a lightweight version optimized through channel pruning and layer pruning based on the original YOLO-MS network. Its core retains the classic YOLO architecture of "backbone feature extraction + neck feature fusion + head detection output," while enhancing feature fusion capabilities for underwater fish detection scenarios to extract the fish's appearance and motion features. Channel pruning uses feature importance scoring based on the fish detection task, calculating the contribution of each convolutional channel to the fish detection loss through Taylor expansion and discarding channels with a contribution below a threshold (set to 0.15). Layer pruning focuses on non-critical downsampling layers: retaining the first four downsampling layers (ensuring small target detection) and simplifying the residual connections of the last two downsampling layers to avoid redundant computation.

[0044] The feature extraction layer is a simplified version of the original YOLO-MS CSPDarknet architecture. L1 regularization is used to prune the convolutional layer channels, removing redundant residual blocks: four core residual stages are retained (originally eight), with the number of convolutional channels in each stage reduced by 30%-40%. Only feature channels sensitive to fish contours and motion edges are retained. An optical flow feature branch (such as the Farneback algorithm or a simplified version of FlowNet) is inserted at the end of the pruned backbone network. The feature extraction layer includes a dual-input processing unit and a pruned CSP fusion unit. The dual-input processing unit includes an appearance feature branch and an optical flow feature branch. The pruned CSP fusion unit consists of four cascaded CSP modules, each including a main branch and a residual branch. Channel pruning (retaining channels sensitive to fish features and removing background response channels) reduces computational cost.

[0045] The feature fusion layer is formed by pruning the FPN-PAN structure of the original YOLO-MS network and embedding appearance and motion attention fusion branches in each fusion node of the pruned FPN-PAN structure through skip connections. The feature fusion layer includes FPN upsampling fusion units and PAN downsampling fusion units, and each fusion unit has appearance and motion attention fusion branches at its fusion node.

[0046] The detection output layer is formed by pruning the prediction head of the original YOLO-MS network (reducing it from 3 to 2), retaining only the anchor boxes sensitive to fish size (suitable for fish sizes of 5-30cm), and adding a fully connected layer to the prediction head to embed the motion feature regression branch. The detection output layer includes a detection unit, a motion feature regression branch, and a post-processing unit.

[0047] This invention extracts both appearance and motion features through a feature extraction layer. A 640×640×3 RGB real-time image (containing appearance information such as the shape and color of the fish school) is input into the appearance feature branch of the dual-input processing unit. The image is downsampled to 320×320 using a 7×7 pruned convolution (channel number 3→48, original 64) to extract basic appearance features such as fish contour and texture. Simultaneously, a 640×640×2 optical flow map (obtained by calculating the optical flow field using the Farneback algorithm on two consecutive frames, containing pixel-level motion vectors reflecting the swimming trend of the fish school) is input into the optical flow feature branch of the dual-input processing unit. This map is downsampled to 320×320 using the same 7×7 pruned convolution (channel number 2→48) as the appearance feature branch to extract basic features of the motion trajectory (such as pixel displacement trends). The features obtained from these two branches are then input into the pruned CSP fusion unit, where CSP1-CSP4 modules... The main branch in each block (the feature processing process is the same for each CSP) compresses channels through 1×1 convolution, and the residual branch contains 3-9 pruned residual blocks (the number of 3×3 convolution channels in each residual block is 60% of the original) to process the input features. The processed appearance and motion features are then processed by the cross-branch fusion gate (1×1 convolution + sigmoid) set at the output of each CSP module to dynamically fuse the features of the appearance branch and the motion branch (such as the binding of fish body outline and corresponding motion trajectory), and output three multi-scale fused feature maps that simultaneously contain appearance and motion features: P3: 80×80×128 (1 / 8 downsampling, retaining the detailed features of small target fish bodies, such as the direction of the fish mouth and small movements), P4: 40×40×256 (1 / 16 downsampling, containing the aggregation pattern and movement trend of medium-scale fish groups), P5: 20×20×512 (1 / 32 downsampling, containing global fish group distribution and large-scale motion features).

[0048] The feature maps output by the feature extraction layer are sampled and fused using a feature fusion layer: The P3, P4, and P5 feature maps output by the feature extraction layer are input into the feature fusion layer. In the upsampling fusion unit of the FPN (the processing of the three feature maps is the same in both, taking P5 as an example), P5 is compressed in dimension by a 1×1 pruned convolution (the number of channels is halved after pruning, e.g., 512→256), and then upsampled to the same scale as P4 by bilinear interpolation. It is then element-wise added to the P4 feature map (after 1×1 convolution compression) to generate P4_fused; this process is repeated. This process yields P3_fused (upsampled features fused from P3 and P4_fused); in the PAN downsampling fusion unit, P3_fused is downsampled using a 3×3 pruned convolution (using depthwise separable convolution to reduce computation) and a stride of 2, and added to P4_fused (after 3×3 convolution processing) to generate P4_enhanced; this process is repeated to obtain P5_enhanced; then, bidirectional fusion is repeated twice for each scale (e.g., 1 / 32→1 / 16→1 / 8→1 / 16→1 / 32). After performing the same feature fusion process on the three multi-scale fusion feature maps, the final output consists of three enhanced multi-scale feature maps (with the same scale as the input, but with stronger feature representation capabilities): F3: 80×80×128 (enhancing the "size-instantaneous velocity" association feature of small target fish), F4: 40×40×256 (enhancing the "orientation-movement direction" association feature of medium-scale fish groups), and F5: 20×20×512 (enhancing the "distribution-overall movement trend" association feature of global fish groups).

[0049] The appearance and motion attention fusion branch processes the input features as follows: Before entering the appearance and motion attention fusion branch, the fusion node (e.g., P4_fused) has acquired two types of basic features: appearance features and motion features. The appearance features of layer P4 are passed to the fusion node P4_fused via the Neck's PAN structure as input. The motion feature layer corresponding to layer P4 is passed to the fusion node P4_fused via the Neck structure. Then, the appearance and motion attention fusion branch embedded in each fusion node (e.g., P4_fused) performs global average pooling (GAP) on the appearance and motion features in P4_fused based on the channel attention mechanism, compressing the spatial information of each channel into a statistical value. These two statistics are then concatenated, and the correlation between channels is learned through a shared fully connected layer. The Sigmoi algorithm is then used to perform the concatenation. The d activation function generates normalized channel weights, which are then multiplied channel-by-channel with the original appearance / motion feature map to dynamically enhance key channels and obtain enhanced features. Simultaneously, this branch uses a spatial attention mechanism to reduce the dimensionality of the two enhanced features processed by channel attention using 1×1 convolutions, concatenating them along the channel dimension. A spatial attention heatmap is then generated using a 3×3 convolution. Subsequently, the optical flow vector (X / Y direction) in the enhanced motion feature is normalized and its cosine similarity is calculated with the fish body contour edge (extracted using the Sobel operator) in the appearance enhancement feature to generate an alignment score map. This alignment score map is multiplied with the spatial attention heatmap to strengthen regions where the motion direction aligns with the fish body contour (e.g., the area where the fish mouth opens and closes rapidly during feeding). Finally, the generated spatial attention heatmap is multiplied pixel-by-pixel with the original fused feature map to highlight the fish's activity area, resulting in two fused features. Spatial attention can use an SE module, and channel attention can use CBAM, or other structures that achieve the above functionality.

[0050] Taking low-level → high-level upsampling fusion as an example (e.g., 1 / 32 → 1 / 16 scale), the complete process of bidirectional feature fusion and attention embedding in the feature fusion layer is illustrated: The appearance feature A_low is upsampled to a 1 / 16 scale through transposed convolution, and then element-wise added to the mid-scale appearance feature (A_mid). The motion feature M_low is upsampled to a 1 / 16 scale through bilinear interpolation, concatenated with the mid-scale motion feature (M_mid), and then fused through a 1×1 convolution to achieve upsampling and initial fusion. Then, the fused appearance (A_fu) is... The appearance (A_weighted) and motion (M_fused) features are processed through a channel attention mechanism to calculate and weight the channels separately, resulting in A_weighted and M_weighted features, which are then concatenated to generate a spatial heatmap. A spatial attention mechanism is then used to simultaneously recalibrate both features, resulting in A_att and M_att, thus enhancing the attention of the features. Finally, the attention-enhanced appearance (A_att) and motion (M_att) features are concatenated again, and a 3×3 convolution is used to generate an enhanced multi-scale feature map. Other scale transformation processes and their downsampling fusion processes follow the same principle as the above transformations and will not be elaborated upon here.

[0051] The enhanced multi-scale feature map output from the feature fusion layer is processed by the detection output layer: F3, F4, and F5 are input to the detection branch of the detection unit for processing. This detection branch includes three scales corresponding to three inputs. Each scale contains two 3×3 pruned convolutions (depthseparable convolutions, with 50% of the original number of channels) and one 1×1 output convolution. After processing, two outputs are obtained: Output 1 (bounding box and confidence score): 3×(4+1) channels (3 anchor boxes, 4 bounding box coordinates, 1 target confidence score), used to locate the fish body and calculate its size (obtained by taking the square root of the product of the bounding box width and height); and Output 2 (orientation angle): 3×1 channels, output by regression. The fish's orientation angle (0°-360°, calculated based on the directional gradient of the F4 feature map) is calculated. Simultaneously, the F3 (fine-grained motion) and F4 (medium-scale motion) feature maps are input into the motion feature regression branch, which outputs an instantaneous velocity vector (x / y direction pixel displacement / frame) through two 1×1 pruned convolutions. The number of channels is 3×2 (3 anchor boxes, x / y direction velocity). The results obtained from the detection unit and the regression branch are input into the post-processing unit for non-maximum suppression (NMS) to filter redundant boxes, retaining fish targets with a confidence level ≥0.5. Based on the DeepSort tracking algorithm, the fish trajectory ID is generated according to the bounding box and velocity vector, and the acceleration of the trajectory (rate of change of velocity between adjacent frames) is calculated. In other words, the final output of the detection output layer includes the basic detection results: an array [N,6] (where N is the number of fish targets), containing "x1,y1,x2,y2 (bounding boxes), confidence, and angle)," as well as motion features: the instantaneous velocity of each target (obtained by converting the motion branch output using pixel-centimeter calibration coefficients, in pixels / frame), and the acceleration (ax,ay) (pixels / frame) derived from the tracking trajectory. 2 The detection results output by the detection output layer are used as visual features.

[0052] This invention simultaneously captures both static appearance features and dynamic motion features of fish schools through a feature extraction layer, addressing the problem of single features being susceptible to environmental interference. The generated multi-scale fused feature map covers different spatial resolutions, adapting to scenarios with dynamically changing fish school density. A bidirectional fusion mechanism is employed: the feature fusion layer, through upsampling and downsampling, enables deep interaction between motion and appearance features at multiple scales, enhancing the representation of complex behaviors. An implicit attention mechanism is introduced during the fusion process, automatically focusing on feature regions that contribute significantly to the task and reducing redundant information interference. Furthermore, the detection output layer avoids the cumulative errors required by traditional methods that require multiple post-processing steps, improving real-time performance and accuracy.

[0053] In one embodiment, the improved lightweight two-stream network model further includes a parallel dual-branch structure network, wherein the parallel dual-branch structure network includes a MobileNetV2-CBAM branch; the MobileNetV2-CBAM branch includes a first input layer, a lightweight feature extraction layer, a CBAM attention layer, and a first output layer; wherein,

[0054] Step S2 also includes:

[0055] The real-time audio data is converted into a Mel spectrogram and input into the first input layer for normalization processing to obtain a normalized Mel spectrogram.

[0056] The time-frequency domain features of the standardized Mel spectrum are extracted by the lightweight feature extraction layer to obtain a multi-scale time-frequency feature map that characterizes the energy distribution of fish feeding sounds in multiple frequency bands.

[0057] The CBAM attention layer is used to perform feature weight enhancement processing on the multi-scale time-frequency feature map to obtain a weighted time-frequency feature map.

[0058] The weighted time-frequency feature map is compressed based on the first output layer to obtain a time-frequency feature vector as the acoustic feature; the time-frequency feature vector includes the target frequency band energy of the fish feeding sound and the time-frequency domain peak value.

[0059] Specifically, the parallel dual-branch network in this invention is designed to address the time-frequency distribution and transient characteristics of fish feeding sounds. It extracts time-frequency features through the MobileNetV2-CBAM branch and captures time-domain events through the TCN branch, achieving comprehensive coverage of acoustic features.

[0060] The MobileNetV2-CBAM branch takes Mel spectrograms as input and uses lightweight convolution and attention mechanisms to focus on the energy distribution characteristics of key frequency bands (5-15kHz) of feeding sounds. The implementation process includes:

[0061] The power spectrum of each frame of the real-time audio data (i.e., the audio data after preprocessing and spatiotemporal alignment) is obtained by performing a Fourier transform. This power spectrum is then converted into a Mel spectrum through a 40-Melb filter bank and the logarithm is taken to obtain a log-Melb spectrum. The obtained Mel spectrum is then input into the first input layer of the MobileNetV2-CBAM branch, which standardizes the Mel spectrum based on the statistical mean and standard deviation of the cage acoustic environment to map the energy values ​​in the graph to the [-1,1] interval, thereby enhancing the network's generalization ability and outputting a standardized Mel spectrum (128×128×1).

[0062] The lightweight feature extraction layer is designed based on the inverted residual structure of MobileNetV2 to extract multi-scale features in the time-frequency domain. It can reduce the computational cost through pruning and lightweight design. It includes 5 pruned "bottleneck layers". Each bottleneck layer consists of three steps: "dimensionality increase-feature extraction-dimensionality reduction". Bottleneck layer 1: The input normalized Mel spectrogram is increased in dimensionality by 1×1 convolution, so that its input channels increase from 1 to 16 (the dimensionality is increased by 1×1 convolution to improve the feature representation ability). Then, 3×3 depthwise separable convolution is used to extract features from the dimensionality increase result. Channel-by-channel convolution (without changing the number of channels) with a stride of 1 (preserving the time dimension) is used to extract basic features in the low-frequency band (0-5kHz). Finally, 1×1 convolution is used to reduce the dimensionality of the extracted basic features, that is, channels 16 to 8. (Compressing dimensions to reduce redundancy), and finally activated by BatchNorm and Swish; the processing of bottleneck layers 2-5 is the same as that of bottleneck layer 1, but after bottleneck layers 2-5, the number of channels increases sequentially (8→16→32→64→128), and the receptive field of the 3×3 depth separable convolution gradually expands (from 3×3→7×7). The composite features of the mid-frequency band (5-10kHz), high-frequency band (10-15kHz) and cross-frequency band (i.e., the energy peak of feeding sound at around 10kHz) are extracted respectively. Finally, a multi-scale time-frequency feature map (32×32×128) is output to characterize the energy distribution of fish feeding sound in multiple frequency bands, where 32×32 is the spatial size (the size after 4 convolutions of stride 1 for 128×128), and 128 is the number of channels (containing energy distribution features of different frequency bands).

[0063] The CBAM attention layer dynamically enhances the feature weights of key frequency bands and time regions of feeding sounds while suppressing noise interference. It includes channel attention units and spatial attention units. In the channel attention unit, the input multi-scale time-frequency feature map (32×32×128) undergoes global average pooling (GAP) to obtain a 1×1×128 channel descriptor. This descriptor is then compressed and expanded using two 1×1 convolutional layers (128→32→128), followed by Sigmoid activation to generate channel weights (1×1×128). Higher weight values ​​indicate a stronger correlation between the corresponding frequency band (e.g., 10kHz) and the feeding sound. Finally, the channel weights are multiplied by the input feature map to obtain the enhanced 5- Feature maps of the 15kHz key frequency band are generated. In the spatial attention unit, channel-weighted feature maps are averaged by channel dimension to obtain a 32×32×1 spatial descriptor. Then, spatial weights (32×32×1) are generated by 3×3 convolution (1→1) and Sigmoid activation. The higher the weight value, the more important the feature of the time frame (such as the continuous collision sound region). Finally, the spatial weights are multiplied by the channel-weighted feature maps spatially to enhance the time regions with dense acoustic events. The output is a weighted time-frequency feature map (32×32×128) after attention weighting, in which the features of the target frequency band (5-15kHz) and the high-activity time region are significantly enhanced.

[0064] The weighted time-frequency feature map is input into the first output layer for global average pooling (GAP), which means taking the average of the 32×32 spatial dimensions of each channel and outputting a 1×1×128 vector, i.e., a 128-dimensional time-frequency feature vector, as an acoustic feature. This vector contains information such as the target frequency band energy of the fish feeding sound and the peak value in the time-frequency domain (e.g., the energy proportion of the 10kHz frequency band and the time distribution of continuous collision sound).

[0065] This invention converts real-time sound data into Mel spectrograms, which can simulate the nonlinear perception of frequency by the human ear, highlighting key frequency bands in speech or bioacoustic signals while reducing noise interference. It extracts multi-scale time-frequency features through lightweight convolutional layers, reducing computational load while preserving local and global features, making it suitable for real-time processing on edge devices. It dynamically allocates weights in the time-frequency domain, highlighting key frequency bands and time-frequency peaks of feeding sounds while suppressing background noise or irrelevant frequency bands. Finally, it compresses the weighted feature map into a vector, retaining core information such as the energy proportion of key frequency bands and peak positions, reducing the complexity of subsequent classification or detection tasks.

[0066] In one embodiment, the parallel dual-branch network further includes a TCN branch, which comprises a second input layer, a temporal convolutional layer, and a second output layer; wherein,

[0067] Step S2 also includes:

[0068] The real-time sound data is divided into frames by the second input layer to extract the zero-crossing rate (used to characterize the intensity of the feeding signal fluctuations), the spectral entropy (used to characterize the complexity of the feeding frequency distribution), and the MFCC (used to characterize the voiceprint features) from each frame of signal. The frames are then spliced ​​together according to time steps to obtain a time-series feature matrix.

[0069] The temporal convolutional layer is used to extract transient events of fish feeding collisions from the temporal feature matrix to obtain a feeding collision temporal feature map; each channel in the feeding collision temporal feature map corresponds to a transient event pattern of fish feeding collisions.

[0070] The feeding collision temporal feature map is compressed based on the second output layer to obtain a temporal feature vector as the acoustic feature; the temporal feature vector includes the frequency, intensity and duration of the fish feeding collision transient event.

[0071] Specifically, the TCN branch captures transient events (such as single collisions and consecutive collisions) of fish feeding collisions from the temporal features (zero-crossing rate, spectral entropy, MFCC) of real-time sound data through temporal convolution. The implementation process includes:

[0072] Real-time audio data (i.e., preprocessed and spatiotemporally aligned audio data) is input to the second input layer, segmented into 20ms / frames, with a frame shift of 10ms (50% overlap). A Hanning window is added to each frame to extract three types of temporal features: Zero-crossing rate (ZCR): the number of zero-crossing points in each frame (reflecting the intensity of feeding signal fluctuations; ZCR > 0.3 during feeding); Spectral entropy (SE): the information entropy of the power spectrum in each frame (reflecting the complexity of the feeding frequency distribution; SE during feeding). >1.2) MFCC: Extract 13-dimensional Mel-frequency cepstral coefficients (including static coefficients + first-order difference, totaling 26 dimensions, used to characterize voiceprint features); then concatenate these three types of features into a 3×50 matrix (50 time steps, covering 500ms audio), where 3 corresponds to a simplified combination of ZCR (1 dimension) + SE (1 dimension) + MFCC (taking the first 1 dimension of key coefficients from 26 dimensions), and finally outputs a temporal feature matrix (3×50), with each column corresponding to a 3-dimensional temporal feature of one time step.

[0073] The temporal convolutional layer comprises three cascaded dilated convolutional blocks, each including dilated convolution, residual connections, and an activation function. This invention utilizes the dilated convolutional blocks in the temporal convolutional layer to capture transient events (single collisions, consecutive collision sequences) of fish feeding collisions at different time scales within the temporal feature matrix. Specifically, convolutional block 1 includes a 1D dilated convolution: kernel size 3, input channels 3, output channels 32, dilation rate 1 (receptive field for 3 time steps), capturing single collision transients (such as sudden ZCR peaks) within 50-150ms; residual connections: the input (3×50) and convolutional output (32×50) are added after matching channels via a 1×1 convolution; LeakyReLU activation: enhances nonlinear representation. Convolutional block 2 includes a 1D dilated convolution: kernel size 3, input channels 32, output channels 64, dilation rate 2 (receptive field 7 time steps), capturing continuous collisions within 100-300ms (e.g., interval features of 2-3 collisions); residual connections and activation: same as convolutional block 1. Convolutional block 3 includes a 1D dilated convolution: kernel size 3, input channels 64, output channels 128, dilation rate 4 (receptive field 15 time steps), capturing collision rhythms within 200-500ms (e.g., frequency changes of dense collisions); residual connections and activation: same as before. After processing by the three dilated convolutional blocks, the temporal convolutional layer finally outputs a 128×50 feeding collision temporal feature map. Each channel in this feature map corresponds to a transient event pattern of fish feeding collisions (e.g., high-frequency single collision, low-frequency continuous collision), and the time dimension retains the dynamic changes of 50 time steps.

[0074] The feeding collision temporal feature map output from the temporal convolutional layer is input into the second output layer and subjected to temporal global pooling, i.e., averaging over 50 time steps for each channel to obtain a 1×128 vector. This compresses the feeding collision temporal feature map into a fixed-dimensional temporal transient feature vector, resulting in a 128-dimensional temporal feature vector. This vector contains information such as the frequency (times / second), intensity (peak energy), and duration (ms) of the fish feeding collision transient events (e.g., the number of collisions within 500ms and the average interval). The time-frequency feature vector output from the MobileNetV2-CBAM branch and the temporal feature vector output from the TCN branch constitute the acoustic features in the real-time sound data.

[0075] This invention improves the ability to recognize transient events of fish feeding collisions (such as clicks and pulses of fish feeding sounds) by extracting zero-crossing rate, spectral entropy, and MFCC, which comprehensively characterize the temporal fluctuation (zero-crossing rate), frequency domain complexity (spectral entropy), and spectral envelope (MFCC) of sound. After framing, the features are spliced ​​according to time steps to preserve the temporal dynamics of sound, making it suitable for capturing non-stationary signals. Transient patterns (such as rapid energy changes and specific frequency jumps) in the temporal features are specifically extracted through temporal convolutional layers, enhancing the sensitivity to short-lived signals. The feeding collision temporal feature map is compressed into a vector, preserving the core information such as the frequency, intensity, and duration of the fish feeding collision transient events, reducing the computational complexity of subsequent tasks.

[0076] S3. The visual features are transformed into high-level visual semantics through multi-scale causal feature sublimation method, and the acoustic features are transformed into high-level acoustic semantics through temporal causal filtering method.

[0077] In one embodiment, the step of transforming the visual features into high-level visual semantics using a multi-scale causal feature sublimation method includes:

[0078] The real-time image data is decomposed into several scale grids, and the spatial distribution statistics of the visual features are calculated on each scale grid.

[0079] Based on the visual features, target acoustic features are selected from the acoustic features, and the visual features and the target acoustic features are determined in each of the scale grids. of Causality strength, and obtain the causality strength score;

[0080] Based on the spatial distribution statistics and the causal strength score, the number is transformed into a group size causal semantic factor, the orientation angle is transformed into a motion coordination causal semantic factor, and the instantaneous velocity is transformed into a motion response causal semantic factor.

[0081] The visual high-level semantics are obtained by combining the group-scale causal semantic factors, motion-co-causal semantic factors, and action-response causal semantic factors in each scale grid.

[0082] Specifically, this invention employs a multi-scale causal feature enhancement technique to extract high-level semantic factors that are causally related to feeding behavior from the original visual features (number of fish, orientation angle, instantaneous speed). This technique captures information at different scales (from global group behavior to local individual actions) through multi-scale decomposition and strengthens the visual region features that truly cause or influence acoustic events (such as feeding sounds) through causal correlation analysis. This suppresses the interference of noise and non-causal correlation, upgrading the visual features from "simply describing the state of the fish school" to causal semantics that "can predict acoustic feeding events".

[0083] Real-time image data is decomposed into regions of different spatial scales, resulting in several scale grids. The entire image is treated as a unit, and the overall fish population distribution (e.g., overall density, movement trends) is analyzed to obtain a macro-scale grid (global). The image is further divided into several grids (e.g., 3×3 or 4×4), and regional clustering patterns are analyzed to obtain a meso-scale grid (region). Key local regions are focused on through saliency detection or edge recognition to obtain a micro-scale grid (local). At each scale, spatial distribution statistics of visual features are calculated, with the fish population density being the quotient of the number of fish within the grid and the grid area. The mean orientation angle of all fish within the grid is calculated (to reflect the overall orientation of the group), and the variance is calculated as the consistency variance. The mean swimming speed of all fish within the grid is calculated (to reflect the overall activity level), and the variance of speed changes between adjacent frames is calculated as the abrupt change variance.

[0084] Statistical analysis or modeling tools are used to quantify the correlation between each visual feature (and feature combinations) and acoustic feeding events to select target acoustic features from the acoustic features. This involves calculating the correlation coefficient between a single visual feature (e.g., fish population size) and the feeding event label (0 = no feeding, 1 = feeding) in the acoustic features, and using acoustic features with correlation coefficients greater than a preset coefficient (preferably 0.5-0.8) as initial selection results. Alternatively, a simple prediction model (e.g., random forest, XGBoost) can be constructed. This model takes feeding events in the acoustic features as input and visual features as output. The model automatically outputs an importance score for each input feature, and acoustic features with scores greater than a preset score (preferably 0.5-0.8) are used as initial selection results. The initial selection results are then verified using causal inference tools (e.g., causal graphs, Do-calculus) and combined with biological knowledge of fish feeding to further filter out features that are "statistically correlated but have no practical significance," thus obtaining the target acoustic features. Next, the visual features and target acoustic features are determined. of Causal strength is defined as the mutual information between the visual features at time t and the target acoustic features at time t+Δt within each grid scale. The distribution of data in the mutual information calculation process can be determined using kernel density estimation (KDE), where Δt is the lag time (e.g., -0.5s to +0.5s, negative representing visual changes first, followed by acoustic changes), covering a reasonable interval before and after sound generation. The maximum mutual information when Δt < 0 (i.e., retaining only strong correlations where visual leads acoustic, excluding non-causal random correlations) is then taken as the causal strength score. This score reflects the maximum predictive power of visual features for acoustic events at that scale.

[0085] For macro-scale grids, the product of fish density and causal strength score is used as the group size causal semantic factor, which can intuitively reflect the number and aggregation of fish. For meso-scale grids, the product of consistency variance and causal strength score is used as the motion coordination causal semantic factor, which represents the overall movement direction of the fish. For micro-scale grids, the product of mutation variance and causal strength score is used as the action response causal semantic factor, which reflects the reaction speed and action intensity of the fish to external stimuli (such as feeding).

[0086] Finally, the causal intensity scores at each scale are normalized using the softmax function to obtain the scale weights of the corresponding factors. For each scale grid, the "group size causality, motion coordination causality, and action response causality" of the grid at the three scales are weighted and summed according to the scale weights, and combined into a vector to obtain the visual high-level semantics.

[0087] This invention employs spatial multi-scale decomposition to comprehensively cover all levels of fish behavior, from micro to macro, and combines it with time-lag causal analysis to deeply explore the underlying logic of fish behavior. It also captures the hierarchical structure and causal temporal relationships of visual events. Through multi-scale causal feature sublimation, it organically integrates two modalities of information, allowing them to complement and verify each other, thus improving the reliability and robustness of fish behavior analysis and reducing errors caused by the limitations of single-modal information. Furthermore, the semantic factors obtained after transformation not only reflect the visual features themselves but also include their causal strength with acoustic events, enhancing feature robustness.

[0088] In one embodiment, the step of converting the acoustic features into high-level acoustic semantics using a temporal causal filtering method includes:

[0089] The acoustic features are decomposed into several temporal dimensions to obtain several temporal dimension acoustic features; the temporal dimension acoustic features include the intensity of fish feeding collision events, the slope of the target frequency band energy envelope, and the variance of the peak interval in the time and frequency domain.

[0090] Based on the acoustic features, target visual features are selected from the visual features, and the causal triggering strength between the target visual features and the acoustic features is determined to obtain a causal triggering strength score.

[0091] Based on the temporal dimension acoustic features and the causal triggering intensity score, the intensity of the fish feeding collision event is transformed into a collision burst causal semantic factor, the target frequency band energy envelope slope is transformed into a frequency band energy-oriented causal semantic factor, and the time-frequency domain peak interval variance is transformed into a peak period causal semantic factor.

[0092] The high-level acoustic semantics are obtained by combining the collision burst causal semantic factor, the frequency band energy-directed causal semantic factor, and the peak periodic causal semantic factor.

[0093] Specifically, this invention uses temporal causal filtering technology to extract high-level semantic factors that have a causal triggering relationship with visual features from the original acoustic features, namely the energy of the feeding sound target frequency band, the peak value in the time and frequency domain, and the collision transient events (frequency / intensity / duration). This technology strengthens the acoustic features that truly trigger or respond to visual feeding behavior by analyzing the temporal patterns (suddenness, periodicity) of acoustic signals and their lag effects on visual events, while suppressing interference from environmental noise and non-causal associations.

[0094] The acoustic features are decomposed into instantaneous, short-term, and long-term time dimensions, allowing different features to correspond to signals at different time stages of feeding behavior. Specifically, a peak detection algorithm (such as SciPy's find_peaks) is used to extract collision transient events from the features as the intensity of fish feeding collision events, retaining its core parameters—event intensity (signal amplitude, reflecting the collision force) and event frequency (corresponding to the fish mouth / body collision type). This invention does not directly use "duration" (which will be indirectly represented by intensity later), thus obtaining the instantaneous pulse dimension, which corresponds to the collision transient event (frequency / intensity / duration). The target frequency band energy in the features is processed by the sliding window integration method (window length = 100ms, step size = 100ms, covering the time of a single feeding action of the fish school), and the energy mean within each window is calculated. Then, the energy envelope slope of the window is obtained through linear fitting, thus obtaining the short-term trend dimension, i.e., the target frequency band energy envelope slope. By extracting the time-frequency domain peak value at each time point in the acoustic feature map, and then using autocorrelation analysis to calculate the time interval between peak values, the time-frequency domain peak interval variance is obtained, i.e., the long-term period dimension. These temporal acoustic features are all temporal sequences and are aligned with the timestamps of the visual data.

[0095] The process of selecting target visual features is similar to that of selecting target acoustic features, and will not be elaborated upon here. Next, the causal triggering strength between the target visual and acoustic features is determined. This is achieved by calculating the mutual information between the acoustic features at time t and the target visual features at time t+Δt, and then multiplying the calculated mutual information by an indicator function I (Δt>0) to obtain the directional triggering strength of the acoustic features to visual events (such as sudden changes in fish speed). The distribution data in the mutual information calculation process can be determined using kernel density estimation (KDE). Δt is the lag time; using the indicator function I (Δt>0) preserves Δt within the range of 0~0.5s to ensure that only the causal relationship of acoustics preceding vision is considered (consistent with the biological principle of "sound triggering fish responses"). Then, the directional triggering strength at Δt>0 is taken as the causal triggering strength score, which reflects the maximum triggering ability of the acoustic feature to the visual feature in that dimension.

[0096] The product of the intensity of the fish feeding collision event and the causal triggering intensity score is used as the collision burst causal semantic factor. The product of the target frequency band energy envelope slope and the causal triggering intensity score is used as the frequency band energy-oriented causal semantic factor. The difference between 1 and the time-frequency domain peak interval variance is multiplied by the causal triggering intensity score, and the resulting product is used as the peak period causal semantic factor.

[0097] Finally, for each time point (e.g., one time point is taken every 100ms), the above three causal semantic factors are combined in the order of "collision burstiness → frequency band energy orientation → peak periodicity" to form the acoustic causal semantic vector for that time point, thereby obtaining the high-level acoustic semantics.

[0098] This invention performs multi-dimensional temporal decomposition of acoustic features, meticulously depicting their dynamic changes from different perspectives and providing rich information for a comprehensive understanding of fish school behavior. By mining cross-modal causal relationships, it reveals the driving factors behind fish school behavior, making the understanding of acoustic features no longer isolated but closely linked to the overall behavioral patterns of the fish school. The calculated causal trigger strength score helps determine the importance and relevance of different acoustic features in different behavioral scenarios, thereby more accurately selecting and transforming acoustic features, improving the matching degree between high-level acoustic semantics and the actual behavioral state of the fish school, and enabling the transformed semantics to more realistically and accurately reflect the behavioral characteristics and state of the fish school. The transformation of semantic factors accurately quantifies acoustic behavior, improving the ability to identify and analyze acoustic features, and providing convenience for the automated monitoring and evaluation of fish school behavior.

[0099] S4. Extract the spatial association information of the visual high-level semantics and the temporal association information of the acoustic high-level semantics, and fuse the spatial association information and the temporal association information to obtain a fused feature vector;

[0100] In one embodiment, step S4 includes:

[0101] Visual spatial correlation analysis is performed on the visual high-level semantics to obtain spatial correlation information, and acoustic temporal correlation analysis is performed on the acoustic high-level semantics to obtain temporal correlation information.

[0102] A visual causal prediction matrix is ​​constructed based on the visual high-level semantics, and an acoustic triggering matrix is ​​constructed based on the acoustic high-level semantics.

[0103] A synchronization index is determined based on the visual causal prediction matrix and the acoustic triggering matrix, and the spatial correlation information and the temporal correlation information are fused using the synchronization index to obtain the fused feature vector.

[0104] Specifically, the mutual information between visual high-level semantic vectors within grids of different scales is calculated using kernel density estimation, i.e., visual spatial correlation analysis is performed to obtain a spatial correlation graph representing spatial correlation information. This graph also represents the degree of semantic dependence between grids of different scales. The spatial correlation graph is multiplied by visual high-level semantics related to feeding behavior (such as around the fish's mouth) to emphasize this prominent semantic, resulting in a weighted spatial correlation vector. Next, the mutual information of acoustic high-level semantics in each dimension is calculated using kernel density estimation to perform acoustic temporal correlation analysis, resulting in a temporal correlation sequence representing temporal correlation information. This sequence represents the correlation strength between time points and lag time points. The temporal correlation sequence is multiplied by the acoustic high-level semantics corresponding to the sudden time point of feeding sounds to emphasize this semantic, resulting in a weighted temporal correlation vector.

[0105] The visual causal prediction matrix is ​​obtained by calculating the predictive power of high-level visual semantics for acoustic events at each scale grid after a lag time Δt using cross-modal regression analysis. The acoustic triggering matrix is ​​obtained by calculating the triggering intensity of acoustic semantics for visual events at each time point after a lag time Δt using nonlinear kernel regression.

[0106] The synchronization index is determined based on the visual causal prediction matrix and the acoustic triggering matrix. This process is expressed by the following formula:

[0107]

[0108] In the formula, Synchronization index; This is a visual causal prediction matrix; For acoustic triggering matrix; i Scaled grid; k This is the time point index in the acoustic high-level semantic vector group, and its value range is all currently considered time points. The summation symbol ∑ in the denominator represents this index. kThe variable acts as a loop variable, allowing the denominator to iterate through all... k The scale grid representing all visual regions was calculated. i With any acoustic time point k The sum of the strengths of causal relationships; j For a point in time; The preset lag time covers a reasonable physical delay range (e.g., ±0.3 seconds).

[0109] In the above calculation process, the denominator is normalized using softmax to ensure... S i,j ∈[0,1] and for each i , ∑ S i,j =1, used to characterize the scale grid. i With time point j The causal synchronization strength.

[0110] Next, a mutual information adjustment factor is introduced based on the synchronization index, and the weighted spatial correlation vector and the weighted temporal correlation vector are fused using the mutual information adjustment factor to obtain the fused feature vector. This process is expressed by the following formula:

[0111]

[0112]

[0113] In the formula, The mutual information adjustment factor is normalized to ensure... w i,j ∈[0,1] and ∑ w i,j =1; This is a scaling factor (default 1) to control the intensity of mutual information influence. MI () represents the mutual information calculated by kernel density estimation, used to capture nonlinear causal relationships; It is a weighted spatial correlation vector; This is a weighted time-related vector; F To fuse feature vectors; The vectors are concatenated so that the resulting fused feature vector simultaneously contains spatial semantics, temporal semantics, and the strength of their causal relationships.

[0114] This invention enhances semantic factors causally related to feeding behavior and improves noise resistance by using multi-scale causal feature sublimation and temporal causal filtering. It introduces a visual prediction matrix and an acoustic triggering matrix to accurately capture the physical causal chain of audio-visual synchronization (such as the fish mouth action preceding the sound). Based on the synchronization calculation of mutual information and causal association, it avoids the limitations of linear dot product and handles nonlinear relationships more flexibly. It introduces a mutual information adjustment factor to fuse spatial and temporal correlation information, highlighting the audio-visual synchronization region and suppressing irrelevant regions.

[0115] S5. Dynamically evaluate the feeding intensity of the fish population based on the fused feature vector, and verify the evaluation results through a temporal consistency verification mechanism to obtain the feeding intensity level;

[0116] In one embodiment, the dynamic evaluation of fish feeding intensity based on the fused feature vector includes:

[0117] The fused feature vector is input into a time-gated loop unit for modeling, and the hidden state vector at each time step is output.

[0118] The implicit state vectors are analyzed to obtain the density index, motion intensity, acoustic event rate, and feeding behavior frequency, so as to construct a four-dimensional evaluation vector.

[0119] The four-dimensional evaluation vector is classified based on the preset evaluation rules, and the feeding intensity level and confidence level of each frame are output as the evaluation result.

[0120] Specifically, this invention achieves dynamic evaluation of feeding intensity by performing temporal modeling and multi-dimensional index quantification on the fused feature vector. The process includes:

[0121] The fused feature vector is input into the temporally gated recurrent unit (GRU) so that the GRU can capture the temporal dependencies of the feature sequence through gating mechanisms (update gate and reset gate). That is, for the fused feature vector at each time step t, the GRU dynamically decides how much historical information (such as the feeding state of the previous time step) to retain and how much current information (such as the fish movement features of the current frame) to incorporate through gating mechanisms (update gate and reset gate). For example, the intensity change of the fish population from "none" to "weak" at the beginning of feeding needs to be judged in combination with the feature trend of the previous 3-5 frames; the maintenance of "strong" intensity at the peak of feeding needs to identify the persistence of continuous high activity features. Finally, the hidden state vector of each time step is output. Each vector encodes the dynamic change features of feeding intensity within the corresponding time window (such as speed change process, frequency band peak duration) and static features (number of fish, etc.).

[0122] Extract the number of fish schools from the hidden state vector at each time step, and count the number of fish schools per unit area through threshold segmentation or clustering algorithms, which is normalized to a density index V[0] between 0 and 1 (the higher the value, the denser the aggregation), or use the density index = number of effective targets (N) / area of the feeding area (pixels 2 ), and normalize it to [0, 1] (the area is calculated based on the preset coordinates of the net cage); extract the instantaneous velocity of each target from the hidden state vector and calculate its average value, with the unit of px / s (pixels per second, reflecting the swimming activity of the fish school) as the motion intensity score V[1] (such as a high intensity score if the number of tail beats per second exceeds the threshold); count the number of effective events of feeding collision sounds per second from the hidden state vector (filter out noise through an energy threshold) as the acoustic event rate V[2]; determine the frequency of food-grabbing behavior based on the hidden state vector. An event that meets the following conditions is recorded as food-grabbing: the absolute value of acceleration > 200 px / frame 2 (vigorous motion), and the overlap rate of the target boxes ≥ 30% (collision between fish bodies or between fish bodies and the feeding area). The frequency of food-grabbing behavior V[3] = the number of food-grabbing events within a unit time (1 second); splice the above four indicators into a four-dimensional evaluation vector to represent the feeding behavior characteristics at the current time step.

[0123] Finally, classify the four-dimensional vector based on preset rules, and output the feeding intensity level and confidence (Softmax probability) of each frame as the evaluation result: Strong: V[0] > 0.8 (high density) and V[1] > 120 px / s (high-speed motion) and V[2] > 50 times / second (high-frequency acoustic events) and V[3] > 30% (high food-grabbing ratio); Medium: 0.5 < V[0] ≤ 0.8, 80 px / s < V[1] ≤ 120 px / s, 30 times / second < V[2] ≤ 50 times / second, 15% < V[3] ≤ 30% (meeting 3 or more items); Weak: 0.2 < V[0] ≤ 0.5, 40 px / s < V[1] ≤ 80 px / s, 10 times / second < V[2] ≤ 30 times / second, 5% < V[3] ≤ 15% (meeting 2 or more items); None: V[0] < 0.2 (low density) or V[2] < 5 times / second (low acoustic event rate) (judgment is made if 1 item is met).

[0124] The present invention breaks through the limitations of traditional static threshold evaluation, captures the trend changes of feeding intensity through GRU modeling of time series dependence instead of single isolated judgment, making the evaluation more in line with the natural dynamics of fish school feeding; quantifies the feeding intensity from four dimensions of density, motion intensity, acoustic event rate, and food-grabbing behavior frequency, covering multi-modal information of vision - acoustics - behavior, and avoiding misjudgment caused by a single indicator.

[0125] In one embodiment, the evaluation result is verified through a time series consistency verification mechanism to obtain the feeding intensity level, including:

[0126] The evaluation result of the current frame is compared with the evaluation results of adjacent frames, and if there is a conflict in the comparison results, the current frame is regarded as a suspicious frame.

[0127] The DeepSort tracking algorithm is used to obtain the fish swarm trajectory of the suspicious frame and its adjacent frames, and the spatial continuity and velocity continuity are quantized based on the fish swarm trajectory, so as to make a reliability judgment on the evaluation result based on the quantization result.

[0128] If the evaluation result is determined to be unreliable, the evaluation result is corrected according to a preset correction rule, and the feeding intensity level in the corrected result is output.

[0129] Specifically, this invention employs temporal consistency verification to correct misjudgments caused by transient noise (such as the sound of bursting bubbles or fish accidentally swimming past the lens), ensuring that the evaluation results conform to natural behavioral patterns (feeding intensity does not suddenly increase or decrease). The implementation process includes:

[0130] The intensity level (Lt) of the current frame (t) is compared with the levels of its adjacent frames, namely the previous frame (t-1, Lt-1) and the next frame (t+1, Lt+1): if Lt conflicts with Lt-1 and Lt+1 (e.g., Lt=strong, Lt-1=Lt+1=weak, i.e. the same before and after but opposite to the current), then the current frame is marked as a suspicious frame; otherwise, the evaluation result of the current frame is confirmed to be correct.

[0131] Next, a visual flow-based verification mechanism is initiated for suspicious frames: the fish movement trajectories of frames t-1, t, and t+1 are extracted (the trajectory IDs are tracked using the DeepSort algorithm; other existing trajectory algorithms can also be used to determine the fish movement trajectories. For specific tracking processes, please refer to the implementation process of the corresponding algorithms in the prior art. Since the trajectory algorithm is not the focus of this invention, it will not be described in detail here). The spatial continuity of the trajectory (Euclidean distance between trajectory points in adjacent frames ≤ threshold, such as 50px) and velocity continuity (velocity change rate ≤ 30%) are calculated. If ≥ 80% of the fish trajectories show abnormalities in frame t (such as sudden disappearance or sudden velocity change), it is determined that the visual features of frame t have noise interference, and its intensity level is unreliable; otherwise, the verification result of the suspicious frame is determined to be correct.

[0132] Finally, for unreliable and suspicious frames, a preset correction rule, "interpolation correction method," is used for correction: if Lt-1 = Lt+1, Lt is directly corrected to Lt-1 (if both t-1 and t+1 are weak, then frame t is corrected from strong to weak); if Lt-1 and Lt+1 are adjacent levels (e.g., Lt-1 = weak, Lt+1 = medium), then the correction is weighted according to the confidence level (e.g., if the confidence level of frame t is biased towards weak, it is corrected to weak, otherwise it is corrected to medium), and the correction result is used as the final evaluation result output to achieve the evaluation of the feeding intensity of the fish school.

[0133] It should be noted that the evaluation method described in this invention is only one embodiment. Alternatively, after obtaining the fused features, the feeding intensity assessment can be regarded as a classification task, and the mapping from fused features to intensity levels can be achieved through an end-to-end network, such as the multimodal Transformer classifier (Fish-MulT). That is, cross-modal attention is introduced into the Transformer architecture to automatically learn the correlation between acoustic and visual features. After fusing features through the multimodal transfer module (MMTM), the features are weighted and added using an adaptive weight allocation strategy. Finally, the intensity probability distribution is output through a fully connected layer. Other existing evaluation methods can also be used, which are not limited here.

[0134] This invention is based on a verification mechanism for the continuity of movement trajectories. It binds the verification criteria to the actual behavior of the fish school, rather than simply smoothing the time sequence. It can effectively distinguish between real intensity abrupt changes (such as the burst of feeding at the moment of feeding) and misjudgments caused by noise (such as interference from water splashes in the lens). It adopts backward verification (combining the results of t+1 frames) instead of relying solely on historical frames, which solves the problem of the cumulative transmission of misjudgments in the current frame in traditional forward verification and improves the stability of long sequence evaluation.

[0135] In one embodiment, the effect of feature extraction and feature fusion in the fish feeding intensity evaluation method of the present invention is compared with other schemes, and the results are shown in the table below:

[0136] Table 1 Comparison of Results

[0137]

[0138] Table 1 shows that traditional visual methods employ image dehazing based on dark channel priors combined with YOLO target detection, while pure acoustic methods use Mel spectrum analysis combined with event detection (such as the Teager-Kaiser energy operator). Simple multimodal fusion uses feature concatenation of acoustic and visual features. As can be seen from Table 1, this invention, through an improved lightweight dual-stream network model and a cross-modal attention fusion design of "sound-guided vision, vision-assisted sound," overcomes the limitations of complex underwater environments. Compared to traditional single-modal or shallow fusion schemes, it comprehensively improves robustness in turbid water, adaptability to low light, and noise resistance, providing a reliable technical foundation for intelligent aquaculture while significantly reducing operation and maintenance costs.

[0139] The method for evaluating fish feeding intensity described in this invention was applied to various underwater scenarios, and the results were compared with those of the traditional YOLOv4+ optical flow method, as shown in the table below:

[0140] Table 2 Comparison of Technical Effects

[0141]

[0142] As shown in Table 2, compared with the accuracy of traditional evaluation methods, the present invention improves accuracy by 6% in clear water under strong light, 34% in turbid water under cloudy conditions, 94% in nighttime feeding conditions, and 42% in rainstorm interference conditions. This indicates that the present invention not only improves the evaluation accuracy in conventional scenarios, but is also applicable to extreme scenarios under non-ideal conditions. It breaks through the limitations of existing evaluation schemes in complex and extreme underwater environments (turbidity, weak light, noise), and greatly improves the accuracy of evaluating the feeding intensity of fish.

[0143] This invention addresses the problem of accurately and in real-time assessing the feeding intensity of fish schools by designing a method for evaluating feeding intensity based on target detection and acoustic detection. This method employs multi-source data acquisition, comprehensively utilizing video streams and acoustic signals to fully reflect the feeding status of the fish school and improve the accuracy of the assessment. An improved lightweight dual-stream network model is used for feature extraction, effectively reducing the model's complexity and computational load while ensuring accuracy. This allows the model to accurately output visual and acoustic features in real-time, meeting the needs of timely monitoring of fish school status in actual production. Furthermore, multi-scale causal feature enhancement is employed. The method and temporal causal filtering method perform semantic transformation and fusion of features, thereby deeply mining the deep information contained in image and sound data, and elevating low-level features to high-level representations with clear semantics. Based on the fused feature vector, dynamic evaluation of fish feeding intensity can reflect changes in fish feeding status in real time, providing timely and accurate decision-making basis for aquaculture personnel. At the same time, the temporal consistency verification mechanism is used to verify the evaluation results, which can eliminate erroneous evaluations caused by data noise or instantaneous model errors, further improving the reliability and stability of the evaluation results and ensuring the scientificity and effectiveness of aquaculture decisions.

[0144] It should be noted that although the steps in the flowchart above are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order requirement for the execution of these steps, and they can be executed in other orders.

[0145] In another embodiment, such as Figure 2 As shown, a second aspect of the present invention provides a fish feeding intensity evaluation system based on target detection and acoustic detection, comprising:

[0146] Data acquisition module 10 is used to synchronously acquire real-time image data and real-time sound data of fish in deep-water cages after feeding;

[0147] Feature extraction module 20 is used to input the real-time image data and the real-time sound data into an improved lightweight dual-stream network model for feature extraction, and output visual features and acoustic features.

[0148] The feature transformation module 30 is used to transform the visual features into high-level visual semantics through a multi-scale causal feature sublimation method, and to transform the acoustic features into high-level acoustic semantics through a temporal causal filtering method.

[0149] The feature fusion module 40 is used to extract the spatial correlation information of the visual high-level semantics and the temporal correlation information of the acoustic high-level semantics, and fuse the spatial correlation information and the temporal correlation information to obtain a fused feature vector;

[0150] The evaluation and verification module 50 is used to dynamically evaluate the feeding intensity of fish groups based on the fused feature vector, and to verify the evaluation results through a temporal consistency verification mechanism to obtain the feeding intensity level.

[0151] It should be noted that each module in the aforementioned target detection and acoustic detection-based fish feeding intensity evaluation system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module. For specific limitations regarding the target detection and acoustic detection-based fish feeding intensity evaluation system, please refer to the limitations of the target detection and acoustic detection-based fish feeding intensity evaluation method described above; both have the same function and role, and will not be repeated here.

[0152] In summary, this invention relates to the field of aquaculture monitoring technology, and discloses a method and system for evaluating the feeding intensity of fish schools using target detection and acoustic detection. An improved lightweight dual-stream network model is used to process image and sound data of fish schools after feeding, outputting visual and acoustic features. A multi-scale causal feature sublimation method is used to transform the visual features, and a temporal causal filtering method is used to transform the acoustic features, obtaining high-level visual and acoustic semantics. Spatial correlation information of the high-level visual semantics and temporal correlation information of the high-level acoustic semantics are extracted and fused to obtain a fused feature vector for dynamic evaluation of fish feeding intensity. The evaluation results are then verified through a temporal consistency check mechanism to obtain the feeding intensity level. This invention, by deeply mining the complementary information of multimodal data, achieves accurate and real-time assessment of the feeding intensity of fish schools in deep-sea cages, providing a reliable basis for scientific aquaculture management.

[0153] The various embodiments in this specification are described in a progressive manner. For directly identical or similar parts of the embodiments, refer to each other. Each embodiment focuses on its differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. It should be noted that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0154] The above-described embodiments are merely preferred embodiments of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various improvements and substitutions without departing from the principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for evaluating the feeding intensity of fish schools using target detection and acoustic detection, characterized in that, include: Simultaneously acquire real-time image and sound data of fish in deep-sea cages after feeding; The real-time image data and the real-time sound data are input into an improved lightweight two-stream network model for feature extraction, and visual and acoustic features are output. The visual features are transformed into high-level visual semantics through a multi-scale causal feature sublimation method, and the acoustic features are transformed into high-level acoustic semantics through a temporal causal filtering method. Extract the spatial association information of the visual high-level semantics and the temporal association information of the acoustic high-level semantics, and fuse the spatial association information and the temporal association information to obtain a fused feature vector; The feeding intensity of fish schools is dynamically evaluated based on the fused feature vector, and the evaluation results are verified through a temporal consistency verification mechanism to obtain the feeding intensity level. The improved lightweight dual-stream network model includes a pruned YOLO-MS network. The real-time image data is processed by the pruned YOLO-MS network to obtain visual features, which include the number of fish, their orientation angle, and their instantaneous speed. The improved lightweight two-stream network model also includes a parallel dual-branch structure network; the parallel dual-branch structure network includes the MobileNetV2-CBAM branch and the TCN branch; The real-time sound data is processed by the MobileNetV2-CBAM branch to obtain a time-frequency feature vector, which includes the target frequency band energy of the fish feeding sound and the time-frequency domain peak value. The real-time sound data is processed by the TCN branch to obtain a time-domain feature vector, which includes the frequency, intensity and duration of the fish feeding collision transient event. The time-frequency feature vector and the time-domain feature vector are used as the acoustic features.

2. The method for evaluating fish feeding intensity based on target detection and acoustic detection according to claim 1, characterized in that, The pruned YOLO-MS network includes a feature extraction layer, a feature fusion layer, and a detection output layer; wherein... The step of inputting the real-time image data and the real-time audio data into an improved lightweight two-stream network model for feature extraction, and outputting visual and acoustic features, includes: The feature extraction layer extracts appearance and motion features from the real-time image data to obtain several multi-scale fusion feature maps. The feature fusion layer is used to perform upsampling fusion processing and downsampling fusion processing on each of the multi-scale fusion feature maps to obtain several enhanced multi-scale feature maps; The enhanced multi-scale feature maps are input into the detection output layer for processing, and the number of fish, orientation angle, and instantaneous speed are output as the visual features.

3. The method for evaluating fish feeding intensity based on target detection and acoustic detection according to claim 1, characterized in that, The MobileNetV2-CBAM branch includes a first input layer, a lightweight feature extraction layer, a CBAM attention layer, and a first output layer; wherein... The step of inputting the real-time image data and the real-time audio data into an improved lightweight two-stream network model for feature extraction, and outputting visual and acoustic features, further includes: The real-time audio data is converted into a Mel spectrogram and input into the first input layer for normalization processing to obtain a normalized Mel spectrogram. The time-frequency domain features of the standardized Mel spectrum are extracted by the lightweight feature extraction layer to obtain a multi-scale time-frequency feature map that characterizes the energy distribution of fish feeding sounds in multiple frequency bands. The CBAM attention layer is used to perform feature weight enhancement processing on the multi-scale time-frequency feature map to obtain a weighted time-frequency feature map. The weighted time-frequency feature map is compressed based on the first output layer to obtain a time-frequency feature vector as the acoustic feature; the time-frequency feature vector includes the target frequency band energy of the fish feeding sound and the time-frequency domain peak value.

4. The method for evaluating fish feeding intensity using target detection and acoustic detection according to claim 3, characterized in that, The parallel dual-branch network further includes a TCN branch, which comprises a second input layer, a temporal convolutional layer, and a second output layer; wherein... The step of inputting the real-time image data and the real-time audio data into an improved lightweight two-stream network model for feature extraction, and outputting visual and acoustic features, further includes: The real-time sound data is divided into frames by the second input layer to extract the zero-crossing rate (used to characterize the intensity of the feeding signal fluctuations), the spectral entropy (used to characterize the complexity of the feeding frequency distribution), and the MFCC (used to characterize the voiceprint features) from each frame of signal. The frames are then spliced ​​together according to time steps to obtain a time-series feature matrix. The temporal convolutional layer is used to extract transient events of fish feeding collisions from the temporal feature matrix to obtain a feeding collision temporal feature map; each channel in the feeding collision temporal feature map corresponds to a transient event pattern of fish feeding collisions. The feeding collision temporal feature map is compressed based on the second output layer to obtain a temporal feature vector as the acoustic feature; the temporal feature vector includes the frequency, intensity and duration of the fish feeding collision transient event.

5. The method for evaluating fish feeding intensity using target detection and acoustic detection according to claim 2, characterized in that, The process of transforming visual features into high-level visual semantics through multi-scale causal feature sublimation includes: The real-time image data is decomposed into several scale grids, and the spatial distribution statistics of the visual features are calculated on each scale grid. Based on the visual features, target acoustic features are selected from the acoustic features, and the visual features and the target acoustic features are determined in each of the scale grids. of Causality strength, and obtain the causality strength score; Based on the spatial distribution statistics and the causal strength score, the number is transformed into a group size causal semantic factor, the orientation angle is transformed into a motion coordination causal semantic factor, and the instantaneous velocity is transformed into a motion response causal semantic factor. The visual high-level semantics are obtained by combining the group-scale causal semantic factors, motion-co-causal semantic factors, and action-response causal semantic factors in each scale grid.

6. The method for evaluating fish feeding intensity using target detection and acoustic detection according to claim 4, characterized in that, The process of converting the acoustic features into high-level acoustic semantics using a temporal causal filtering method includes: The acoustic features are decomposed into several temporal dimensions to obtain several temporal dimension acoustic features; the temporal dimension acoustic features include the intensity of fish feeding collision events, the slope of the target frequency band energy envelope, and the variance of the peak interval in the time and frequency domain. Based on the acoustic features, target visual features are selected from the visual features, and the causal triggering strength between the target visual features and the acoustic features is determined to obtain a causal triggering strength score. Based on the temporal dimension acoustic features and the causal triggering intensity score, the intensity of the fish feeding collision event is transformed into a collision burst causal semantic factor, the target frequency band energy envelope slope is transformed into a frequency band energy-oriented causal semantic factor, and the time-frequency domain peak interval variance is transformed into a peak period causal semantic factor. The high-level acoustic semantics are obtained by combining the collision burst causal semantic factor, the frequency band energy-directed causal semantic factor, and the peak periodic causal semantic factor.

7. The method for evaluating fish feeding intensity using target detection and acoustic detection according to claim 1, characterized in that, The step of extracting the spatial correlation information of the visual high-level semantics and the temporal correlation information of the acoustic high-level semantics, and fusing the spatial and temporal information to obtain a fused feature vector, includes: Visual spatial correlation analysis is performed on the visual high-level semantics to obtain spatial correlation information, and acoustic temporal correlation analysis is performed on the acoustic high-level semantics to obtain temporal correlation information. A visual causal prediction matrix is ​​constructed based on the visual high-level semantics, and an acoustic triggering matrix is ​​constructed based on the acoustic high-level semantics. A synchronization index is determined based on the visual causal prediction matrix and the acoustic triggering matrix, and the spatial correlation information and the temporal correlation information are fused using the synchronization index to obtain the fused feature vector.

8. The method for evaluating fish feeding intensity based on target detection and acoustic detection according to claim 1, characterized in that, The dynamic evaluation of fish feeding intensity based on the fused feature vector includes: The fused feature vector is input into a time-gated loop unit for modeling, and the hidden state vector at each time step is output. The implicit state vectors are analyzed to obtain the density index, motion intensity, acoustic event rate, and feeding behavior frequency, so as to construct a four-dimensional evaluation vector. The four-dimensional evaluation vector is classified based on the preset evaluation rules, and the feeding intensity level and confidence level of each frame are output as the evaluation result.

9. The method for evaluating fish feeding intensity based on target detection and acoustic detection according to claim 1, characterized in that, The evaluation results are verified through a temporal consistency verification mechanism to obtain the feeding intensity level, including: The evaluation result of the current frame is compared with the evaluation results of adjacent frames, and if there is a conflict in the comparison results, the current frame is regarded as a suspicious frame. The DeepSort tracking algorithm is used to obtain the fish swarm trajectory of the suspicious frame and its adjacent frames, and the spatial continuity and velocity continuity are quantized based on the fish swarm trajectory, so as to make a reliability judgment on the evaluation result based on the quantization result. If the evaluation result is determined to be unreliable, the evaluation result is corrected according to a preset correction rule, and the feeding intensity level in the corrected result is output.

10. A system for evaluating the feeding intensity of fish schools using target detection and acoustic detection, characterized in that, include: The data acquisition module is used to synchronously acquire real-time image and sound data of fish in deep-water cages after feeding. The feature extraction module is used to input the real-time image data and the real-time sound data into an improved lightweight dual-stream network model for feature extraction, and output visual features and acoustic features. The feature transformation module is used to transform the visual features into high-level visual semantics through a multi-scale causal feature sublimation method, and to transform the acoustic features into high-level acoustic semantics through a temporal causal filtering method. The feature fusion module is used to extract the spatial correlation information of the visual high-level semantics and the temporal correlation information of the acoustic high-level semantics, and fuse the spatial correlation information and the temporal correlation information to obtain a fused feature vector; The evaluation and verification module is used to dynamically evaluate the feeding intensity of fish based on the fused feature vector, and to verify the evaluation results through a temporal consistency verification mechanism to obtain the feeding intensity level. The improved lightweight dual-stream network model includes a pruned YOLO-MS network. The real-time image data is processed by the pruned YOLO-MS network to obtain visual features, which include the number of fish, their orientation angle, and their instantaneous speed. The improved lightweight two-stream network model also includes a parallel dual-branch structure network; the parallel dual-branch structure network includes the MobileNetV2-CBAM branch and the TCN branch; The real-time sound data is processed by the MobileNetV2-CBAM branch to obtain a time-frequency feature vector, which includes the target frequency band energy of the fish feeding sound and the time-frequency domain peak value. The real-time sound data is processed by the TCN branch to obtain a time-domain feature vector, which includes the frequency, intensity and duration of the fish feeding collision transient event. The time-frequency feature vector and the time-domain feature vector are used as the acoustic features.

Images