Fish school movement state anomaly early warning method and system based on sonar space-time trajectory diagram

Abstract

The application discloses a fish school movement state abnormal early warning method and system based on a sonar space-time trajectory diagram, first acquires continuous sonar images of fish schools in a breeding water area, and obtains to-be-detected sonar frames through noise reduction and enhancement processing; the to-be-detected sonar frames are input into a trained sonar fish body detection network to obtain the number of fish bodies, a boundary box, detection confidence and position information; cross-frame correlation is performed on adjacent frame detection results, the fish bodies are mapped into space-time nodes containing position, size, confidence, speed and direction, and a sonar space-time trajectory diagram is constructed based on a sliding time window; spatial structure, time movement and overall state features are extracted, a normal state center and an early warning threshold are established according to a normal swimming stage, a state judgment score of a to-be-detected window is calculated for comparison, and a fish school movement state normal or abnormal early warning result is output. The application is suitable for turbid and low-visibility water areas, can reduce the dependence on artificial patrol and abnormal sample labeling, and improves early warning stability and interpretability.

Description

Technical Field

[0001] This invention belongs to the fields of smart fisheries, target detection and graph data modeling technology, and relates to a method and system for early warning of abnormal fish movement, specifically a method and system for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps. Background Technology

[0002] Real-time monitoring of the aquatic environment and fish health status is a widely concerned technology in aquaculture. Changes in water quality, feeding competition, external disturbances, and early disease stress can all cause variations in fish swimming speed, direction of movement, aggregation, and spatial distribution. Fish are highly sensitive to environmental changes, and their movement patterns can reflect these changes and their stress responses, serving as important behavioral indicators for assessing fish health and aquaculture risks. Failure to detect abnormal changes in fish movement patterns in a timely manner can lead to accumulated stress, further affecting fish feeding, growth, and the stability of the aquaculture process. Traditional manual inspections rely on observations by aquaculture personnel at specific times, which are easily affected by factors such as water turbidity, lighting conditions, observation range, and subjective experience, making long-term, continuous monitoring difficult. Therefore, researching automated analysis and anomaly warning methods for fish movement patterns is crucial for timely detection of aquaculture environmental risks, improving fish health management, and promoting the intelligent development of aquaculture.

[0003] Currently, scholars both domestically and internationally have conducted extensive research on fish movement behavior recognition, behavior quantification, and abnormal state detection, achieving significant results. Early studies primarily utilized image processing methods such as background subtraction, inter-frame subtraction, and optical flow to extract information on the movement region, swimming speed, direction, and trajectory of fish schools, describing their movement state using parameters such as velocity entropy, angular entropy, activity level, and aggregation index. Some studies further utilized gray-level co-occurrence matrices to extract texture features such as energy, entropy, contrast, and correlation from fish school images, fusing motion features with overall image features to quantify fish feeding behavior and abnormal swimming states. Building upon this foundation, machine learning methods such as support vector machines, XGBoost, LightGBM, and deep neural networks have been used to establish classification models between normal and abnormal states, improving the automatic identification capability of abnormal fish behavior. In recent years, deep learning object detection and multi-object tracking methods have further enabled the acquisition of fish body position, scale, and cross-frame motion trajectories, providing new technical means for individual fish movement analysis. Furthermore, graph neural networks demonstrate superior relationship modeling capabilities in temporal anomaly detection, capable of describing the spatial relationships between different objects and their changes over time through nodes and edges, providing a new research approach for the analysis of complex group behavior. These studies have propelled fish behavior monitoring from manual observation to automated identification and quantitative analysis, laying a solid foundation for intelligent aquaculture monitoring.

[0004] While existing methods have achieved good detection results, they still have certain shortcomings in early warning of abnormal fish movement in complex aquaculture scenarios. First, most studies use visible light images as data sources, which are easily affected by factors such as water turbidity, lighting changes, water surface reflection, and fish occlusion, making it difficult to reliably obtain continuous fish movement information. Second, existing methods often use artificial statistical features such as speed, turning angle, texture, and activity level to describe the state of the fish school, usually compressing the entire fish school or time segment into several independent features, making it difficult to fully characterize the spatial proximity relationships between different fish at the same time, the movement connections of the same fish at consecutive times, and the dynamic changes of individuals relative to the overall distribution of the fish school. Third, some methods mainly classify states based on single-frame images or fixed time windows, failing to adequately depict the continuous evolution of fish movement relationships over time, making the detection results susceptible to single-frame missed detections, fish overlap, and short-term movement fluctuations. Furthermore, supervised classification-based anomaly recognition methods usually require the pre-acquisition of a certain number and type of anomaly samples, with the model focusing more on identifying anomaly categories defined during the training phase, limiting its adaptability to unknown state changes. Therefore, how to stably acquire fish target information in complex underwater environments and uniformly express the relationship between individual fish movement, cross-frame trajectory and the overall state of the fish school is a key problem that needs to be solved to achieve early warning of abnormal fish movement. Summary of the Invention

[0005] To address the technical problems in existing technologies, such as heavy reliance on manual inspections, significant impact of water transparency on optical monitoring, instability in small-scale fish target detection by sonar, and difficulty in expressing the dynamic relationships of fish groups using single-frame statistical features, this invention provides a method and system for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps. This method can stably detect fish targets in sonar images and further organize the fish detection results in consecutive sonar frames into a structured expression with spatial, temporal, and overall relational constraints, thereby achieving timely early warning of fish movement patterns deviating from normal patterns.

[0006] The technical solution adopted by the present invention is: a method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps, comprising the following steps: Step 1: Obtain a continuous sonar image sequence of fish in the aquaculture area, and perform noise reduction and enhancement preprocessing to obtain the sonar frame to be detected; Step 2: Input the sonar frame to be detected into the trained sonar fish detection network, and output the fish detection results in each frame, including the bounding box position parameters of the fish target and the detection confidence. Step 3: Perform cross-frame association based on the fish detection results in adjacent sonar frames, and map the fish target as a spatiotemporal node with spatial location, target scale and detection confidence; using a sliding time window as the basic unit, construct a sonar spatiotemporal trajectory map based on the spatiotemporal nodes, including spatial relationships, temporal relationships and overall relationships; Step 4: Based on the sonar spatiotemporal trajectory map, extract the fish movement features from the spatial hyperedge, temporal hyperedge, and overall hyperedge; calculate the normal baseline using the normal movement time period; calculate the degree of deviation between the test time period and the normal baseline, and complete the anomaly judgment based on the threshold.

[0007] Preferably, the noise reduction and enhancement preprocessing in step 1 involves first performing Gaussian filtering on the original sonar image to smooth the background noise, and then performing filtering to enhance the distinction between the fish body region and the background region. The filtering is based on anisotropic diffusion filtering, which changes the traditional fixed-parameter anisotropic diffusion to an adaptive diffusion controlled by the local noise intensity and the fish body edge information.

[0008] Preferably, the sonar fish detection network in step 2 includes a first Conv layer, a first SPDConv layer, a first C3K2 layer, a second SPDConv layer, a second C3K2 layer, a third SPDConv layer, a third C3K2 layer, a second Conv layer, a fourth C3K2 layer, an SPPF layer, and a DLKA layer connected in sequence. The output of the DLKA layer is processed by the first upsampling layer, and then fused with the output of the third C3K2 layer by the first concat layer before being input into the fifth C3K2 layer. The output of the fifth C3K2 layer is processed by the second upsampling layer, and then fused with the output of the second C3K2 layer by the second concat layer before being input into the sixth C3K2 layer. The output of the sixth C3K2 layer is processed by the first Detect layer to output a prediction result suitable for small-scale fish target detection, including the bounding box position parameters and class confidence of the fish target; after being processed by the third Conv layer, it is fused with the output of the fifth C3K2 layer by the third Concat layer and then input into the seventh C3K2 layer. The output of the seventh C3K2 layer is processed by the second Detect layer to output a prediction result suitable for mid-scale fish target detection. The prediction result includes the bounding box position parameters and class confidence of the fish target. After processing by the fourth Conv layer, it is fused with the output of the DLKA layer by the fourth Concat layer and then input into the eighth C3K2 layer. Finally, after processing by the third Detect layer, it outputs a prediction result suitable for large-scale fish target detection. The prediction result includes the bounding box position parameters and class confidence of the fish target. The first, second, and third SPDConv layers are all spatial depth transformation convolutional layers, used to rearrange spatial information to the channel dimension during downsampling. The DLKA layer is a deformable large kernel attention layer, used for adaptive feature enhancement of sonar fish targets with weak boundaries, small scale, and irregular shape. The first, second, third, fourth, fifth, sixth, seventh, and eighth C3K2 layers are all cross-stage feature extraction and fusion modules, used to reuse and fuse features at different levels through branching, convolutional transformation, and channel concatenation, enhancing the local details and semantic feature expression capabilities of sonar fish targets. The SPPF layer is a fast spatial pyramid pooling layer, used to aggregate multi-scale contextual information under different receptive fields, expand the network's receptive field, and enhance the feature expression capabilities of fish targets at different scales.

[0009] Preferably, the spatial depth transformation convolutional layer is applied to the input feature map. Given a spatial rearrangement ratio First, the input feature map is rearranged from spatial to depth. Convert to Then, the output features are obtained through ordinary convolution. ;in, These represent the height, width, and number of channels of the feature map, respectively, and R is a real number field.

[0010] Preferably, the deformable large kernel attention layer is configured for the input feature map. First, large kernel convolution is used to expand the receptive field and obtain spatial context features. ;in, These represent the height, width, and number of channels of the feature map, respectively, where R is a real number. Indicates the kernel size as void ratio Large kernel convolution operation, The obtained attention features are then represented; then the attention map is used to analyze the features. Perform recalibration to obtain enhanced output features. ,in, This represents the Sigmoid activation function. This indicates element-wise multiplication.

[0011] Preferably, in step 3, the construction of the sonar spatiotemporal trajectory map based on the spatiotemporal nodes is performed for the... A sonar video composed of consecutive sonar images is denoted as the first frame. The first frame Fish detection results for individual fish targets ;in, The coordinates of the center of the bounding box representing the fish target; and Indicates the width and height of the bounding box; This is expressed as the detection confidence level; Each detected fish target is constructed as a node. This node contains the location of the fish target. ,scale Confidence level and the frame number It is used to simultaneously describe the spatial location and temporal frame information of the fish target; Constructing a sonar spatiotemporal trajectory map ;in, This represents the set of nodes consisting of detected fish targets; , and Let these represent the set of spatial hyperedges, the set of temporal hyperedges, and the set of global hyperedges, respectively. The spatial hyperedge, for nodes in frame t. ,according to Calculate its Euclidean distance with other nodes in the same frame; let Represents a node In the t-th frame The nearest neighbor node, its local average nearest neighbor distance and spatial neighborhood radius They are represented as follows: ;in, Represents a node Other nodes The center distance; The neighborhood range adjustment coefficient; based on the node Centered on, its corresponding spatial hyperedge is constructed as ;in, This represents all fish target nodes in frame t; The time superedge, for nodes in frame t. and the nodes in frame t+1 Define the comprehensive matching cost as ;in, This represents the normalized center displacement of the two nodes; Indicates the relative changes in the width and height of the bounding box; Indicates the difference in detection confidence; , and The corresponding weighting coefficients are used; among the candidate nodes that satisfy the maximum displacement and scale change, the node with the minimum comprehensive matching cost is selected as the corresponding target, and the nodes obtained by continuous matching within a time window are organized into time hyperedges in chronological order using a one-to-one matching constraint. ;in, This indicates the number of valid nodes contained in the trajectory within the current time window, and that all adjacent nodes satisfy the above matching conditions. The overall hyperedge, if the m-th time window contains L consecutive frames of sonar images, then all fish target nodes in that window constitute a set. Then the overall hyperedge is ;in, Indicates the detection confidence threshold; Represents a node The effective length of the trajectory within the m-th time window; This represents the minimum effective trajectory length.

[0012] Preferably, in step 4, for the sonar spatiotemporal trajectory map constructed within a certain time period, fish movement features are extracted from the spatial hyperedge, temporal hyperedge, and global hyperedge to obtain the fish movement feature vector for that time period. ;in, Represents the space hyperedge Extracted features after normalization; Indicates the time-bound edge Extracted features after normalization; Indicates the global hyperedge Extracted features after normalization.

[0013] Preferably, in step 4, the normal baseline is calculated using a period of normal movement. This involves selecting a period of stable fish movement without external interference as the training sample and calculating the mean value of the fish movement characteristics as the normal baseline for fish movement. ; in, Indicates the first Motion characteristics of a school of fish under normal movement in a single frame; Indicates the number of frames in the training samples; Indicates the normal reference; Calculate all training samples relative to the normal baseline Judgment score Then calculate the mean of these judgment scores. and standard deviation The exception determination rule is as follows: ; in, Represents Euclidean distance. This represents the threshold adjustment coefficient, used to control the sensitivity of anomaly detection; Indicates the first One sonar segment was identified as abnormal; This indicates that the segment is within the normal range.

[0014] Preferably, in step 2, the trained sonar fish detection network uses a bounding box regression loss function during training. ; Let the predicted bounding box and the ground truth bounding box be represented as follows:

[0015] in, and These represent the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. These represent the width and height of the bounding boxes corresponding to the predicted and ground truth boxes, respectively. but: ; in, This represents the normalized Wasserstein distance loss. Indicates weighted IoU loss. This is the balance coefficient;

[0016]

[0017]

[0018]

[0019]

[0020]

[0021]

[0022] in, Represents the second-order Wasserstein distance; This is a normalization constant; and These represent the width and height of the smallest bounding rectangle of the predicted bounding box and the ground truth bounding box, respectively; The exponential moving average representing IoU loss; and Indicates dynamic focusing parameters; This represents a pre-defined minimum positive number; This indicates a gradient truncation operation. During the forward computation, the value of the variable within the parentheses remains unchanged, while during the backpropagation, the gradient corresponding to that variable is set to zero, so that it only participates in the loss calculation as a dynamic weight, and does not participate in the gradient update of the network parameters.

[0023] The technical solution adopted by the system of the present invention is: an early warning system for abnormal fish movement based on sonar spatiotemporal trajectory maps, comprising: One or more processors; A storage device is provided for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the aforementioned method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps.

[0024] Compared with the prior art, the beneficial effects of the present invention include: (1) The present invention uses imaging sonar image sequences as the main input, does not rely on underwater lighting and high transparency conditions, can be adapted to low visibility aquaculture waters such as turbid ponds, and has the potential for non-contact, continuous and all-weather monitoring. (2) The sonar fish detection network constructed in this invention improves the difficulty of fish detection in sonar images with "small scale, weak boundary and strong noise". It improves the stability of sonar fish target detection and the accuracy of bounding box localization by synergistically using SPDConv, DLKA and NWD-WIoU. (3) The present invention maps discrete detection boxes into spatiotemporal nodes with spatial location, scale, confidence and motion attributes, and constructs a sonar spatiotemporal trajectory map containing spatial, temporal and overall relationships, which can simultaneously describe the local proximity relationship of fish, cross-frame motion continuity and global distribution change of the group; (4) The present invention establishes the normal state center and early warning threshold by adopting the normal swimming stage, which does not require a large number of complete abnormal samples in advance, and is suitable for application conditions where there are few abnormal samples and labeling is difficult in actual breeding scenarios. (5) The present invention outputs a status judgment score and anomaly warning results, which can provide interpretable risk warning information for aquaculture personnel, which is conducive to intelligent management and early risk detection in the aquaculture process. Attached Figure Description

[0025] The technical solutions of the present invention will be further illustrated below using embodiments and specific implementation methods. In addition, some accompanying drawings are used in the description of the technical solutions. Those skilled in the art can obtain other drawings and the intent of the present invention from these drawings without any creative effort.

[0026] Figure 1 This is a flowchart illustrating the overall process of an embodiment of the present invention. Figure 2 This is a schematic diagram of the sonar fish detection network structure according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the construction of a sonar spatiotemporal trajectory diagram according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the anomaly detection flowchart according to an embodiment of the present invention; Figure 5 This is a comparison diagram of class activation maps of the YOLO11n baseline model and the SFD-YOLO model used in the experiments of this invention embodiment; Figure 6 Figure (a) shows the anomaly detection score variation curves under different perturbation time frames in the experiment of the embodiment of the present invention, and (b) shows the anomaly detection results under the condition of no artificial perturbation, and (b) shows the anomaly detection results under the condition of artificial perturbation around frame 450. Detailed Implementation

[0027] To facilitate understanding and implementation of the present invention by those skilled in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0028] Please see Figure 1 This embodiment provides a method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps, including the following steps: Step 1: Obtain a continuous sonar image sequence of fish in the aquaculture area, and perform noise reduction and enhancement preprocessing to obtain the sonar frame to be detected; In one embodiment, the noise reduction and enhancement preprocessing involves first performing Gaussian filtering on the original sonar image to smooth background noise, and then performing filtering to enhance the distinction between the fish body region and the background region. The filtering is based on anisotropic diffusion filtering, which changes the traditional fixed-parameter anisotropic diffusion to an adaptive diffusion jointly controlled by local noise intensity and fish body edge information.

[0029] Step 2: Input the sonar frame to be detected into the trained sonar fish detection network, and output the fish detection results in each frame, including the bounding box position parameters of the fish target and the detection confidence. In one implementation, please see Figure 2 The sonar fish detection network includes a first Conv layer, a first SPDConv layer, a first C3K2 layer, a second SPDConv layer, a second C3K2 layer, a third SPDConv layer, a third C3K2 layer, a second Conv layer, a fourth C3K2 layer, an SPPF layer, and a DLKA layer connected in sequence. The output of the DLKA layer is processed by the first upsampling layer, and then fused with the output of the third C3K2 layer by the first concat layer before being input into the fifth C3K2 layer. The output of the fifth C3K2 layer is processed by the second upsampling layer, and then fused with the output of the second C3K2 layer by the second concat layer before being input into the sixth C3K2 layer. The output of the sixth C3K2 layer is processed by the first Detect layer to output a prediction result suitable for small-scale fish target detection, including the bounding box position parameters and class confidence of the fish target; after being processed by the third Conv layer, it is fused with the output of the fifth C3K2 layer by the third Concat layer and then input into the seventh C3K2 layer. The output of the seventh C3K2 layer is processed by the second Detect layer to output a prediction result suitable for mid-scale fish target detection. The prediction result includes the bounding box position parameters and class confidence of the fish target. After processing by the fourth Conv layer, it is fused with the output of the DLKA layer by the fourth Concat layer and then input into the eighth C3K2 layer. Finally, after processing by the third Detect layer, it outputs a prediction result suitable for large-scale fish target detection. The prediction result includes the bounding box position parameters and class confidence of the fish target. The first, second, and third SPDConv layers are all spatial depth transformation convolutional layers, used to rearrange spatial information to the channel dimension during downsampling. The DLKA layer is a deformable large kernel attention layer, used for adaptive feature enhancement of sonar fish targets with weak boundaries, small scale, and irregular shape. The first, second, third, fourth, fifth, sixth, seventh, and eighth C3K2 layers are all cross-stage feature extraction and fusion modules, used to reuse and fuse features at different levels through branching, convolutional transformation, and channel concatenation, enhancing the local details and semantic feature expression capabilities of sonar fish targets. The SPPF layer is a fast spatial pyramid pooling layer, used to aggregate multi-scale contextual information under different receptive fields, expand the network's receptive field, and enhance the feature expression capabilities of fish targets at different scales.

[0030] In sonar fish swarm videos, fish targets exhibit varying scales. Smaller targets are prone to losing local details during downsampling using traditional stride convolution, thus impacting detection performance. Therefore, in one implementation, the spatial depth transformation convolutional layer SPDConv, for input feature maps... Given a spatial rearrangement ratio The space-to-depth rearrangement operation can transform the input feature map into... Then, the output feature Y1 is obtained through ordinary convolution.

[0031] , ,

[0032] in, These represent the height, width, and number of channels of the feature map, respectively. This represents the feature map after rearranging from spatial to channel dimensions. This represents the output features of SPDConv. Compared to traditional stride convolution, SPDConv reduces spatial detail loss during downsampling, allowing the network to retain more local structural information related to small-scale fish targets in subsequent feature fusion stages. For the sonar fish swarm detection task in this embodiment, SPDConv effectively alleviates the detail loss problem during downsampling, providing higher-quality basic features for subsequent multi-scale feature fusion and accurate detection.

[0033] Fish targets in sonar videos exhibit uneven scale and complex contours. Traditional convolutional sampling positions are fixed, failing to adapt to the dynamic adjustment of the receptive field based on the fish's contour, resulting in a limited perception range that struggles to capture the structural features of small-scale fish. To enhance the feature representation capability of small fish targets, in one implementation, the deformable large-kernel attention layer DLKA utilizes large-kernel convolution to expand the receptive field and acquire spatial contextual features. , ; in, As input features, These represent the height, width, and number of channels of the feature map, respectively. Indicates the kernel size as void ratio Large kernel convolution operation, This represents the obtained attention features. DLKA introduces learnable offsets through deformable convolutions, enabling the sampling position to adaptively adjust according to the target fish morphology. The input features are then recalibrated using the attention map. ; in, This represents the Sigmoid activation function. This represents element-wise multiplication. 2 indicates the enhanced output features. Through "large receptive field modeling + deformable sampling + attention enhancement", DLKA can improve the model's feature representation ability for small fish targets, providing more discriminative features for subsequent head detection prediction.

[0034] Step 3: Perform cross-frame association based on the fish detection results in adjacent sonar frames, and map the fish target as a spatiotemporal node with spatial location, target scale and detection confidence; using a sliding time window as the basic unit, construct a sonar spatiotemporal trajectory map based on the spatiotemporal nodes, including spatial relationships, temporal relationships and overall relationships; In one implementation, to address the challenge that graph structures based on pairwise node connections cannot adequately represent the high-order relationships formed by multiple individuals in fish swarming, synchronous swimming, and overall dispersion, this embodiment introduces a hypergraph to model the movement relationships of fish swarms. A hyperedge in a hypergraph can simultaneously connect multiple fish target nodes, making it more suitable for describing the complex group relationships during fish movement. Specifically, this embodiment constructs nodes based on fish detection results in consecutive sonar frames and adaptively constructs spatial hyperedges, temporal hyperedges, and global hyperedges based on the changes in the spatial distribution and movement state of the fish swarm over time, forming a sonar spatiotemporal trajectory hypergraph. The three types of hyperedges adopt a unified construction rule, but their node composition, hyperedge size, and hypergraph topology are not fixed in advance. Instead, they are dynamically adjusted according to changes in the number of fish, their spatial location, detection confidence, and cross-frame correspondences in different frames and time periods. This organizes the fish targets scattered in consecutive sonar frames into a structured representation that includes local spatial relationships, cross-frame movement continuity, and overall group relationships.

[0035] Please see Figure 3 Suppose a sonar video clip is composed of... The frame consists of consecutive sonar images, the first frame. After SFD-YOLO detection, several fish targets can be obtained from the frame. For the first... The first frame The detection results for each fish body target are expressed as follows: ; in, The coordinates of the center of the bounding box representing the fish target; and Indicates the width and height of the bounding box; This is expressed as the detection confidence level; In this embodiment, each detected fish target is constructed as a node. This node contains the fish target's location, scale, confidence level, and frame number, which simultaneously describes the fish target's spatial location and temporal frame information, and can be represented as: This process maps fish detection boxes, which were originally scattered across consecutive sonar frames, into nodes, providing a foundation for subsequent relationship modeling.

[0036] After the node construction is completed, this embodiment further establishes spatial hyperedges, temporal hyperedges, and global hyperedges. Unlike graph structure modeling where an edge connects only two nodes, a hyperedge in a hypergraph can connect multiple nodes. Therefore, hyperedges are more suitable for describing the relationships such as local aggregation, continuous motion, and overall distribution changes of multiple fish in a school of fish. The sonar spatiotemporal trajectory diagram constructed in this embodiment can be represented as: ; in, This represents the set of nodes consisting of fish-body detection targets; , and These represent the spatial hyperedge set, the temporal hyperedge set, and the global hyperedge set, respectively.

[0037] First, construct the spatial hyperedge. For a node in frame t... The Euclidean distance between a fish and other nodes in the same frame is calculated based on the center coordinates of the fish detection box. To adapt the spatial neighborhood to the fish distribution density at different locations, this paper uses the local nearest neighbor distance of a node to determine its neighborhood range.

[0038] set up Represents a node In the t-th frame The local average nearest neighbor distance and spatial neighborhood radius of the nearest neighbor nodes are respectively expressed as: ; in, Represents a node Distance from the center of other nodes; This is a neighborhood range adjustment coefficient. Based on the node... Centered on the hyperedge, its corresponding spatial hyperedge is constructed as follows: ; in, This represents all fish target nodes in frame t.

[0039] When fish are densely distributed locally, the average nearest neighbor distance between nodes is small, and the spatial neighborhood is correspondingly smaller, which avoids including too many spatially distant fish in the same local unit. When fish are sparsely distributed locally, the average nearest neighbor distance increases, and the spatial neighborhood expands accordingly, thus preserving necessary local correlation information. Therefore, the spatial hyperedge can dynamically adjust its node composition according to the actual distribution of fish in the current frame.

[0040] Secondly, a temporal hyperedge is constructed. Considering that the appearance features of fish in sonar images are weak and easily affected by noise, occlusion, and detection fluctuations, this embodiment integrates the displacement of the fish center, the change in the detection box scale, and the change in detection confidence to match fish targets in adjacent frames.

[0041] For the node in frame t and the nodes in frame t+1 The comprehensive matching cost is defined as: ; in, This represents the normalized center displacement of the two nodes; This indicates the relative change in the width and height of the detection frame; Indicates the difference in detection confidence; , and These are the corresponding weighting coefficients.

[0042] Among candidate nodes that meet the basic association conditions such as maximum displacement and scale change, the node with the minimum comprehensive matching cost is selected as the corresponding target, and a one-to-one matching constraint is adopted to avoid the same node being repeatedly associated. Nodes obtained by continuous matching within a time window are organized into time hyperedges according to time order: ; in, This indicates the number of valid nodes contained in the trajectory within the current time window, and that all adjacent nodes satisfy the above matching conditions.

[0043] The number of nodes and duration of the temporal hyperedge are not fixed in advance, but are determined by the fish detection and cross-frame matching results within the current time period. When the fish target can be continuously and stably detected, the temporal hyperedge contains a large number of nodes and has a long duration; when the fish is briefly occluded, missed, or leaves the sonar observation range, the length of the temporal hyperedge shortens accordingly. Therefore, the temporal hyperedge can be dynamically formed continuously with the actual movement of the fish.

[0044] Finally, spatial hyperedges primarily reflect the local fish swarm structure within a single frame, while temporal hyperedges mainly describe the cross-frame motion trajectory of a single fish. However, relying solely on local spatial relationships and the continuity of individual motion is still insufficient to fully represent the changes in the overall number, coverage area, and organizational structure of the fish swarm over a certain time period. Therefore, this embodiment further constructs an overall hyperedge using a fixed-length time window as the overall analysis unit.

[0045] Suppose that the m-th time window contains L consecutive frames of sonar images, and all fish target nodes in this window constitute a set. Considering that low-confidence detection results, occasional false detections in a single frame, and trajectories with excessively short durations can easily interfere with group characteristics, this embodiment filters reliable fish targets based on node detection confidence and trajectory continuity. The overall hyperedge is represented as: ; in, Indicates the detection confidence threshold; Represents a node The effective length of the trajectory within the m-th time window; This represents the minimum effective trajectory length.

[0046] The above screening process can reduce the impact of low-confidence false detections and short-term unstable trajectories on the overall motion state description. Although the same node screening rules are used for each time window, the number of nodes and the composition of the overall hyperedge will dynamically change with the current fish swarm state due to differences in the number of fish, detection confidence, and trajectory duration within different time windows.

[0047] In summary, the spatial hyperedge, temporal hyperedge, and global hyperedge are dynamically adjusted based on the local distribution of the fish swarm, the continuity of cross-frame motion, and the reliability of nodes, respectively. This enables the sonar spatiotemporal trajectory map to adapt to changes in the movement state of the fish swarm over different time periods, providing a structured basis for subsequent anomaly detection.

[0048] Step 4: Based on the sonar spatiotemporal trajectory map, extract the fish movement features from the spatial hyperedge, temporal hyperedge, and global hyperedge; calculate the normal baseline using the normal movement time period; calculate the deviation between the test time period and the normal baseline, and complete the anomaly judgment based on the threshold determined by the training sample distribution.

[0049] In one implementation, please see Figure 4 After constructing the sonar spatiotemporal trajectory map, this embodiment further extracts fish movement features from the hypergraph structure and uses an unsupervised method for anomaly detection. In actual aquaculture, the number of abnormal fish movement samples is usually small, and the anomaly types are not fixed. If supervised training is carried out entirely based on manually labeled anomaly samples, it is easily affected by insufficient samples and incomplete categories. Therefore, this embodiment does not directly learn a specific abnormal state, but uses the time period of normal fish movement to establish a normal baseline, and then judges whether the fish movement is abnormal based on the degree of deviation between the test time period and the normal baseline.

[0050] For a sonar spatiotemporal trajectory map constructed within a certain time period, this embodiment extracts fish school movement features from the spatial hyperedge, temporal hyperedge, and global hyperedge, respectively. Combining these features yields the fish school movement feature vector for that time period, which can be represented as: ; in, Represents the space hyperedge Extracted features; Indicates the time-bound edge Extracted features; Indicates the global hyperedge Extracted features.

[0051] Since the spatial hyperedge, temporal hyperedge, and global hyperedge are all adaptively constructed based on the number of fish, spatial density, motion continuity, and overall distribution status in the current time period, the features extracted from them can also be adjusted according to the actual movement of the fish swarm. In other words, this embodiment does not obtain a simple statistical representation of the fixed detection box position, but rather a comprehensive expression of the fish swarm movement formed by combining local clustering, motion continuity, and overall distribution information. To avoid the difference in numerical scale between different types of features affecting subsequent distance calculations, this embodiment normalizes the spatial, temporal, and global features before feature fusion, enabling the three types of features to participate in anomaly detection at similar scales.

[0052] During the normal baseline establishment phase, time periods when fish movement was relatively stable and not significantly affected by external interference were selected as training samples, and the mean value of the fish movement characteristics was calculated as the normal baseline for fish movement: ; in, Indicates the first Motion characteristics of a school of fish under normal movement in a single frame; This indicates the number of frames in the training samples; This represents the normal baseline. This baseline can be understood as the common movement pattern of fish in a stable aquaculture environment, i.e., the normal condition. In subsequent anomaly detection, the movement characteristics of the fish during the test period will be compared with this baseline to measure the degree of change relative to the normal condition.

[0053] For the time period to be tested, this embodiment extracts the fish movement characteristics using the same steps and calculates the results compared to the normal baseline. The degree of deviation between them is used as an anomaly score:

[0054] in, Represents Euclidean distance. The smaller the value, the closer the fish movement in that segment is to the normal baseline. The larger the value, the more significant the difference between the fish movement in the current time period and the normal baseline, and the higher the probability of abnormal fish movement.

[0055] In the anomaly detection phase, this embodiment does not use a fixed empirical threshold, but instead determines the threshold based on the anomaly judgment score distribution of normal samples. Specifically, it first calculates the judgment scores of all training samples relative to the normal baseline, and then calculates the mean and standard deviation of these scores. Let the mean and standard deviation of the training sample judgment scores be respectively... and The anomaly detection rules are as follows:

[0056] in, This represents the threshold adjustment coefficient, used to control the sensitivity of anomaly detection; Indicates the first One sonar segment was identified as abnormal; This indicates that the segment is within the normal range. The threshold here is determined by the fluctuation range of normal samples, rather than being arbitrarily fixed, thus adapting to differences in normal movement under different aquaculture environments, fish densities, and sonar acquisition conditions. When the judgment score for the tested time period exceeds the threshold, it indicates that the fish movement in that segment has exceeded the normal fluctuation range and is judged as abnormal; otherwise, it is considered to still be within the normal movement range.

[0057] The unsupervised motion anomaly detection process in this embodiment includes three specific steps: First, based on the adaptively constructed sonar spatiotemporal trajectory map, fish movement features are extracted from spatial hyperedges, temporal hyperedges, and global hyperedges; second, a normal baseline is calculated using the time period of normal movement; finally, the deviation between the test time period and the normal baseline is calculated, and anomaly determination is completed based on a threshold determined by the distribution of training samples. This method does not rely on anomaly sample labeling, focuses on determining whether the current movement of the fish school deviates from the normal state, and can achieve fish school motion anomaly detection from three levels: local aggregation, motion continuity, and overall distribution.

[0058] Through the above process, this embodiment realizes a complete modeling process from sonar fish detection results to sonar spatiotemporal trajectory maps and then to early warning of abnormal fish movement status.

[0059] Fish targets in sonar images are small in scale. When there is a slight positional offset between the predicted bounding box and the ground truth bounding box, traditional IoU-based losses may fluctuate significantly, affecting the stability of small target bounding box regression. Therefore, in one implementation, a joint NWD-WIoU loss is constructed using a dynamic weighting mechanism of Normalized Wasserstein Distance (NWD) and WIoU. .

[0060] Let the predicted bounding box and the ground truth bounding box be represented as follows:

[0061] in, and These represent the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. These represent the width and height of the bounding boxes corresponding to the predicted and ground truth boxes, respectively. but: ; in, This represents the normalized Wasserstein distance loss. Indicates weighted IoU loss. This is the balance coefficient;

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068] in, Represents the second-order Wasserstein distance; This is a normalization constant; and These represent the width and height of the smallest bounding rectangle of the predicted bounding box and the ground truth bounding box, respectively; The exponential moving average representing IoU loss; and Indicates dynamic focusing parameters; This represents a pre-defined minimum positive number; This indicates a gradient truncation operation. During the forward computation, the value of the variable within the parentheses remains unchanged, while during the backpropagation, the gradient corresponding to that variable is set to zero, so that it only participates in the loss calculation as a dynamic weight, and does not participate in the gradient update of the network parameters.

[0069] Predicted box and real frame Both are modeled as two-dimensional Gaussian distributions. The mean of the Gaussian distribution is used as the coordinates of the bounding box center point, and the covariance matrix of the Gaussian distribution is determined by the width and height of the bounding box. The difference between the predicted and ground truth boxes in terms of center position and bounding box scale is comprehensively measured using the second-order Wasserstein distance. This second-order Wasserstein distance is then normalized using an exponential function to obtain the normalized Wasserstein similarity between the predicted and ground truth boxes.

[0070] The invention will be further illustrated by specific experiments below.

[0071] (1) Model Comparison Analysis: To verify the impact of each improved module in the model designed in this invention (Sonar Fish Detection YOLO, SFD-YOLO) on the sonar fish target detection performance, this experiment used YOLO11n as the baseline model, introduced the spatial depth transformation convolution module and the spatial depth transformation convolution module respectively, and added the NWD-WIoU joint loss function on this basis to carry out ablation experiments. The experiment was implemented using Python 3.8 and run on a server. The server hardware environment was: CPU Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz, GPU RTX 3080 Ti (12GB).

[0072] Table 1. Results of SFD-YOLO model ablation experiments

[0073] Table 1 shows that the detection performance of the YOLO11n baseline model changed to varying degrees after introducing different improved modules. The YOLO11n baseline model's Precision, Recall, and mAP@0.5 were 0.866, 0.867, and 0.906, respectively. After introducing the SPDConv module, the model's Precision, Recall, and mAP@0.5 reached 0.865, 0.847, and 0.894, respectively, a slight decrease compared to the baseline model. This result indicates that SPDConv, through its spatial depth transformation mechanism, can retain more local spatial information during downsampling; however, replacing the original downsampling structure alone may affect the overall feature extraction process, causing fluctuations in detection performance. After introducing the DLKA module, the model's Precision improved to 0.887, a 2.1 percentage point increase compared to YOLO11n, while Recall and mAP@0.5 reached 0.858 and 0.907, respectively. Experimental results demonstrate that the deformable large kernel attention mechanism enhances the model's ability to focus on key target regions and improves the response intensity of fish target features. When both SPDConv and DLKA modules are introduced simultaneously, the model's Precision, Recall, and mAP@0.5 reach 0.871, 0.870, and 0.911, respectively, representing a 0.5 percentage point improvement in mAP@0.5 compared to the YOLO11n benchmark model. Compared to using either module alone, the joint model exhibits superior overall detection performance, indicating that SPDConv helps preserve detailed information about the fish target during the feature downsampling stage, while DLKA enhances the feature expression of the target region by expanding the effective receptive range. The two modules complement each other in the feature extraction process.

[0074] The final SFD-YOLO model, formed by combining NWD-WIoU joint bounding box regression loss, achieved Precision, Recall, and mAP@0.5 of 0.878, 0.884, and 0.920, respectively, representing improvements of 1.2, 1.7, and 1.4 percentage points compared to the YOLO11n benchmark model, achieving the best detection results among all models. Experimental results show that the feature enhancement module and the improved bounding box regression strategy can synergistically improve the model's representation ability and localization stability for sonar-detected fish targets, effectively enhancing the accuracy and robustness of fish target detection in complex underwater environments.

[0075] (2) Visualization Analysis of Target Detection Based on Grad-CAM: To further analyze the target interest region of SFD-YOLO in the sonar fish detection process, this experiment uses Grad-CAM to visualize and compare the YOLO11n benchmark model and SFD-YOLO. The results are as follows: Figure 5 As shown.

[0076] Depend on Figure 5 It is evident that the activation regions of YOLO11n are relatively dispersed, and some responses are easily affected by sonar background noise, resulting in insufficient focus on the edges of fish targets and weak target regions. In contrast, Figure 5 The high response region of SFD-YOLO is more concentrated in the fish target and its adjacent edge region, and the response to background noise is relatively weakened, indicating that the improved model in this paper can capture the fish target features in sonar images more effectively.

[0077] (3) Design of simulated anomaly experiments: The anomalies studied in this experiment mainly refer to the phenomenon that, under external disturbance conditions, the spatial distribution, swimming speed, local density, and directional consistency of fish schools deviate significantly from the normal stable swimming conditions. Such anomalies can serve as behavioral representations of external disturbances or aquaculture environmental disturbances, but do not directly correspond to specific water quality stresses or disease types. Due to limitations in pond aquaculture conditions and animal welfare, directly applying extreme stresses such as hypoxia, water pollution, or elevated ammonia nitrogen may have irreversible effects on fish health. Therefore, the experiment constructs an external disturbance scenario by having personnel approach the aquaculture area and causing local water disturbances, in order to simulate the changes in fish swimming caused by equipment operation, pre-harvest disturbances, or environmental disturbances during actual aquaculture.

[0078] Experimental data were collected from aquaculture ponds at the Aquaculture Teaching and Research Base of Huazhong Agricultural University. The movement of fish schools under natural aquaculture conditions was continuously recorded using a Lawrence imaging sonar device. In the initial stage of video acquisition, the fish schools were kept in a relatively stable natural swimming state, and a normal baseline was established using the data before perturbation. Subsequently, a pre-set perturbation operation was performed near frame 450 of the video sequence, causing significant changes in the spatial distribution, local aggregation, and swimming state of the fish schools. The continuous movement process after perturbation was then used as abnormal state data to verify the proposed method's ability to detect abnormal changes in fish school movement.

[0079] To ensure the reliability of the experimental results, multiple perturbation experiments were conducted following the same acquisition procedure. The acquired data were then filtered based on the severity of changes in fish movement, sonar image quality, and data completeness, selecting representative experimental data as the final test data. In the experiment, the stable swimming phase before the perturbation was defined as the normal phase, and the phase after the perturbation where the fish's movement deviated significantly from the normal phase was defined as the abnormal phase. The classification of the abnormal phase comprehensively considered the preset perturbation time, the characteristics of fish behavior changes in the sonar video, and changes in state indicators such as spatial distribution, movement speed, and group structure.

[0080] (4) Ablation Analysis of Experimental Methods: To analyze the impact of different types of hyperedges on the detection results of fish movement anomalies, this experiment conducted ablation experiments on spatial hyperedges, temporal hyperedges, and global hyperedges in the sonar spatiotemporal trajectory map. Three hyperedge combination methods were set up in the experiment: using only spatial hyperedges; adding temporal hyperedges to spatial hyperedges; and using spatial hyperedges, temporal hyperedges, and global hyperedges simultaneously. By gradually increasing the types of hyperedges, the contribution of different levels of motion information to anomaly detection performance was observed. The experimental results are shown in Table 2.

[0081] Table 2. Results of Spatial-Temporal-Global Hyperedge Ablation Experiments

[0082] Table 2 shows that different combinations of hyperedge types significantly affect the performance of anomaly detection in fish swarms. When using only spatial hyperedges, the anomaly detection accuracy, precision, and recall reach 0.943, 0.952, and 0.952, respectively, indicating that the spatial proximity between fish within the same time frame can provide effective anomaly discrimination information. Since abnormal fish movement is usually accompanied by changes in spatial distribution, spatial hyperedges can capture some motion state change features. After adding temporal hyperedges to the spatial hyperedges, the accuracy, precision, and recall are 0.928, 0.950, and 0.927, respectively, a decrease of 1.5, 0.2, and 2.5 percentage points compared to using only spatial hyperedges. This result indicates that while continuous frame motion information can supplement the characteristics of fish displacement and motion continuity, introducing temporal correlations alone may be affected by factors such as sonar detection fluctuations, target matching errors, and short-term occlusion, leading to instability in some cross-frame relationships and thus reducing anomaly detection performance. When global hyperedges were further incorporated, the model performance significantly improved, with Accuracy, Precision, and Recall reaching 0.957, 0.953, and 0.976, respectively. These figures represent improvements of 2.9, 0.3, and 4.9 percentage points compared to the combination of spatial and temporal hyperedges, achieving the best results among the three sets of experiments. This result demonstrates that relying solely on local spatial relationships and the continuity of individual movements is insufficient to fully describe changes in the state of a fish school. Global hyperedges, on the other hand, can characterize the overall distribution range, aggregation degree, and changes in the group structure of the fish school over a period of time, providing more stable constraint information for anomaly detection.

[0083] Experimental results show that spatial hyperedges, temporal hyperedges, and global hyperedges describe the characteristics of fish swarm movement from three levels: local spatial relationships, individual motion continuity, and overall group state, respectively. The hypergraph constructed by the joint construction of these three types of hyperedges can fully integrate motion information at different levels, improving the representation ability of abnormal states in fish swarms. This experiment verifies the effectiveness of the sonar spatiotemporal trajectory map constructed in this invention in the task of detecting abnormal fish swarm movement.

[0084] (5) Anomaly Detection Analysis: Based on the above comparative experiments and super-edge ablation experiments, this experiment further analyzes the changes in the anomaly determination score over time in the test data to verify the responsiveness of the proposed method to changes in the movement state of the fish school. For the test video, the movement characteristics of the fish school within a continuous time period were extracted according to the aforementioned method, and the degree of deviation from the normal baseline was calculated. The results were used as the anomaly determination score. The higher the anomaly determination score, the more obvious the difference in the spatial distribution, local aggregation degree, or swimming pattern of the fish school in the current time period compared to the normal stage; when the anomaly determination score exceeds the set threshold, it is determined that an abnormal movement has occurred in that time period.

[0085] To visually demonstrate the response process of the proposed method to disturbance events, Figure 6 Representative test data were selected to demonstrate the changing trend of anomaly detection scores over time before and after disturbance. By analyzing the changes in the anomaly score curve before and after the disturbance, the model was further validated to effectively capture changes in fish movement patterns and promptly identify abnormal states caused by external disturbances.

[0086] Depend on Figure 6 It can be seen that, under conditions of no human disturbance, the fish abnormality judgment score remained at a low level overall, and did not exceed the set threshold for most of the time period, indicating that the spatial distribution and movement pattern of the fish were relatively stable under normal aquaculture conditions. Figure 6 (a) shows the anomaly detection results under no human disturbance. The anomaly judgment score mainly fluctuates within a certain range, with only a brief increase at individual moments. This indicates that the natural swimming of the fish and detection errors may cause a certain degree of change in movement characteristics, but overall no continuous abnormal state has been formed. Figure 6 (b) Shows the anomaly detection results under artificial perturbation conditions around frame 450. It can be observed that the anomaly detection score significantly increased and exceeded the set threshold before the artificial perturbation marker time, indicating that the fish's movement state had deviated from the normal baseline at this stage. This phenomenon may be related to factors such as personnel approaching, equipment operation, or changes in the aquatic environment before the perturbation. The fish may have sensed external stimuli in advance and generated behavioral responses, causing their movement patterns to change before the artificial perturbation occurred. After the artificial perturbation, the anomaly detection score fluctuated and then increased again in subsequent stages, reflecting the continuous adjustment process of the fish's movement state under the influence of external stimuli. Compared with the undisturbed case, the anomaly detection score in the perturbation experiment showed a larger overall fluctuation range and exceeded the threshold for a longer period, indicating that artificial perturbation enhanced the instability of the fish's movement patterns.

[0087] The above results demonstrate that this invention can effectively capture the deviation of fish school movement from its normal state over a continuous time period and accurately locate time intervals where movement changes are significant. The sonar spatiotemporal trajectory map, integrating spatial correlation and temporal evolution information of the fish school, can form a more complete structured representation, providing a valid basis for anomaly detection.

[0088] It should be understood that the embodiments described above are only some, not all, of the embodiments of the present invention. Furthermore, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0089] It should be understood that the above description of the preferred embodiments is quite detailed, but it should not be considered as a limitation on the scope of protection of this invention. Those skilled in the art, under the guidance of this invention, can make substitutions or modifications without departing from the scope of protection of the claims of this invention, and all such substitutions or modifications fall within the scope of protection of this invention. The scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps, characterized in that, Includes the following steps: Step 1: Obtain a continuous sonar image sequence of fish in the aquaculture area, and perform noise reduction and enhancement preprocessing to obtain the sonar frame to be detected; Step 2: Input the sonar frame to be detected into the trained sonar fish detection network, and output the fish detection results in each frame, including the bounding box position parameters of the fish target and the detection confidence. Step 3: Perform cross-frame association based on the fish detection results in adjacent sonar frames, and map the fish target as a spatiotemporal node with spatial location, target scale and detection confidence. Using a sliding time window as the basic unit, a sonar spatiotemporal trajectory map is constructed based on the spatiotemporal nodes, including spatial relationships, temporal relationships, and overall relationships; Step 4: Based on the sonar spatiotemporal trajectory map, extract the fish swarm movement features from the spatial hyperedge, temporal hyperedge, and global hyperedge; Calculate the normal baseline using the normal movement time period; calculate the deviation between the test time period and the normal baseline, and complete the anomaly judgment based on the threshold.

2. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to claim 1, characterized in that: The noise reduction and enhancement preprocessing described in step 1 involves first performing Gaussian filtering on the original sonar image to smooth the background noise, and then performing filtering to enhance the distinction between the fish body region and the background region. The filtering is based on anisotropic diffusion filtering, which changes the traditional anisotropic diffusion with fixed parameters to adaptive diffusion controlled by local noise intensity and fish body edge information.

3. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to claim 1, characterized in that: The sonar fish detection network described in step 2 includes the first Conv layer, the first SPDConv layer, the first C3K2 layer, the second SPDConv layer, the second C3K2 layer, the third SPDConv layer, the third C3K2 layer, the second Conv layer, the fourth C3K2 layer, the SPPF layer, and the DLKA layer, which are connected in sequence. The output of the DLKA layer is processed by the first upsampling layer, and then fused with the output of the third C3K2 layer by the first concat layer before being input into the fifth C3K2 layer. The output of the fifth C3K2 layer is processed by the second upsampling layer, and then fused with the output of the second C3K2 layer by the second concat layer before being input into the sixth C3K2 layer. The output of the sixth C3K2 layer is processed by the first Detect layer to output a prediction result suitable for small-scale fish target detection, including the bounding box position parameters and class confidence of the fish target; after being processed by the third Conv layer, it is fused with the output of the fifth C3K2 layer by the third Concat layer and then input into the seventh C3K2 layer. The output of the seventh C3K2 layer is processed by the second Detect layer to output a prediction result suitable for mid-scale fish target detection. The prediction result includes the bounding box position parameters and class confidence of the fish target. After processing by the fourth Conv layer, it is fused with the output of the DLKA layer by the fourth Concat layer and then input into the eighth C3K2 layer. Finally, after processing by the third Detect layer, it outputs a prediction result suitable for large-scale fish target detection. The prediction result includes the bounding box position parameters and class confidence of the fish target. The first, second, and third SPDConv layers are all spatial depth transformation convolutional layers, used to rearrange spatial information to the channel dimension during downsampling. The DLKA layer is a deformable large kernel attention layer, used for adaptive feature enhancement of sonar fish targets with weak boundaries, small scale, and irregular shapes. The first, second, third, fourth, fifth, sixth, seventh, and eighth C3K2 layers are all cross-stage feature extraction and fusion modules, used to reuse and fuse features at different levels through branching, convolutional transformation, and channel concatenation, enhancing the local details and semantic feature expression capabilities of sonar fish targets. The SPPF layer is a fast spatial pyramid pooling layer, used to aggregate multi-scale contextual information under different receptive fields, expand the network's receptive field, and enhance the feature expression capabilities of fish targets at different scales.

4. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to claim 3, characterized in that: The spatial depth transformation convolutional layer is applied to the input feature map. Given a spatial rearrangement ratio First, the input feature map is rearranged from spatial to depth. Convert to Then, the output features are obtained through ordinary convolution. ;in, These represent the height, width, and number of channels of the feature map, respectively, and R is a real number field.

5. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to claim 3, characterized in that: The deformable large kernel attention layer is designed for the input feature map. First, large kernel convolution is used to expand the receptive field and obtain spatial context features. ;in, These represent the height, width, and number of channels of the feature map, respectively, where R is a real number. Indicates the kernel size as void ratio Large kernel convolution operation, The obtained attention features are then represented; then the attention map is used to analyze the features. Perform recalibration to obtain enhanced output features. ,in, This represents the Sigmoid activation function. This indicates element-wise multiplication.

6. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to claim 1, characterized in that: In step 3, the construction of the sonar spatiotemporal trajectory map based on the spatiotemporal nodes is performed for the... A sonar video composed of consecutive sonar images is denoted as the first frame. The first frame Fish detection results for individual fish targets ;in, The coordinates of the center of the bounding box representing the fish target; and Indicates the width and height of the bounding box; This is expressed as the detection confidence level; Each detected fish target is constructed as a node. This node contains the location of the fish target. ,scale Confidence level and the frame number It is used to simultaneously describe the spatial location and temporal frame information of the fish target; Constructing a sonar spatiotemporal trajectory map ;in, This represents the set of nodes consisting of detected fish targets; , and Let these represent the set of spatial hyperedges, the set of temporal hyperedges, and the set of global hyperedges, respectively. The spatial hyperedge, for nodes in frame t. ,according to Calculate its Euclidean distance with other nodes in the same frame; let Represents a node In the t-th frame The nearest neighbor node, its local average nearest neighbor distance and spatial neighborhood radius They are represented as follows: ;in, Represents a node Other nodes The center distance; The neighborhood range adjustment coefficient; based on the node Centered on, its corresponding spatial hyperedge is constructed as ;in, This represents all fish target nodes in frame t; The time superedge, for nodes in frame t. and the nodes in frame t+1 Define the comprehensive matching cost as ;in, This represents the normalized center displacement of the two nodes; Indicates the relative changes in the width and height of the bounding box; Indicates the difference in detection confidence; , and The corresponding weighting coefficients are used; among the candidate nodes that satisfy the maximum displacement and scale change, the node with the minimum comprehensive matching cost is selected as the corresponding target, and the nodes obtained by continuous matching within a time window are organized into time hyperedges in chronological order using a one-to-one matching constraint. ;in, This indicates the number of valid nodes contained in the trajectory within the current time window, and that all adjacent nodes satisfy the above matching conditions. The overall hyperedge, if the m-th time window contains L consecutive frames of sonar images, then all fish target nodes in that window constitute a set. Then the overall hyperedge is ;in, Indicates the detection confidence threshold; Represents a node The effective length of the trajectory within the m-th time window; This represents the minimum effective trajectory length.

7. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to claim 1, characterized in that: In step 4, for the sonar spatiotemporal trajectory map constructed within a certain time period, fish movement features are extracted from the spatial hyperedge, temporal hyperedge, and global hyperedge to obtain the fish movement feature vector within that time period. ;in, Represents the space hyperedge Extracted features after normalization; Indicates the time-bound edge Extracted features after normalization; Indicates the global hyperedge Extracted features after normalization.

8. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to claim 7, characterized in that: In step 4, the normal baseline is calculated using a period of normal movement. This involves selecting a time period when the fish movement is stable and undisturbed by external factors as the training sample, and calculating the mean value of the fish movement characteristics during this period as the normal baseline for fish movement. ； in, Indicates the first Motion characteristics of a school of fish under normal movement in a single frame; Indicates the number of frames in the training samples; Indicates the normal reference; Calculate all training samples relative to the normal baseline Judgment score Then calculate the mean of these judgment scores. and standard deviation The exception determination rule is as follows: ； in, Represents Euclidean distance. This represents the threshold adjustment coefficient, used to control the sensitivity of anomaly detection; Indicates the first One sonar segment was identified as abnormal; This indicates that the segment is within the normal range.

9. The method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps according to any one of claims 1-8, characterized in that: In step 2, the trained sonar fish detection network uses a bounding box regression loss function during training. ; Let the predicted bounding box and the ground truth bounding box be represented as follows: in, and These represent the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. These represent the width and height of the bounding boxes corresponding to the predicted and ground truth boxes, respectively. but: ； in, This represents the normalized Wasserstein distance loss. Indicates weighted IoU loss. This is the balance coefficient; in, Represents the second-order Wasserstein distance; This is the normalization constant; and These represent the width and height of the smallest bounding rectangle of the predicted bounding box and the ground truth bounding box, respectively; The exponential moving average representing IoU loss; and Indicates dynamic focusing parameters; This represents a pre-defined minimum positive number; This indicates a gradient truncation operation, which keeps the value of the variable within the parentheses unchanged during the forward propagation, and sets the gradient of that variable to zero during the backpropagation.

10. A fish school movement anomaly early warning system based on sonar spatiotemporal trajectory mapping, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method for early warning of abnormal fish movement based on sonar spatiotemporal trajectory maps as described in any one of claims 1 to 9.

Images