A method, device and system for tracking and monitoring individual fish

Abstract

Embodiments of the present application provide a fish population individual tracking and monitoring method and system, belonging to the field of artificial intelligence technology. The method comprises: acquiring video data, voiceprint features and environmental parameters of fish; constructing a multi-dimensional tensor data stream according to the video data, voiceprint features and environmental parameters, wherein the multi-dimensional tensor data stream comprises video frames, voiceprint vectors and environmental parameter vectors; using a preset hybrid architecture model to obtain a bounding box and a fish body visual feature vector according to the video frames; splicing the fish body visual feature vector, the voiceprint vector and the environmental state vector into a fusion vector; calculating the similarity of the fusion vector; filtering the fusion vector according to the similarity; and performing a fish population individual tracking operation according to the filtered fusion vector. Compared with the commonly used single feature tracking operation, the method, device and system provided by the present application achieve more accurate counting and tracking technical effects due to the use of multiple dimensions.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a method, apparatus, and system for tracking and monitoring individual fish populations. Background Technology

[0002] Fish counting is an important statistical method in the fish farming industry. Current fish counting algorithms suffer from the following drawbacks:

[0003] (1) Single-modal limitation: Most existing systems rely on single-modal solutions such as optical cameras (e.g., RGB images) or sonar, which result in high rates of missed detection and false detection in scenarios with unclear water quality, low light, or dense fish populations.

[0004] (2) Lack of image distortion correction: Underwater wide-angle lenses cause distortion of the fish body edges, and the bounding box positioning error of traditional detection models is large;

[0005] (3) Limited dynamic occlusion handling capability: When the fish are densely packed, occlusion can easily occur between individuals, leading to tracking failure or ID switching problems;

[0006] (4) Real-time challenge: In order to achieve high accuracy, the model often requires high computing resources, which makes it difficult to achieve real-time monitoring on low-configuration devices. Summary of the Invention

[0007] The purpose of this invention is to provide a method, apparatus, and system for tracking and monitoring individual fish populations, which can improve the statistical accuracy of fish population counting.

[0008] To achieve the above objectives, embodiments of the present invention provide a method for tracking and monitoring individual fish in a school, comprising:

[0009] Acquire video data, voiceprint features, and environmental parameters of fish;

[0010] A multidimensional tensor data stream is constructed based on the video data, voiceprint features, and environmental parameters, wherein the multidimensional tensor data stream includes video frames, voiceprint vectors, and environmental parameter vectors;

[0011] A preset hybrid architecture model is used to obtain bounding boxes and fish visual feature vectors based on the video frames;

[0012] The fish visual feature vector, voiceprint vector, and environmental state vector are concatenated into a fusion vector.

[0013] Calculate the similarity of the fused vectors;

[0014] The fusion vector is selected based on the similarity.

[0015] The individual fish in the school are tracked based on the fused vectors after filtering.

[0016] Optionally, a multidimensional tensor data stream is constructed based on the video data, voiceprint features, and environmental parameters, including:

[0017] A dynamic Gaussian kernel is used to eliminate high-frequency bubble noise in the video data;

[0018] The low-frequency water ripple disturbance in the video data is processed using a nonlocal mean filtering method.

[0019] The U-Net semantic segmentation model was used to remove suspended matter from the video data.

[0020] Optionally, the hybrid architecture model includes:

[0021] Multiple feature extraction units are connected in series. The first feature extraction unit is used to receive the input video frame and perform feature extraction, while the remaining feature extraction units are used to receive the output of the previous feature extraction unit and perform feature extraction.

[0022] A feature combination and mapping unit, connected to each of the feature extraction units, is used to combine the outputs of each feature extraction unit and perform feature mapping operations to obtain the bounding box and the visual feature vector of the fish body.

[0023] Optionally, the feature extraction unit includes:

[0024] The deformable convolution enhancement module is used to receive input features and perform deformable convolution enhancement operations.

[0025] The distortion adaptive module is used to correct distortion in the results of deformable convolution enhancement operations;

[0026] The EfficientViT encoder is used to encode the results of distortion correction.

[0027] Optionally, the deformable convolution enhancement module is used for:

[0028] Perform the deformable convolution enhancement operation according to formula (1):

[0029] (1)

[0030] in, The result of deformable convolution enhancement operation is denoted as any point on the feature map. , For the first Dynamic weights of each sampling point The number of sampling points, On the feature map The eigenvalue at the location, This represents the offset of each sampling point on the convolution kernel relative to the center point. For the network in The offset learned from the foundation.

[0031] Optionally, the distortion adaptive module is used for:

[0032] The attention weights are determined according to formula (2):

[0033] (2)

[0034] in, radial and angle Attention weights This represents the radial query feature after polar coordinate transformation. For coordinate transformation functions, These are the radial bond features after polar coordinate transformation. The feature of the angle query key after polar coordinate transformation. The angular key features are after polar coordinate transformation. For the corresponding K The dimension of the feature;

[0035] Determine the polar coordinates according to formula (3):

[0036] (3)

[0037] in, These are the coordinates in the Cartesian coordinate system. and These are the coordinates corresponding to the x-axis and y-axis;

[0038] A weighted operation is performed based on the attention weights and polar coordinates;

[0039] The result of the weighted operation is then subjected to a coordinate transformation to obtain the feature in Cartesian coordinate form.

[0040] Optionally, the fish visual feature vector, voiceprint vector, and environmental state vector are concatenated into a fusion vector, including:

[0041] The fish visual feature vector, voiceprint vector, and environmental state vector are concatenated into a high-dimensional vector.

[0042] Determine the attention weights of the high-dimensional vector;

[0043] The attention weights and the high-dimensional vector are weighted together.

[0044] The result of the weighting operation is input into the fully connected layer to obtain the fusion vector.

[0045] Optionally, filtering the fusion vector based on the similarity includes:

[0046] A preset modal verification method, wherein the modal verification method includes: visual confidence greater than a first preset value, acoustic matching degree less than a second preset value, and / or parameter stability value less than a third preset value;

[0047] The fusion vector is determined using the Hungarian algorithm based on the modality verification method.

[0048] On the other hand, the present invention also provides a fish school individual tracking and monitoring device, the device comprising:

[0049] Image acquisition unit, used to acquire video data of underwater fish schools;

[0050] Hydrophones are used to collect underwater sound data;

[0051] Environmental sensors are used to collect underwater environmental parameters;

[0052] Processor, used for:

[0053] The sound data is converted into voiceprint features;

[0054] As described in any of the methods above.

[0055] In another aspect, the present invention also provides a fish school individual tracking and monitoring system, the system including a processor for performing any of the methods described above.

[0056] Through the above technical solutions, the embodiments of the present invention provide a method, device and system for tracking and monitoring individual fish. The method, device and system perform individual tracking operations by combining video data, voiceprint features and environmental parameter vectors. Compared with the single feature tracking operations in the prior art, the method, device and system provided by the present invention achieve a more accurate counting and tracking effect due to the use of multiple dimensions.

[0057] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0058] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0059] Figure 1 This is a flowchart of a method for tracking and monitoring individual fish in a school according to an embodiment of the present invention;

[0060] Figure 2 This is a structural block diagram of a hybrid architecture model according to an embodiment of the present invention;

[0061] Figure 3 This is a structural block diagram of a hybrid architecture model according to an embodiment of the present invention;

[0062] Figure 4 This is a structural block diagram of a hybrid architecture model according to an embodiment of the present invention;

[0063] Figure 5 This is a structural block diagram of a feature extraction unit according to an embodiment of the present invention;

[0064] Figure 6 This is a structural block diagram of a deformable convolution enhancement module according to an embodiment of the present invention;

[0065] Figure 7 This is a structural block diagram of a distortion adaptive module according to an embodiment of the present invention;

[0066] Figure 8 This is a structural block diagram of a fish school individual tracking and monitoring device according to one embodiment of the present invention. Detailed Implementation

[0067] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0068] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application all comply with relevant laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0069] like Figure 1 The diagram shows a flowchart of a method for tracking and monitoring individual fish in a school according to an embodiment of the present invention. Figure 1 In this context, the method may include:

[0070] In step S10, video data, voiceprint features, and environmental parameters of the fish are acquired;

[0071] In step S11, a multidimensional tensor data stream is constructed based on video data, voiceprint features, and environmental parameters. The multidimensional tensor data stream includes video frames, voiceprint vectors, and environmental parameter vectors.

[0072] In step S12, a preset hybrid architecture model is used to obtain bounding boxes and fish visual feature vectors based on video frames;

[0073] In step S13, the fish visual feature vector, voiceprint vector, and environmental state vector are concatenated into a fusion vector;

[0074] In step S14, the similarity of the fused vectors is calculated;

[0075] In step S15, the fusion vector is selected based on similarity;

[0076] In step S16, individual fish tracking is performed based on the filtered fusion vector.

[0077] In such Figure 1 In the method shown, step S10 can be used to acquire video data, acoustic signature features, and environmental parameters of the fish. The video data can be acquired using an underwater image acquisition unit. Specifically, the image acquisition device can be, for example, a high frame rate underwater camera. The acoustic signature features can be obtained by using an underwater hydrophone. The environmental parameters can be acquired by using multiple sensors installed underwater. Specifically, these sensors can be, for example, temperature sensors, dissolved oxygen sensors, and salinity sensors.

[0078] Step S11 can be used to construct a multidimensional tensor data stream based on video data, voiceprint features, and environmental parameters. Specifically, considering that underwater video data may contain many interference factors, in one example of this invention, when processing the video data, a dynamic Gaussian kernel can be used to eliminate high-frequency bubble noise in the video data, a nonlocal mean filtering method can be used to process low-frequency water ripple disturbances in the video data, and a U-Net semantic segmentation model can be used to remove suspended matter in the video data. It should be emphasized here that the execution order of using a dynamic Gaussian kernel to eliminate high-frequency bubble noise in the video data, using a nonlocal mean filtering method to process low-frequency water ripple disturbances in the video data, and using a U-Net semantic segmentation model to remove suspended matter in the video data is only an example to supplement the explanation of this invention and does not limit the execution order of the three. Those skilled in the art should understand that different execution orders of the above three are also within the protection scope of the technical solution of this invention.

[0079] Step S12 can be used to obtain bounding boxes and fish visual feature vectors based on video frames using a preset hybrid architecture model. The specific structure of this hybrid architecture model can be of various forms known to those skilled in the art. Considering the multi-dimensional feature combination used in this invention, in one example of this invention, such as... Figure 2As shown, the hybrid architecture model can include multiple feature extraction units 1 and feature combination and mapping units 2 connected in series. The first feature extraction unit 1 can receive the input video frame and perform feature extraction, while the remaining feature extraction units 1 receive the output of the previous feature extraction unit 1 and perform feature extraction. The feature combination and mapping unit 2 can be connected to each feature extraction unit 1 to combine the output of each feature extraction unit 1 and perform feature mapping operations to obtain bounding boxes and fish visual feature vectors. Further, to achieve stepwise feature extraction of video frames, in one example of the present invention, such as... Figure 3 As shown, the feature extraction unit 1 may further include a deformable convolution enhancement module 11 (ConvNeXt Block), a distortion adaptation module 12 (Destortion module), and an EfficientViT encoder 13 (EfficientViT Block). The deformable convolution enhancement module 11 can receive input features and perform deformable convolution enhancement operations. The specific method for this deformable convolution enhancement operation can be of various forms known to those skilled in the art. In one example of the present invention, the deformable convolution enhancement module 11 can perform the deformable convolution enhancement operation using the following formula (1):

[0080] (1)

[0081] in, The result of deformable convolution enhancement operation is denoted as any point on the feature map. , For the first Dynamic weights of each sampling point The number of sampling points, On the feature map The eigenvalue at the location, This represents the offset of each sampling point on the convolution kernel relative to the center point. For the network in The offset learned from the foundation.

[0082] Furthermore, considering that the input conditions for each feature extraction unit 1 are different, using only the exact same feature extraction unit 1 would result in relatively poor feature extraction performance. Therefore, in one example of this invention, such as... Figure 4 As shown, in Figure 3When the number of feature extraction units 1 shown is 4, the first feature extraction unit 1 can include 3 deformable convolutional enhancement modules 11, which are connected in series at the front end of the distortion adaptive module 12. The second feature extraction unit 11 can include 3 deformable convolutional enhancement modules 11, which are connected in series at the front end of the distortion adaptive module 12. The third feature extraction unit 11 can include 9 deformable convolutional enhancement modules 11, which are connected in series at the front end of the distortion adaptive module 12. The fourth feature extraction unit 11 can include 3 deformable convolutional enhancement modules 11, which are connected in series at the front end of the distortion adaptive module 12.

[0083] The specific structure of each deformable convolutional enhancement module 11 can be of various forms known to those skilled in the art, including but not limited to structures obtained by concatenating single or multiple convolutional or pooling layers. In one example of the present invention, considering the preservation of original features and the fusion of processed features, the deformable convolutional enhancement module 11 may include, for example... Figure 5 The structure shown is described. In this... Figure 5 The deformable convolution enhancement module 11 may include a normalization layer (LN) 111, a convolutional layer 112, a GRU layer 113, a deformable convolutional layer 114, and a stacking layer 115. The normalization layer 111, convolutional layer 112, GRU layer 113, and deformable convolutional layer 114 are sequentially connected. The normalization layer 111 is used to normalize the input features. The convolutional layer 112 is used to perform a convolution operation on the result of the normalization operation. The GRU layer 113 is used to expand the result of the convolution operation. The deformable convolutional layer 114 is used to perform a deformable convolution operation on the expanded result. The stacking layer 115 is used to stack the input of the normalization layer 111 and the result of the deformable convolution operation to obtain the output of the deformable convolution enhancement operation.

[0084] The distortion adaptive module 12 can be used to correct distortion of the results of deformable convolution enhancement operations. The specific method for distortion correction can be of various forms known to those skilled in the art. In one example of the present invention, the distortion correction method can be to first map the original feature map to a distortion-invariant space using polar coordinate transformation, then determine the attention weight of each sampling point by calculating the attention weights in the radial and angular directions, and finally complete the distortion correction operation by weighting the attention weights with the original sampling point values. Specifically, in this example, the distortion adaptive module 12 can determine the attention weights using formula (2):

[0085] (2)

[0086] in, radial and angle Attention weights This represents the radial query feature after polar coordinate transformation. For coordinate transformation functions, These are the radial bond features after polar coordinate transformation. The feature of the angle query key after polar coordinate transformation. The angular key features are after polar coordinate transformation. For the corresponding K The dimensions of the features.

[0087] To facilitate weighted operations, the distortion adaptive module 12 can also determine the polar coordinates using the following formula (3):

[0088] (3)

[0089] in, These are the coordinates in the Cartesian coordinate system. and These are the coordinates corresponding to the x-axis and y-axis.

[0090] After the weighted calculation is completed, the distortion adaptive module 12 can perform a coordinate transformation operation on the result of the weighted operation (e.g., through the inverse operation of formula (3)) to obtain the feature in Cartesian coordinate form. The structure of the distortion adaptive module 12 can also be, for example... Figure 6 As shown.

[0091] The EfficientViT encoder 13 can be used to encode the results of distortion correction.

[0092] Step S13 can be used to concatenate the fish's visual feature vector, voiceprint vector, and environmental state vector into a fusion vector. The specific method for concatenating the fusion vector can be of various forms known to those skilled in the art, including but not limited to vector combination, vector addition, and other methods known to those skilled in the art. In one example of this invention, the method for concatenating the fusion vector can be as follows: first, concatenate the fish's visual feature vector, voiceprint vector, and environmental state vector into a high-dimensional vector; then, determine the attention weights of the high-dimensional vector; then, perform a weighted operation on the attention weights and the high-dimensional vector; finally, input the result of the weighted operation into a fully connected layer to obtain the fusion vector. The corresponding logic diagram is as follows: Figure 7 As shown.

[0093] Step S14 can be used to calculate the similarity of the fused vectors. The specific method for calculating this similarity can be any form known to those skilled in the art, including but not limited to cosine similarity, Euclidean distance, etc.

[0094] Step S15 can be used to filter fusion vectors based on similarity. Specifically, in one example of the present invention, in order to ensure the accuracy of similarity, the present invention can also use the appearance similarity and motion similarity methods commonly used in the prior art, and combine them with the similarity of the fusion vectors to make a combined judgment. The detection box-trajectory matching process is completed by using a preset modality verification method and the Hungarian algorithm. Then, the visual confidence is greater than a first preset value (e.g., 0.7), the acoustic matching degree is less than a second preset value (e.g., 0.5), and / or the parameter stability value is less than a third preset value (e.g., 0.3℃ / s).

[0095] Step S16 can be used to perform individual fish tracking operations based on the filtered fusion vector. This individual fish tracking operation can take various forms known to those skilled in the art. In one example of the invention, considering the characteristics of the fish's environment, a virtual detection line can be set at a preset location in the fishpond. When the center point of the individual fish tracking trajectory crosses this line for the first time, a count is triggered, and the acoustic signal intensity is checked in conjunction with the stability of environmental parameters to complete the individual tracking count. If the acoustic signal remains silent or the environmental parameters remain unchanged, it is judged as an abnormal scenario, the data is recorded to trigger manual review, and backtracking corrections are made based on historical trajectories.

[0096] Furthermore, after implementing individual fish tracking, a pre-defined display interface system can be used to more intuitively showcase the results. Specifically, in one example of this invention, this display interface system can be a cross-platform visualization interface developed based on the Qt framework to meet the real-time monitoring needs of aquaculture sites. It is divided into four main functional modules:

[0097] 1) Environmental parameter panel: Dynamically displays real-time values ​​and historical trend curves of temperature, dissolved oxygen, and salinity (sampling interval 1 second). Abnormal values ​​(such as dissolved oxygen <4mg / L) will automatically trigger a red warning flashing.

[0098] 2) Video monitoring interface: An OpenGL-accelerated video stream is embedded in the central area, overlaid with target detection bounding boxes and individual motion trajectories (the path of the most recent 30 seconds is drawn with gradient lines), and it supports clicking on the target to view detailed information such as ID and speed.

[0099] 3) Fish swarm statistics dashboard: Displays the number of fish detected in the current frame using a digital counter, and integrates historical quantity change curves.

[0100] 4) Trajectory Analysis Module: The bottom of the module displays a heat map showing the density distribution of the fish population, generated by a kernel density estimation algorithm; it also provides a trajectory playback function, allowing users to drag the timeline to view the movement pattern at any given moment.

[0101] On the other hand, the present invention also provides a fish individual tracking and monitoring device, such as... Figure 8 As shown, the device may include an image acquisition unit 3, a hydrophone 4, an environmental sensor 5, and a processor 6. The image acquisition unit 3 can be used to acquire video data of underwater fish schools. The hydrophone 4 can be used to acquire underwater sound data. The environmental sensor 5 can be used to acquire underwater environmental parameters. The processor 6 can be used to convert the sound data into acoustic signature features and perform functions such as... Figure 1 The method shown is as follows.

[0102] In another aspect, the present invention also provides a fish school individual tracking and monitoring system, the system including a processor for performing any of the methods described above.

[0103] Through the above technical solutions, the embodiments of the present invention provide a method, device and system for tracking and monitoring individual fish. The method, device and system perform individual tracking operations by combining video data, voiceprint features and environmental parameter vectors. Compared with the single feature tracking operations in the prior art, the method, device and system provided by the present invention achieve a more accurate counting and tracking effect due to the use of multiple dimensions.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0109] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0110] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0112] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method of tracking and monitoring individual fish, characterized by, include: Acquire video data, voiceprint features, and environmental parameters of fish; A multidimensional tensor data stream is constructed based on the video data, voiceprint features, and environmental parameters, wherein the multidimensional tensor data stream includes video frames, voiceprint vectors, and environmental parameter vectors; A preset hybrid architecture model is used to obtain bounding boxes and fish visual feature vectors based on the video frames; The fish visual feature vector, voiceprint vector, and environmental state vector are concatenated into a fusion vector. Calculate the similarity of the fused vectors; The fusion vector is selected based on the similarity. The fish swarm individual tracking operation is performed based on the filtered fusion vector; The hybrid architecture model includes: Multiple feature extraction units are connected in series. The first feature extraction unit is used to receive the input video frame and perform feature extraction, while the remaining feature extraction units are used to receive the output of the previous feature extraction unit and perform feature extraction. A feature combination and mapping unit, connected to each of the feature extraction units, is used to combine the outputs of each feature extraction unit and perform feature mapping operations to obtain the bounding box and the visual feature vector of the fish body. The feature extraction unit includes: The deformable convolution enhancement module is used to receive input features and perform deformable convolution enhancement operations. The distortion adaptive module is used to correct distortion in the results of deformable convolution enhancement operations; The EfficientViT encoder is used to encode the results of distortion correction. The distortion adaptive module is used for: The attention weights are determined according to formula (2): ，（2） in, radial and angle Attention weights This represents the radial query feature after polar coordinate transformation. For coordinate transformation functions, These are the radial bond features after polar coordinate transformation. The feature of the angle query key after polar coordinate transformation. The angular key features are after polar coordinate transformation. For the corresponding K The dimension of the feature; Determine the polar coordinates according to formula (3): ，（3） in, These are the coordinates in the Cartesian coordinate system. and These are the coordinates corresponding to the x-axis and y-axis; A weighted operation is performed based on the attention weights and polar coordinates; The result of the weighted operation is then subjected to a coordinate transformation to obtain the feature in Cartesian coordinate form.

2. The method according to claim 1, characterized in that, A multidimensional tensor data stream is constructed based on the video data, voiceprint features, and environmental parameters, including: A dynamic Gaussian kernel is used to eliminate high-frequency bubble noise in the video data; The low-frequency water ripple disturbance in the video data is processed using a nonlocal mean filtering method. The U-Net semantic segmentation model was used to remove suspended matter from the video data.

3. The method according to claim 1, characterized in that, The deformable convolution enhancement module is used for: Perform the deformable convolution enhancement operation according to formula (1): ，（1） in, The result of deformable convolution enhancement operation is denoted as any point on the feature map. , For the first Dynamic weights of each sampling point The number of sampling points, On the feature map The eigenvalue at the location, This represents the offset of each sampling point on the convolution kernel relative to the center point. For the network in The offset learned from the foundation.

4. The method according to claim 1, characterized in that, The fish's visual feature vector, voiceprint vector, and environmental state vector are concatenated into a fusion vector, including: The fish visual feature vector, voiceprint vector, and environmental state vector are concatenated into a high-dimensional vector. Determine the attention weights of the high-dimensional vector; The attention weights and the high-dimensional vector are weighted together. The result of the weighting operation is input into the fully connected layer to obtain the fusion vector.

5. The method according to claim 1, characterized in that, Filtering the fusion vector based on the similarity includes: A preset modal verification method, wherein the modal verification method includes: visual confidence greater than a first preset value, acoustic matching degree less than a second preset value, and / or parameter stability value less than a third preset value; The fusion vector is determined using the Hungarian algorithm based on the modality verification method.

6. A device for tracking and monitoring individual fish populations, characterized in that, The device includes: Image acquisition unit, used to acquire video data of underwater fish schools; Hydrophones are used to collect underwater sound data; Environmental sensors are used to collect underwater environmental parameters; Processor, used for: The sound data is converted into voiceprint features; The method for tracking and monitoring individual fish groups as described in any one of claims 1 to 5.

7. A fish school individual tracking and monitoring system, characterized in that, The system includes a processor for performing the individual fish tracking and monitoring method as described in any one of claims 1 to 5.

Images