Fish school individual tracking and monitoring method, device and system

Abstract

The embodiment of the invention provides a fish school individual tracking and monitoring method and system, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring video data, voiceprint features and environmental parameters of fishes; a multi-dimensional tensor data stream is constructed according to the video data, the voiceprint features and the environmental parameters, and the multi-dimensional tensor data stream comprises video frames, voiceprint vectors and environmental parameter vectors; adopting a preset hybrid architecture model to obtain a bounding box and a fish body visual feature vector according to the video frame; splicing the fish body visual feature vector, the voiceprint vector and the environment state vector into a fusion vector; calculating the similarity of the fusion vectors; screening the fusion vector according to the similarity; and performing fish school individual tracking operation according to the screened fusion vector. Compared with the common single feature tracking operation, the method, the device and the system provided by the invention have the advantages that the technical effect of more accurate counting tracking is realized due to the adoption of multiple dimensions.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device and system for tracking and monitoring individual fish schools. Background Art

[0002] Fish counting is an important statistical method in the fish farming industry. Current fish counting algorithms have the following drawbacks: (1) Single modality limitation: Most existing systems rely solely on single-modality solutions such as optical cameras (e.g., RGB images) or sonar. These systems suffer from high rates of missed detection and false detection in scenarios with unclear water quality, low light conditions, or dense fish populations. (2) Lack of image distortion correction: Underwater wide-angle lenses cause distortion of the fish's edges, and the bounding box positioning error of traditional detection models is large; (3) Limited dynamic occlusion processing capabilities: When fish schools are dense, occlusion is likely to occur between individuals, resulting in tracking failure or ID switching problems; (4) Real-time challenge: To achieve high accuracy, the model often requires high computing resources, which makes it difficult to implement real-time monitoring on low-configuration devices. Summary of the Invention

[0003] The purpose of the embodiments of the present invention is to provide a method, device and system for tracking and monitoring individual fish schools, which can improve the statistical accuracy of fish school counting.

[0004] To achieve the above objectives, an embodiment of the present invention provides a method for tracking and monitoring individual fish schools, comprising: Obtain video data, sound pattern characteristics, and environmental parameters of fish; Constructing a multidimensional tensor data stream based on the video data, voiceprint features, and environmental parameters, wherein the multidimensional tensor data stream includes video frames, voiceprint vectors, and environmental parameter vectors; A preset hybrid architecture model is used to obtain a bounding box and a fish body visual feature vector according to the video frame; splicing the fish body visual feature vector, the soundprint vector and the environment state vector into a fusion vector; Calculating the similarity of the fused vectors; screening the fusion vector according to the similarity; The individual fish tracking operation is performed according to the filtered fusion vector.

[0005] Optionally, constructing a multidimensional tensor data stream based on the video data, voiceprint features, and environmental parameters includes: Using a dynamic Gaussian kernel to eliminate high-frequency bubble noise in the video data; Using a non-local mean filtering method to process the low-frequency disturbance of water ripples in the video data; The U-Net semantic segmentation model is used to remove suspended matter in water from the video data.

[0006] Optionally, the hybrid architecture model includes: Multiple feature extraction units are connected in series, wherein the first feature extraction unit is used to receive the input video frame and perform feature extraction, and the remaining feature extraction units are used to receive the output of the previous feature extraction unit and perform feature extraction; The feature combination and mapping unit is connected to each of the feature extraction units and is used to combine the output of each feature extraction unit and perform a feature mapping operation to obtain the bounding box and the fish body visual feature vector.

[0007] Optionally, the feature extraction unit includes: The deformable convolution enhancement module is used to receive input features and perform deformable convolution enhancement operations; Distortion adaptive module, used to correct the distortion of the results of the deformable convolution enhancement operation; The EfficientViT encoder is used to encode the distortion correction results.

[0008] Optionally, the deformable convolution enhancement module is used to: The deformable convolution enhancement operation is performed according to formula (1): , (1) in, is the result of the deformable convolution enhancement operation, and any point on the feature map is recorded as , For the The dynamic weight of the sampling points, is the number of sampling points, On the feature map The eigenvalue at position, is the offset of each sampling point on the convolution kernel relative to the center point, For the network The offset learned based on .

[0009] Optionally, the distortion adaptation module is used to: Determine the attention weight according to formula (2): , (2) in, Radial and angles The attention weight, is the radial query feature after polar coordinate transformation, is the coordinate transformation function, is the radial bond feature after polar coordinate transformation, is the angle query key feature after polar coordinate transformation, is the angle key feature after polar coordinate transformation, To correspond K Dimensionality of features; Determine the polar coordinates according to formula (3): , (3) in, are coordinates in the Cartesian coordinate system, and are the coordinates corresponding to the x-axis and y-axis; Perform a weighted operation according to the attention weight and polar coordinates; The result of the weighted operation is subjected to a coordinate transformation operation to obtain features in the form of a Cartesian coordinate system.

[0010] Optionally, the fish body visual feature vector, the voiceprint vector, and the environment state vector are spliced ​​into a fusion vector, including: Splicing the fish body visual feature vector, the soundprint vector, and the environment state vector into a high-dimensional vector; Determining an attention weight for the high-dimensional vector; Performing a weighted operation on the attention weight and the high-dimensional vector; The result of the weighted operation is input into the fully connected layer to obtain the fusion vector.

[0011] Optionally, screening the fusion vector according to the similarity includes: A preset modal verification method, wherein the modal verification method includes: a visual confidence level greater than a first preset value, an acoustic matching level less than a second preset value, and / or a parameter stability value less than a third preset value; The fusion vector is determined using the Hungarian algorithm according to the modal verification method.

[0012] On the other hand, the present invention also provides a device for tracking and monitoring individual fish schools, the device comprising: An image acquisition unit, used for collecting video data of underwater fish schools; hydrophones, used to collect underwater sound data; Environmental sensors, used to collect underwater environmental parameters; Processor for: Converting the sound data into voiceprint features; As any of the methods above.

[0013] On the other hand, the present invention further provides a system for tracking and monitoring individual fish schools, the system comprising a processor, and the processor is configured to execute any of the methods described above.

[0014] Through the above technical solution, the embodiment of the present invention provides a method, device and system for tracking and monitoring individual fish schools. The method, device and system perform individual tracking operations by combining video data, voiceprint features and environmental parameter vectors. Compared with the single feature tracking operation in the prior art, the method, device and system provided by the present invention achieve a more accurate counting and tracking technical effect due to the use of multiple dimensions.

[0015] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings: Figure 1 is a flow chart of a method for tracking and monitoring individual fish schools according to one embodiment of the present invention; Figure 2 is a structural block diagram of a hybrid architecture model according to one embodiment of the present invention; Figure 3 is a structural block diagram of a hybrid architecture model according to one embodiment of the present invention; Figure 4 is a structural block diagram of a hybrid architecture model according to one embodiment of the present invention; Figure 5 is a structural block diagram of a feature extraction unit according to one embodiment of the present invention; Figure 6 is a structural block diagram of a deformable convolution enhancement module according to one embodiment of the present invention; Figure 7 is a structural block diagram of a distortion adaptation module according to an embodiment of the present invention; Figure 8 It is a structural block diagram of a device for tracking and monitoring individual fish schools according to one embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following describes the specific implementation of the embodiment of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present invention and is not used to limit the embodiment of the present invention.

[0018] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use such solutions.

[0019] like Figure 1 The figure shows a flow chart of a method for tracking and monitoring individual fish schools according to one embodiment of the present invention. Figure 1 In, the method may include: In step S10, video data, voiceprint characteristics and environmental parameters of fish are obtained; In step S11, a multidimensional tensor data stream is constructed based on the video data, voiceprint features, and environmental parameters, wherein the multidimensional tensor data stream includes video frames, voiceprint vectors, and environmental parameter vectors; In step S12, a preset hybrid architecture model is used to obtain a bounding box and a fish body visual feature vector based on the video frame; In step S13, the fish body visual feature vector, the voiceprint vector and the environment state vector are spliced ​​into a fusion vector; In step S14, the similarity of the fused vectors is calculated; In step S15, the fusion vector is selected based on the similarity; In step S16, individual fish tracking operation is performed based on the filtered fusion vector.

[0020] In this Figure 1 In the illustrated method, step S10 can be used to obtain video data, vocal characteristics, and environmental parameters of the fish. The video data can be acquired by using an underwater image acquisition unit. Specifically, the image acquisition unit can be, for example, a high-frame-rate underwater camera. The vocal characteristics can be acquired by using an underwater hydrophone. Environmental parameters can be acquired by using various sensors underwater. Specifically, these sensors can be, for example, temperature sensors, dissolved oxygen sensors, and salinity sensors.

[0021] Step S11 can be used to construct a multidimensional tensor data stream based on video data, voiceprint features and environmental parameters. Specifically, considering that there are many interference factors in the video data collected underwater, in one example of the present invention, when processing the video data, a dynamic Gaussian kernel can be used to eliminate high-frequency bubble noise in the video data, and a non-local mean filtering method can be used to process the low-frequency disturbance of water ripples in the video data, while a U-Net semantic segmentation model is used to remove suspended water matter in the video data. It should be emphasized here that the execution order of using a dynamic Gaussian kernel to eliminate high-frequency bubble noise in the video data, using a non-local mean filtering method to process the low-frequency disturbance of water ripples in the video data, and using a U-Net semantic segmentation model to remove suspended water matter in the video data is only an example for supplementing the explanation of the present invention, and does not limit the execution order of the three. Those skilled in the art should know that the different execution orders of the above three also fall within the scope of protection of the technical solution of the present invention.

[0022] Step S12 can be used to obtain the bounding box and fish body visual feature vector based on the video frame using a preset hybrid architecture model. The specific structure of the hybrid architecture model can be in various forms known to those skilled in the art. Considering the feature combination of multiple dimensions used in the present invention, in one example of the present invention, Figure 2 As shown, the hybrid architecture model may include multiple feature extraction units 1 and feature combination and mapping units 2 connected in series. Among them, the first feature extraction unit 1 can be used to receive the input video frame and perform feature extraction, and the remaining feature extraction units 1 are used to receive the output of the previous feature extraction unit 1 and perform feature extraction. The feature combination and mapping unit 2 can be connected to each feature extraction unit 1 to combine the output of each feature extraction unit 1 and perform feature mapping operations to obtain a bounding box and a fish body visual feature vector. Furthermore, in order to achieve step-by-step feature extraction of video frames, in one example of the present invention, as shown in FIG. Figure 3 As shown, the feature extraction unit 1 may further include a deformable convolution enhancement module 11 (ConvNeXt Block), a distortion adaptation module 12 (Destortion module) and an EfficientViT encoder 13 (EfficientViT Block). The deformable convolution enhancement module 11 may be used to receive input features and perform a deformable convolution enhancement operation. The specific method of the deformable convolution enhancement operation may be in various forms known to those skilled in the art. In one example of the present invention, the deformable convolution enhancement module 11 may use the following formula (1) to perform the deformable convolution enhancement operation: , (1) in, is the result of the deformable convolution enhancement operation, and any point on the feature map is recorded as , For the The dynamic weight of the sampling points, is the number of sampling points, On the feature map The eigenvalue at position, is the offset of each sampling point on the convolution kernel relative to the center point, For the network The offset learned based on .

[0023] In addition, considering that the input conditions of each feature extraction unit 1 are different, if only the same feature extraction unit 1 is used, the effect of feature extraction will be relatively poor. Therefore, in one example of the present invention, Figure 4 As shown, in Figure 3 In the case where the number of feature extraction units 1 shown is 4, the number of deformable convolution enhancement modules 11 included in the first feature extraction unit 1 can be 3, and they are connected in series to the front end of the distortion adaptive module 12. The number of deformable convolution enhancement modules 11 included in the second feature extraction unit 11 can be 3, and they are connected in series to the front end of the distortion adaptive module 12. The number of deformable convolution enhancement modules 11 included in the third feature extraction unit 11 can be 9, and they are connected in series to the front end of the distortion adaptive module 12. The number of deformable convolution enhancement modules 11 included in the fourth feature extraction unit 11 can be 3, and they are connected in series to the front end of the distortion adaptive module 12.

[0024] The specific structure of each deformable convolution enhancement module 11 can be various forms known to those skilled in the art, including but not limited to a structure obtained by connecting a single or multiple convolution or pooling layers in series. In one example of the present invention, considering the preservation of original features and the fusion of processed features, the deformable convolution enhancement module 11 can include: Figure 5 The structure shown in Figure 5In the example, the deformable convolution enhancement module 11 may include a normalization layer (LN) 111, a convolution layer 112, a GRU layer 113, a deformable convolution layer (Deformable Conv) 114, and an overlay layer 115. The normalization layer 111, convolution layer 112, GRU layer 113, and deformable convolution layer 114 are sequentially connected. The normalization layer 111 can be used to normalize input features. The convolution layer 112 is used to perform a convolution operation on the result of the normalization operation. The GRU layer 113 is used to expand the result of the convolution operation. The deformable convolution layer 114 can be used to perform a deformable convolution operation on the expanded result. The overlay layer 115 can be used to overlay the input of the normalization layer 111 with the result of the deformable convolution operation to obtain the output of the deformable convolution enhancement operation.

[0025] The distortion adaptive module 12 can be used to correct the distortion of the result of the deformable convolution enhancement operation. The specific method for performing the distortion correction can be various forms known to those skilled in the art. In one example of the present invention, the method for performing the distortion correction can be to first map the original feature map into the distortion invariant space by a polar coordinate transformation method, and then determine the attention weight of each sampling point by calculating the attention weights in the radial and angular directions, and finally perform weighted processing on the attention weight and the value of the original sampling point to complete the distortion correction operation. Specifically, in this example, the distortion adaptive module 12 can be to determine the attention weight using formula (2): , (2) in, Radial and angles The attention weight, is the radial query feature after polar coordinate transformation, is the coordinate transformation function, is the radial bond feature after polar coordinate transformation, is the angle query key feature after polar coordinate transformation, is the angle key feature after polar coordinate transformation, To correspond K The dimension of the feature.

[0026] In order to facilitate the weighting operation, the distortion adaptive module 12 may also use the following formula (3) to determine the polar coordinates: , (3) in, are coordinates in the Cartesian coordinate system, and are the coordinates corresponding to the x-axis and y-axis.

[0027] After the weighted calculation is completed, the distortion adaptive module 12 can perform a coordinate transformation operation on the result of the weighted operation (for example, by performing an inverse operation of the formula (3)) to obtain a feature in the form of a Cartesian coordinate system. The structure of the distortion adaptive module 12 can also be, for example Figure 6 shown.

[0028] The EfficientViT encoder 13 may be used to perform encoding operations on the distortion correction results.

[0029] Step S13 can be used to splice the fish body visual feature vector, voiceprint vector and environmental state vector into a fusion vector. Among them, the specific method of splicing out the fusion vector can be a variety of forms known to those skilled in the art, including but not limited to vector combination, vector addition and other methods known to those skilled in the art. In one example of the present invention, the method of splicing out the fusion vector can be to first splice the fish body visual feature vector, voiceprint vector and environmental state vector into a high-dimensional vector, then determine the attention weight of the high-dimensional vector, and then perform a weighted operation on the attention weight and the high-dimensional vector, and finally, input the result of the weighted operation into the fully connected layer to obtain the fusion vector. The corresponding logic block diagram is as follows: Figure 7 shown.

[0030] Step S14 can be used to calculate the similarity of the fused vectors. The specific calculation method of the similarity can be various forms known to those skilled in the art, including but not limited to cosine similarity, Euclidean distance, etc. known to those skilled in the art.

[0031] Step S15 can be used to filter the fusion vector based on similarity. Specifically, in one example of the present invention, in order to ensure the accuracy of the similarity, the present invention can also refer to the appearance similarity and motion similarity methods commonly used in the prior art, and combine the similarity of the fusion vector to make a combined judgment, and complete the detection frame-track matching process through a preset modal verification method and the Hungarian algorithm, and then pass the visual confidence greater than the first preset value (for example, 0.7), the acoustic matching degree less than the second preset value (for example, 0.5) and / or the parameter stability value less than the third preset value (for example, 0.3°C / s).

[0032] Step S16 can be used to perform individual fish tracking operations based on the filtered fusion vector. The individual fish tracking operation can be in various forms known to those skilled in the art. In one example of the present invention, taking into account the characteristics of the environment in which the fish are located, a virtual detection line can be set at a preset position in the fish pond. When the center point of the trajectory of the individual fish tracking operation passes through it for the first time, the count is triggered, and the sudden change in the acoustic signal intensity is linked to the stability of the environmental parameters for verification to complete the individual tracking count. If the acoustic signal remains silent for a long time and the environmental parameters remain unchanged, it is judged as an abnormal scene, the recorded data triggers manual review, and retrospective correction is performed in combination with the historical trajectory.

[0033] In addition, after the fish school individual tracking operation is implemented, in order to further intuitively display the results of the fish school individual tracking operation, a display interface system can be preset. Specifically, in one example of the present invention, the display interface system can be a cross-platform visualization interface developed based on the Qt framework to achieve real-time monitoring requirements of the aquaculture site. It is divided into four major functional modules: 1) Environmental parameter panel: Dynamically displays the real-time values ​​and historical trend curves of temperature, dissolved oxygen, and salinity (sampling interval is 1 second). Abnormal values ​​(such as dissolved oxygen <4mg / L) automatically trigger a red warning flash.

[0034] 2) Video surveillance interface: The central area embeds an OpenGL-accelerated video stream, overlaying the target detection bounding box and individual motion trajectory (the path of the last 30 seconds is drawn with gradient lines). You can click on the target to view detailed information such as ID and speed.

[0035] 3) Fish statistics dashboard: A digital counter displays the number of fish detected in the current frame and integrates a historical number change curve.

[0036] 4) Trajectory Analysis Module: A heat map embedded at the bottom displays fish density distribution, generated using a kernel density estimation algorithm. It also provides a trajectory playback function, allowing users to drag the timeline to view movement patterns at any time.

[0037] On the other hand, the present invention also provides a fish individual tracking and monitoring device, such as Figure 8 As shown in FIG. 1 , the device may include an image acquisition unit 3, a hydrophone 4, an environmental sensor 5, and a processor 6. The image acquisition unit 3 may be used to acquire video data of underwater fish schools. The hydrophone 4 may be used to acquire underwater sound data. The environmental sensor 5 may be used to acquire underwater environmental parameters. The processor 6 may be used to convert the sound data into voiceprint features and perform the following operations: Figure 1 The method shown in .

[0038] On the other hand, the present invention further provides a system for tracking and monitoring individual fish schools, the system comprising a processor, and the processor is configured to execute any of the methods described above.

[0039] Through the above technical solution, the embodiment of the present invention provides a method, device and system for tracking and monitoring individual fish schools. The method, device and system perform individual tracking operations by combining video data, voiceprint features and environmental parameter vectors. Compared with the single feature tracking operation in the prior art, the method, device and system provided by the present invention achieve a more accurate counting and tracking technical effect due to the use of multiple dimensions.

[0040] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0041] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0042] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0043] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0044] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0045] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0046] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can be implemented using any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices, or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media, such as modulated data signals and carrier waves.

[0047] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0048] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for tracking and monitoring individual fish schools, characterized in that: include: Obtain video data, sound pattern characteristics, and environmental parameters of fish; Constructing a multidimensional tensor data stream based on the video data, voiceprint features, and environmental parameters, wherein the multidimensional tensor data stream includes video frames, voiceprint vectors, and environmental parameter vectors; A preset hybrid architecture model is used to obtain a bounding box and a fish body visual feature vector according to the video frame; splicing the fish body visual feature vector, the soundprint vector and the environment state vector into a fusion vector; Calculating the similarity of the fused vectors; screening the fusion vector according to the similarity; The individual fish tracking operation is performed according to the filtered fusion vector.

2. The method according to claim 1, characterized in that Constructing a multidimensional tensor data stream based on the video data, voiceprint features, and environmental parameters, including: Using a dynamic Gaussian kernel to eliminate high-frequency bubble noise in the video data; Using a non-local mean filtering method to process the low-frequency disturbance of water ripples in the video data; The U-Net semantic segmentation model is used to remove suspended matter in water from the video data.

3. The method according to claim 1, characterized in that The hybrid architecture model includes: Multiple feature extraction units are connected in series, wherein the first feature extraction unit is used to receive the input video frame and perform feature extraction, and the remaining feature extraction units are used to receive the output of the previous feature extraction unit and perform feature extraction; The feature combination and mapping unit is connected to each of the feature extraction units and is used to combine the output of each feature extraction unit and perform a feature mapping operation to obtain the bounding box and the fish body visual feature vector.

4. The method according to claim 3, characterized in that The feature extraction unit includes: The deformable convolution enhancement module is used to receive input features and perform deformable convolution enhancement operations; Distortion adaptive module, used to correct the distortion of the results of the deformable convolution enhancement operation; The EfficientViT encoder is used to encode the distortion correction results.

5. The method according to claim 4, characterized in that The deformable convolution enhancement module is used to: The deformable convolution enhancement operation is performed according to formula (1): ，（1） in, is the result of the deformable convolution enhancement operation, and any point on the feature map is recorded as , For the The dynamic weight of the sampling points, is the number of sampling points, On the feature map The eigenvalue at position, is the offset of each sampling point on the convolution kernel relative to the center point, For the network The offset learned based on .

6. The method according to claim 4, characterized in that The distortion adaptive module is used for: Determine the attention weight according to formula (2): ，（2） in, Radial and angles The attention weight, is the radial query feature after polar coordinate transformation, is the coordinate transformation function, is the radial bond feature after polar coordinate transformation, is the angle query key feature after polar coordinate transformation, is the angle key feature after polar coordinate transformation, To correspond K Dimensionality of features; Determine the polar coordinates according to formula (3): ，（3） in, are coordinates in the Cartesian coordinate system, and are the coordinates corresponding to the x-axis and y-axis; Perform a weighted operation according to the attention weight and polar coordinates; The result of the weighted operation is subjected to a coordinate transformation operation to obtain features in the form of a Cartesian coordinate system.

7. The method according to claim 1, characterized in that The fish body visual feature vector, the soundprint vector and the environment state vector are spliced ​​into a fusion vector, including: Splicing the fish body visual feature vector, the soundprint vector, and the environment state vector into a high-dimensional vector; Determining an attention weight for the high-dimensional vector; Performing a weighted operation on the attention weight and the high-dimensional vector; The result of the weighted operation is input into the fully connected layer to obtain the fusion vector.

8. The method according to claim 1, characterized in that Filtering the fusion vector according to the similarity includes: A preset modal verification method, wherein the modal verification method includes: a visual confidence level greater than a first preset value, an acoustic matching level less than a second preset value, and / or a parameter stability value less than a third preset value; The fusion vector is determined using the Hungarian algorithm according to the modal verification method.

9. A device for tracking and monitoring individual fish schools, characterized in that: The device comprises: An image acquisition unit, used for collecting video data of underwater fish schools; hydrophones, used to collect underwater sound data; Environmental sensors, used to collect underwater environmental parameters; Processor for: Converting the sound data into voiceprint features; The method according to any one of claims 1 to 8.

10. A fish school individual tracking and monitoring system, characterized in that: The system comprises a processor, and the processor is configured to execute the method according to any one of claims 1 to 8.

Images