A method for fish target tracking and swimming speed calculation based on sonar images

Abstract

The application relates to the technical field of image processing, and discloses a fish school target tracking and swimming speed calculation method based on a sonar image, which comprises the following steps: acquiring a sonar video set and a parameter set collected by a sonar sensor array, performing isochronous interval frame processing to obtain a first video frame set, performing screening and denoising to obtain a second video frame set, determining position data and fish school type data of a target fish school based on the second video frame set, the parameter set, a pre-trained identification model, performing identification frame marking on each video frame image in the second video frame set, extracting target data of the identification frame, and calculating the swimming speed of the target fish school. Through implementation of the application, from comprehensive and accurate data collection to high-quality image processing, accurate target identification and tracking are realized, and the problems that related technologies are not perfect, target loss or misjudgment is prone to occurring when multiple targets are tracked, and the demand for accurate measurement of fish school speed cannot be met are solved.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a method for tracking fish targets and calculating their swimming speed based on sonar images. Background Technology

[0002] In the current booming development of the modern aquaculture industry, achieving refined and scientific aquaculture management has become crucial for improving aquaculture efficiency and ensuring the quality of aquatic products. Monitoring the health status and behavioral characteristics of fish schools plays a vital role in making aquaculture management decisions, and fish school speed, as a core indicator, has received much attention.

[0003] The speed of a fish school directly reflects its behavioral characteristics. Healthy and active fish typically swim at a relatively stable speed within a reasonable range. When fish encounter disease, unsuitable environments, or external disturbances, their swimming speed will change significantly. For example, fish infected with parasites may swim sluggishly; conversely, when the dissolved oxygen level in the pond is insufficient, the fish will swim faster, attempting to find areas with higher oxygen levels. Therefore, precise monitoring and analysis of fish speed can provide a deeper understanding of fish behavior patterns, thus offering a strong basis for assessing the health status of the fish school. Quantitative analysis of fish speed also has irreplaceable value in assessing the overall activity level of fish in the pond. Fish activity level is an important indicator of the quality of the aquaculture environment and the overall health of the fish school. Highly active fish exhibit more active feeding and growth, contributing to increased aquaculture yields. Conversely, low-activity fish schools may face various potential problems, such as water quality deterioration and unpalatable feed. By tracking the speed of each fish in a school and then conducting comprehensive analysis, the activity level of the fish school can be accurately quantified, providing scientific guidance for fish farmers to adjust their farming strategies in a timely manner. However, obtaining accurate images of fish schools in actual fishpond environments faces many challenges. Fishpond water quality is complex and variable, and in most cases, it is not clear. Suspended matter, algae, and microorganisms in the water can severely affect light propagation, making it difficult for traditional cameras to capture clear images of fish schools. Even in fishponds with relatively good water quality, cameras are easily limited by lighting conditions, such as in the early morning or late evening when the light is weak or when direct sunlight reflects off the water surface, resulting in blurry images of fish schools that cannot meet the requirements for accurate identification and tracking. This makes it difficult to effectively implement fish school speed analysis methods based on camera images, and thus, they cannot provide accurate and reliable data support for aquaculture management.

[0004] Currently, while some technologies exist for underwater target monitoring, methods specifically designed for fish target tracking and swimming speed calculation, and suitable for complex fishpond environments, are not yet perfect. When handling multi-target tracking, target loss or misjudgment is prone to occur. Furthermore, some technologies have low computational accuracy, failing to meet the demand for precise measurement of fish speed. These technological bottlenecks limit the aquaculture industry's progress towards intelligent and efficient development. Summary of the Invention

[0005] In view of this, the present invention provides a method for tracking fish targets and calculating swimming speed based on sonar images, in order to solve the problem that the methods in related technologies that are specifically designed for tracking fish targets and calculating swimming speed and are applicable to complex fish pond environments are not perfect. When processing multi-target tracking, they are prone to target loss or misjudgment, and cannot meet the needs for accurate measurement of fish speed.

[0006] This invention provides a method for tracking fish swarms and calculating their swimming speed based on sonar images. The method includes: acquiring a set of sonar videos and a set of parameters collected by a sonar sensor array installed within the acquisition area of ​​a target fishpond; performing isochronous frame de-framing on the sonar video set based on a preset time interval set to obtain a first video frame set; filtering and denoising the first video frame set to obtain a second video frame set; determining the location data and species data of the target fish swarm based on the second video frame set, the parameter set, a pre-trained recognition model, and the target fish swarm's location data and species data; annotating each video frame image in the second video frame set with a bounding box based on the fish swarm's location data and species data; extracting the target data from the bounding boxes in each video frame image; and calculating the swimming speed of the target fish swarm based on the target data, wherein the target data includes: center point coordinates, bounding box number, and video frame image frame number.

[0007] In one optional implementation, the above-mentioned isochronous deframe processing of the sonar video set based on a preset time interval set to obtain a first video frame set includes: obtaining a preset time interval set, wherein the preset time interval set includes at least two time intervals; and performing isochronous deframe processing on each sonar video in the sonar video set based on each time interval to obtain the first video frame set.

[0008] In one optional implementation, the above-mentioned filtering and denoising of the first video frame set to obtain the second video frame set includes: filtering the first video frame set based on a sharpness index to obtain a filtered first video frame set; sorting each video frame image in the filtered first video frame set according to time order to obtain a sorted first video frame set; and denoising each video frame image using a median filtering method to obtain the second video frame set.

[0009] In one optional implementation, the above-mentioned median filtering method is used to denoise each video frame image to obtain a second video frame set, including: using a fixed window to slide sequentially on the i-th video frame image in a row-first-column order to obtain the RGB values ​​of each pixel point within the current fixed window, obtaining a first RGB value sequence, wherein the i-th video frame image is any video frame image in the second video frame set; sorting the first RGB value sequence to obtain a second RGB value sequence; replacing the RGB value of the pixel point at the center position of the current fixed window with the median of the second RGB value sequence; marking the area smaller than a preset area in the i-th video frame image as the background color area; and traversing each video frame image in the second video frame set to obtain the second video frame set.

[0010] In one optional implementation, the above-mentioned determination of the target fish population's location data and fish species data based on the second video frame set, parameter set, pre-trained recognition model, and the second video frame set and parameter set are performed by time association processing to obtain a multimodal data set. The parameter set includes temperature and pressure parameters corresponding to different areas of the target fish pond at each acquisition time. The multimodal data set contains multiple multimodal data groups, each containing a video frame image, a corresponding temperature parameter, and a corresponding pressure parameter. The multimodal data set is input into the pre-trained recognition model, and the pre-trained recognition model is used to extract and classify targets from each video frame image to determine the preliminary location data and preliminary fish species data of the target fish population in each video frame image. The preliminary location data and preliminary fish species data of the target fish population are optimized using the temperature parameters and corresponding pressure parameters corresponding to each video frame image to obtain the target fish population's location data and fish species data.

[0011] In one optional implementation, the above-mentioned identification box annotation of each video frame image in the second video frame set based on the fish school location data and fish school species data includes: acquiring a first video frame image in the second video frame set, wherein the first video frame image is the video frame image with the earliest acquisition time in the second video frame set; performing identification box annotation on the first video frame image based on the fish school location data and fish school species data, wherein the identification box annotation includes annotating the identification box number, identification box start coordinates, identification box width, and identification box height of each identification box, and one identification box number corresponds to one target fish in the target fish school; acquiring the j-th frame in the second video frame set. The video frame image and the (j-1)th video frame image, wherein the j-th video frame image is any video frame image in the second video frame set other than the first video frame image; based on the fish school location data and fish school type data in the j-th video frame image, the starting coordinates, width, and height of each recognition box in the j-th video frame image are determined; based on the starting coordinates, width, and height of each recognition box in the j-th video frame image and the starting coordinates, width, and height of each recognition box in the (j-1)th video frame image, recognition boxes are paired, and recognition box annotations are performed on the j-th video frame image based on the pairing results.

[0012] In one optional implementation, the above-mentioned matching of recognition boxes based on the starting coordinates, width, and height of each recognition box in the j-th video frame image, and the starting coordinates, width, and height of each recognition box in the (j-1)-th video frame image, and the labeling of recognition boxes in the j-th video frame image based on the matching results, includes: determining a set of matching sequences based on a preset distance threshold, wherein the set of matching sequences includes multiple matching sequences, each matching sequence including one recognition box in the j-th video frame image and at least 0 recognition boxes in the (j-1)-th video frame images; in the k-th matching sequence, if there are at least 2 recognition boxes in the (j-1)-th video frame images, calculating the difference between the starting coordinates of the recognition box in the j-th video frame image and the starting coordinates of each recognition box in the (j-1)-th video frame image, the k-th matching sequence is any matching sequence in the set of matching sequences; if there is one The minimum difference is used as the identification box number of the recognition box in the (j-1)th frame of the video image corresponding to the minimum difference. If multiple identical minimum differences exist, the difference between the sum of the width and height of the recognition box in the j-th frame and the sum of the width and height of each recognition box in the (j-1)th frame is calculated, and the identification box number of the recognition box in the (j-1)th frame corresponding to the minimum difference is used as the identification box number of the recognition box in the j-th frame. In the k-th pairing sequence, if there is one recognition box in the (j-1)th frame, the identification box number of the recognition box in the (j-1)th frame is used as the identification box number of the recognition box in the j-th frame. In the k-th pairing sequence, if there are zero recognition boxes in the (j-1)th frame, an additional identification box number is added for the recognition box in the j-th frame.

[0013] In one optional implementation, the above-mentioned extraction of target data of recognition boxes in each video frame image and calculation of the swimming speed of the target fish group based on the target data includes: for the target fish numbered n in the target fish group, finding the recognition box numbered n in each video frame image, sorting the recognition boxes based on the video frame image frame number, and determining the time interval between adjacent recognition boxes, where n is a positive integer greater than 0 and n is less than or equal to the largest recognition box number of the target fish group; determining the swimming speed of the target fish numbered n based on the center point coordinates and time interval of each adjacent recognition box; and calculating the swimming speed of the target fish group based on the swimming speed of each target fish.

[0014] In one optional implementation, the above-mentioned calculation of the swimming speed of the target fish group based on the swimming speed of each target fish includes: if the swimming speed of the target fish numbered n in the target fish group is 0, the target fish numbered n is marked as abnormal data and is not included in the calculation of the swimming speed of the target fish group.

[0015] In one optional implementation, the above-mentioned calculation of the swimming speed of the target fish group based on the swimming speed of each target fish includes: calculating the torsion angle of the target fish numbered n in the target fish group between each adjacent identification frame; if the difference of the torsion angle in three consecutive frames is greater than or equal to 90°, the target fish numbered n is marked as abnormal data and is not included in the calculation of the swimming speed of the target fish group.

[0016] In an optional implementation, the method further includes: generating a fish health analysis report based on the swimming speed and water quality data of the target fish population, wherein the water quality data is collected by sensors installed in the target fish pond and includes: dissolved oxygen data, ammonia nitrogen content data, electrolyte data, pH data and turbidity data.

[0017] The sonar image-based fish target tracking and swimming speed calculation method provided in this embodiment firstly acquires a sonar video set and parameter set by installing a sonar sensor array within the target fishpond area. This allows for comprehensive and real-time capture of dynamic images and related environmental parameters of fish in complex aquatic environments, providing a rich and accurate data foundation for subsequent analysis and solving the problems of incomplete and inaccurate data acquisition in traditional methods. Secondly, by performing equal-time-interval frame de-framing on the sonar video set based on a preset time interval set, a first video frame set is obtained. This allows for the acquisition of a series of discrete and representative image frames from a continuous sonar video stream, facilitating detailed analysis of the fish's state. Furthermore, based on the frame de-framing effect at different time intervals, the most suitable video frames for subsequent processing can be selected, improving the targeting and effectiveness of data processing. Then, by filtering the first set of video frames based on sharpness indicators and using median filtering to denoise, a second set of video frames was obtained. This removed blurry, low-quality image frames and noise interference, significantly improving image sharpness and quality. This provided a solid image foundation for accurately identifying fish targets and determining their location and species, effectively avoiding misjudgments and missed detections due to image quality issues. Next, by associating the second set of video frames with a parameter set and inputting it into a pre-trained recognition model, the model's initial output data was optimized using the complementarity of multimodal data and parameters such as temperature and pressure. This greatly improved the accuracy of determining the location and species of target fish, enabling precise identification of different species of fish in the pond and providing a reliable basis for subsequent tracking and analysis of fish behavior. Subsequently, by first labeling the first video frame with bounding boxes and then pairing and labeling adjacent frame bounding boxes with their parameters, a correspondence between target fish groups in different video frames was established. This enabled continuous and accurate tracking of the fish group, clearly distinguishing the movement trajectory of each individual fish and solving the target loss and misjudgment problems that easily occur in multi-target tracking. Finally, by extracting target data such as the center point coordinates of the recognition box, the recognition box number, and the video frame number, the swimming speed of each fish is accurately calculated. By labeling and eliminating abnormal data, the accuracy and reliability of the calculation results are ensured. This provides precise swimming speed data support for the quantitative analysis of the health status, behavioral characteristics, and activity of fish populations, helping fish farmers to keep abreast of fish dynamics and formulate reasonable aquaculture strategies.The sonar image-based fish target tracking and swimming speed calculation method provided by this invention overcomes the limitations of traditional monitoring methods in complex fishpond environments. It achieves comprehensive and accurate data acquisition, high-quality image processing, precise target identification and tracking, and finally obtains reliable fish swimming speed data. This provides an efficient and intelligent monitoring solution for the aquaculture industry, greatly improving the ability to analyze fish health and behavior. It solves the problem that related technologies, which are specifically designed for fish target tracking and swimming speed calculation and are suitable for complex fishpond environments, are not perfect and are prone to target loss or misjudgment when handling multi-target tracking, thus failing to meet the need for accurate measurement of fish speed. Attached Figure Description

[0018] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a method for tracking fish targets and calculating swimming speed based on sonar images according to an embodiment of the present invention. Detailed Implementation

[0020] In the current booming development of the modern aquaculture industry, achieving refined and scientific aquaculture management has become crucial for improving aquaculture efficiency and ensuring the quality of aquatic products. Monitoring the health status and behavioral characteristics of fish schools plays a vital role in making aquaculture management decisions, and fish school speed, as a core indicator, has received much attention.

[0021] The speed of a fish school directly reflects its behavioral characteristics. Healthy and active fish typically swim at a relatively stable speed within a reasonable range. When fish encounter disease, unsuitable environments, or external disturbances, their swimming speed will change significantly. For example, fish infected with parasites may swim sluggishly; conversely, when the dissolved oxygen level in the pond is insufficient, the fish will swim faster, attempting to find areas with higher oxygen levels. Therefore, precise monitoring and analysis of fish speed can provide a deeper understanding of fish behavior patterns, thus offering a strong basis for assessing the health status of the fish school. Quantitative analysis of fish speed also has irreplaceable value in assessing the overall activity level of fish in the pond. Fish activity level is an important indicator of the quality of the aquaculture environment and the overall health of the fish school. Highly active fish exhibit more active feeding and growth, contributing to increased aquaculture yields. Conversely, low-activity fish schools may face various potential problems, such as water quality deterioration and unpalatable feed. By tracking the speed of each fish in a school and then conducting comprehensive analysis, the activity level of the fish school can be accurately quantified, providing scientific guidance for fish farmers to adjust their farming strategies in a timely manner. However, obtaining accurate images of fish schools in actual fishpond environments faces many challenges. Fishpond water quality is complex and variable, and in most cases, it is not clear. Suspended matter, algae, and microorganisms in the water can severely affect light propagation, making it difficult for traditional cameras to capture clear images of fish schools. Even in fishponds with relatively good water quality, cameras are easily limited by lighting conditions, such as in the early morning or late evening when the light is weak or when direct sunlight reflects off the water surface, resulting in blurry images of fish schools that cannot meet the requirements for accurate identification and tracking. This makes it difficult to effectively implement fish school speed analysis methods based on camera images, and thus, they cannot provide accurate and reliable data support for aquaculture management.

[0022] Currently, while some technologies exist for underwater target monitoring, methods specifically designed for fish target tracking and swimming speed calculation, and suitable for complex fishpond environments, are not yet perfect. When handling multi-target tracking, target loss or misjudgment is prone to occur. Furthermore, some technologies have low computational accuracy, failing to meet the demand for precise measurement of fish speed. These technological bottlenecks limit the aquaculture industry's progress towards intelligent and efficient development.

[0023] The sonar image-based fish target tracking and swimming speed calculation method provided in this embodiment firstly acquires a sonar video set and parameter set by installing a sonar sensor array within the target fishpond area. This allows for comprehensive and real-time capture of dynamic images and related environmental parameters of fish in complex aquatic environments, providing a rich and accurate data foundation for subsequent analysis and solving the problems of incomplete and inaccurate data acquisition in traditional methods. Secondly, by performing equal-time-interval frame de-framing on the sonar video set based on a preset time interval set, a first video frame set is obtained. This allows for the acquisition of a series of discrete and representative image frames from a continuous sonar video stream, facilitating detailed analysis of the fish's state. Furthermore, based on the frame de-framing effect at different time intervals, the most suitable video frames for subsequent processing can be selected, improving the targeting and effectiveness of data processing. Then, by filtering the first set of video frames based on sharpness indicators and using median filtering to denoise, a second set of video frames was obtained. This removed blurry, low-quality image frames and noise interference, significantly improving image sharpness and quality. This provided a solid image foundation for accurately identifying fish targets and determining their location and species, effectively avoiding misjudgments and missed detections due to image quality issues. Next, by associating the second set of video frames with a parameter set and inputting it into a pre-trained recognition model, the model's initial output data was optimized using the complementarity of multimodal data and parameters such as temperature and pressure. This greatly improved the accuracy of determining the location and species of target fish, enabling precise identification of different species of fish in the pond and providing a reliable basis for subsequent tracking and analysis of fish behavior. Subsequently, by first labeling the first video frame with bounding boxes and then pairing and labeling adjacent frame bounding boxes with their parameters, a correspondence between target fish groups in different video frames was established. This enabled continuous and accurate tracking of the fish group, clearly distinguishing the movement trajectory of each individual fish and solving the target loss and misjudgment problems that easily occur in multi-target tracking. Finally, by extracting target data such as the center point coordinates of the recognition box, the recognition box number, and the video frame number, the swimming speed of each fish is accurately calculated. By labeling and eliminating abnormal data, the accuracy and reliability of the calculation results are ensured. This provides precise swimming speed data support for the quantitative analysis of the health status, behavioral characteristics, and activity of fish populations, helping fish farmers to keep abreast of fish dynamics and formulate reasonable aquaculture strategies.The sonar image-based fish target tracking and swimming speed calculation method provided by this invention overcomes the limitations of traditional monitoring methods in complex fishpond environments. It achieves comprehensive and accurate data acquisition, high-quality image processing, precise target identification and tracking, and finally obtains reliable fish swimming speed data. This provides an efficient and intelligent monitoring solution for the aquaculture industry, greatly improving the ability to analyze fish health and behavior. It solves the problem that related technologies, which are specifically designed for fish target tracking and swimming speed calculation and are suitable for complex fishpond environments, are not perfect and are prone to target loss or misjudgment when handling multi-target tracking, thus failing to meet the need for accurate measurement of fish speed.

[0024] According to an embodiment of the present invention, a method for tracking fish swarms and calculating swimming speed based on sonar images is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] This embodiment provides a method for tracking fish shoals and calculating their swimming speed based on sonar images. Figure 1 This is a flowchart of a fish swarm target tracking and swimming speed calculation method based on sonar images according to an embodiment of the present invention, as shown below. Figure 1 As shown, the process includes the following steps:

[0026] Step S101: Obtain the set of sonar videos and parameter sets collected by the sonar sensor array, which is installed in the collection area of ​​the target fish pond.

[0027] Specifically, the sonar fish finder sensor is fixed to the edge of the target fishpond, placed underwater, with the beam emission direction perpendicular to the shore and the sonar lens horizontal. Based on the size of the fish and the depth of the pond, the sonar's identification mode (detection distance) and operating frequency are set, and sonar video is acquired. For example, a high-resolution, wide-beam sonar fish finder sensor array with anti-interference capabilities can be constructed. The sensors are scientifically deployed according to the shape and area of ​​the target fishpond and the expected monitoring accuracy. For rectangular fishponds, sensors can be installed at the four corners and the center to ensure coverage of the entire pond area and reduce blind spots. If the pond is irregularly shaped, sensors are deployed more densely in key areas and locations where fish may congregate, based on the actual terrain, to achieve comprehensive, blind-spot-free monitoring of the fish. The operating parameters of the sonar sensors are adjusted reasonably based on factors such as the size and density of the fish and the depth of the pond. An appropriate operating frequency is set; for example, a higher frequency can be selected to improve resolution when monitoring small fish, while a lower frequency is used to increase the detection distance when monitoring large fish or deeper ponds. At the same time, determine an appropriate detection distance. Generally, based on the maximum length and depth of the fishpond, set the detection distance to be slightly larger than the diagonal length of the fishpond to ensure that the activity range of the fish within the fishpond can be completely captured. Set the acquisition frame rate to a range that ensures the continuous dynamics of the fish are captured without generating too much redundant data, such as 25-50 frames per second, to ensure that the acquired sonar video can clearly reflect the movement state of the fish.

[0028] The parameter set encompasses temperature and pressure parameters for different areas of the target fishpond at various data collection times. High-precision temperature and pressure sensors are installed at different depths and horizontal positions within the fishpond. Waterproof, fast-response temperature sensors are selected and evenly distributed at different depths within the fishpond, such as one every 0.5 meters, to collect water temperature data in real time, as water temperature significantly affects the distribution and activity of the fish. Pressure sensors measure the water pressure at the depth where the fish are located; their installation positions correspond to those of the temperature sensors. By accurately measuring water pressure changes and combining this with sonar video data, the vertical position of the fish in the water can be determined more accurately. These sensors work synchronously with the sonar sensors, recording the collected temperature and pressure data according to timestamps and corresponding them one-to-one with the sonar video data, providing comprehensive and accurate environmental parameter information for subsequent multimodal data analysis.

[0029] Step S102: Perform equal-time frame de-framing on the above sonar video set based on the preset time interval set to obtain the first video frame set.

[0030] Specifically, step S102 includes:

[0031] Step S1021: Obtain a preset time interval set, which includes at least two time intervals.

[0032] Furthermore, the time interval range of the preset time interval set should take into account the swimming speed and behavioral characteristics of the fish school. For fish schools that swim fast and have frequent behavioral changes, the time interval should be relatively short, such as 0.1-0.3 seconds, to capture the details of their rapid movements; while for fish schools that swim slowly and have relatively stable behavior, the time interval can be appropriately increased, such as 0.5-1 second. At the same time, the frame rate of the sonar video and the computational resources and time costs of subsequent data processing should be considered. If the video frame rate is high, the time interval can be appropriately increased to reduce the amount of data; if computational resources are limited, it is also necessary to avoid the time interval being too short, which would lead to an excessive amount of data and affect processing efficiency. For example, in a fish pond for raising bass, bass swim fast, and based on testing and experience, 0.1 seconds and 0.2 seconds are more suitable as preset time intervals; while in a fish pond for raising grass carp, grass carp swim relatively slowly, and 0.3 seconds and 0.5 seconds can be selected as preset time intervals. The preset time interval set includes at least two time intervals. The number is set to find the optimal time interval for frame resolution by comparing the frame resolution effects under different time intervals. Generally, 2-4 time intervals are suitable. If the number of time intervals is too small, it may not be possible to fully compare the best time intervals; if the number of time intervals is too large, it will increase the computational cost and data processing complexity. In practice, the time intervals can be flexibly adjusted according to the complexity of the fish species and the requirements for frame resolution accuracy. For a single species of fish with relatively consistent behavior patterns, 2-3 time intervals may be sufficient; for a mixed species of fish with significant behavioral differences, the number of time intervals can be increased to 3-4.

[0033] Step S1022: Perform equal-time frame de-framing on each sonar video in the sonar video set based on each time interval to obtain the first video frame set.

[0034] Furthermore, for each sonar video in the sonar video set, frame decoding is performed according to each time interval in the preset time interval set. Taking a set containing multiple sonar videos as an example, assuming one sonar video is 60 seconds long and the preset time interval set is {0.2 seconds, 0.4 seconds}, when decoding at a time interval of 0.2 seconds, one frame is extracted every 0.2 seconds starting from the beginning of the video, for a total of 60 ÷ 0.2 = 300 frames; when decoding at a time interval of 0.4 seconds, one frame is extracted every 0.4 seconds, for a total of 60 ÷ 0.4 = 150 frames. The same method is used for the other sonar videos in the sonar video set, decoding them according to different time intervals in the preset time interval set to obtain the first video frame set.

[0035] Step S103: Filter and denoise the first video frame set to obtain the second video frame set.

[0036] Specifically, step S103 includes:

[0037] Step S1031: Filter the first video frame set based on the clarity index to obtain the filtered first video frame set.

[0038] Furthermore, the first set of video frames is processed using an edge detection tool to detect edge changes in each region of the video frame image in the horizontal and vertical directions, and these changes are summarized to calculate the overall sharpness value. Based on practical experience and extensive testing, a sharpness standard value is set. When the overall sharpness value of an image is less than this standard value, the video frame image is rejected, thus obtaining the filtered first set of video frames.

[0039] Step S1032: Sort the video frame images in the filtered first video frame set according to the time order to obtain the sorted first video frame set.

[0040] Specifically, the video frame images in the first set of filtered video frames are arranged in chronological order of their capture time.

[0041] Step S1033: Median filtering is used to denoise each video frame image to obtain the second video frame set.

[0042] Specifically, step S1033 includes:

[0043] Step a1: Using a fixed window, slide it sequentially on the i-th video frame image in a row-first, column-second order to obtain the RGB values ​​of each pixel within the current fixed window, thus obtaining the first RGB value sequence. The i-th video frame image mentioned above is any video frame image in the second video frame set.

[0044] Furthermore, a fixed-size window is determined based on the image size, and the window slides across the image sequentially in a row-to-column order.

[0045] Step a2: Sort the first RGB value sequence to obtain the second RGB value sequence.

[0046] Furthermore, after each swipe, the RGB values ​​of each point within the window are sorted. For example, for a 3×3 window, at a certain position, there are 9 pixels within the window; the RGB values ​​of these 9 pixels are extracted and sorted.

[0047] Step a3: Replace the RGB value of the pixel at the center of the current fixed window with the median of the second RGB value sequence.

[0048] Furthermore, after sorting, the median value obtained from the sorting is used to replace the original value at the center of the window. Assuming the RGB values ​​of the pixels within the window are sorted as [10, 20, 30, 40, 50, 60, 70, 80, 90], then the median is 50, and the original RGB value at the center of the window is replaced with 50. The window is continuously slid from left to right and from top to bottom, repeating the above operations of sorting the RGB values ​​of the pixels within the window and replacing the center value.

[0049] Step a4: Mark the area smaller than the preset area in the i-th video frame image as the background color area.

[0050] Furthermore, the image after initial median filtering is binarized with a threshold of 1000. The area of ​​each region in the image is compared with the preset threshold, and small areas smaller than the threshold are marked. These small areas are then set as the background color, and the smallest areas are removed. This step further optimizes the image, reducing small noise or irrelevant small areas that may interfere with subsequent fish identification.

[0051] Step a5: Traverse each video frame image in the second video frame set to obtain the second video frame set.

[0052] This process continues until all image data has been traversed. This completes the initial median filtering of the entire image, removing most of the noise.

[0053] Step S104: Based on the aforementioned second video frame set, parameter set, pre-trained recognition model, and location data and species data of the target fish group,

[0054] Specifically, step S104 includes:

[0055] Step S1041: The second video frame set and parameter set are correlated according to time to obtain a multimodal data set. The parameter set includes temperature and pressure parameters corresponding to different areas of the target fish pond at each acquisition time. The multimodal data set contains multiple multimodal data groups, and each multimodal data group contains a video frame image, the corresponding temperature parameter, and the corresponding pressure parameter.

[0056] Furthermore, each video frame image has a corresponding precise time record at the time of acquisition, and the temperature and pressure parameters in the parameter set also record the acquisition time. The time information of each video frame image in the second video frame set is extracted. Then, the temperature and pressure parameters of different areas of the target fishpond acquired at the same time point are searched in the parameter set. For example, if a video frame image was acquired at 10:15:05 on March 19, 2025, and the acquisition location is area A, the temperature and pressure data recorded at that time are searched in the parameter set. Assuming the temperature in area A of the fishpond at that time is 25℃ and the pressure is 1013 hPa, then this video frame image is combined with the temperature and pressure parameters of area A to form a multimodal data set. The image size is adjusted to the fixed size required by the YOLO model, such as 416×416 pixels, and the brightness, contrast, etc., of the image are normalized to make the image data conform to the model's input specifications. The YOLO (You Only Look Once) model is an efficient deep learning model for object detection. It treats object detection as a regression problem, and can predict the bounding boxes and class probabilities of multiple objects simultaneously with just one image scan.

[0057] Step S1042: Input the above multimodal dataset into the pre-trained recognition model, and use the above pre-trained recognition model to extract and classify targets in each video frame image, and determine the preliminary location data and preliminary fish species data of the target fish group in each video frame image.

[0058] Furthermore, the pre-trained YOLO model was trained on a dataset containing a large number of sonar images of different fish schools and related environmental parameters. The pre-constructed and pre-processed multimodal dataset was input into the model. In the feature extraction stage, the backbone network of the YOLO model extracts features from the input video frames through a series of convolutional layers. The convolutional kernels in these layers slide across the image, each responsible for extracting a specific type of local feature. For example, some kernels capture the edge contours of the fish school, while others are sensitive to the texture details. Because different regions in the sonar image have different imaging characteristics, the grayscale values ​​and texture changes in the fish school area exhibit specific patterns. After multiple convolutional operations, the model can extract deep feature maps reflecting the characteristics of the fish school. In the target detection and classification stage, the YOLO model divides the feature map into multiple grids. For each grid, the model predicts whether a fish school exists within that grid, as well as the fish school's category and bounding box location. The model judges the prediction results for each grid based on the different fish school feature patterns learned during training. For example, if the predicted features of a certain grid highly match the features of a school of carp in the training data, exhibiting characteristics such as the streamlined body contour and regular scale texture unique to carp, the model will initially determine that a school of carp exists within that grid and output the preliminary location data of the school, represented by the offset relative to the grid and the length and width of the bounding box. After transformation, this yields the approximate coordinate range in the image, such as the region from coordinates (100, 150) to (200, 250). Simultaneously, it outputs preliminary fish species data, identifying it as a school of carp. However, due to noise interference in sonar images and the complexity of actual fish movement and aggregation, these preliminary output data may contain errors and require further optimization.

[0059] Step S1043: Optimize the preliminary location data and preliminary fish species data of the target fish group using the temperature parameters and pressure parameters corresponding to each video frame image to obtain the location data and fish species data of the target fish group.

[0060] Furthermore, fish of the same species exhibit significant differences in their adaptability to temperature and pressure. This characteristic is utilized to optimize the initial fish population data. Assume the model initially identifies a school of carp in a video frame image. The temperature parameters of this image indicate a pond temperature of 28℃, and the pressure parameters suggest the fish are in shallow water. Carp are known to thrive in water temperatures between 20℃ and 30℃ and prefer shallower water; the current temperature and pressure parameters align with carp behavior. Considering the correspondence between the sonar image's imaging area and the actual pond area, in terms of location data optimization, if the initial location data shows the fish population at the image edge, and considering the suitable activity areas for carp as reflected by temperature and pressure, in the actual pond environment, temperature distribution is typically more stable in the center of the pond, and pressure variations in shallow water are relatively small. Fish tend to congregate in such areas, corresponding to these locations in the image, where the probability of fish populations is higher. Therefore, by combining the temperature and pressure parameters, the initial location data output by the YOLO model is corrected. For example, if the center coordinates of the bounding box in the initial location data are near the image edge, based on environmental parameter analysis, the center coordinates of the bounding box can be adjusted to a position in the image that better reflects the suitable activity area of ​​carp, making it more consistent with reality. Another example: if a fish group is initially identified as a tropical fish group, but the current temperature parameters show a water temperature of only 18℃, this is significantly inconsistent with the high-temperature environment suitable for tropical fish. In this case, the features and category judgment output by the YOLO model should be re-examined. By comparing the characteristics of different fish groups in the training data under low-temperature conditions, it may be discovered that the fish group is actually a local cold-water fish group with stronger temperature adaptability, such as a crucian carp group. Crucian carp can still maintain a certain level of activity in low-temperature environments, and their physical characteristics in the sonar image are more consistent with the current image features. Therefore, the initial fish group type data should be corrected, and the category should be changed from tropical fish group to crucian carp group.

[0061] By combining temperature and pressure parameters to optimize the preliminary data, more accurate and reliable data on the location and species of the target fish population can be obtained, providing a precise basis for subsequent fish tracking, behavior analysis, and aquaculture decisions.

[0062] Step S105: Based on the above fish location data and fish species data, identify bounding boxes are marked for each video frame image in the second video frame set.

[0063] Specifically, step S105 includes:

[0064] Step S1051: Obtain the first video frame image from the second video frame set, wherein the first video frame image is the video frame image with the earliest acquisition time in the second video frame set.

[0065] Furthermore, after constructing a multimodal dataset and using a pre-trained model to initially determine the location and species of the fish school, bounding box annotation began on the images in the second video frame set. First, the earliest captured video frame image was obtained from the second video frame set and defined as the first video frame image. This frame image serves as the starting reference for subsequent frame annotations and is crucial for establishing the initial state for fish school tracking. Using timestamp information, the first video frame image can be accurately located, ensuring the consistency and accuracy of the entire annotation process.

[0066] Step S1052: Based on the above fish school location data and fish school type data, the first frame video image is labeled with recognition boxes. The above recognition box labeling includes labeling the recognition box number, recognition box start coordinates, recognition box width and recognition box height of each recognition box. One recognition box number corresponds to one target fish in the target fish school.

[0067] Furthermore, based on the previously determined fish school location and species data, bounding box annotation is performed on the first video frame. For each target fish detected in this frame, a unique bounding box is generated, and comprehensive and crucial annotation information is assigned to it: to achieve accurate tracking of different target fish, each bounding box is assigned a unique number. The numbering rule can adopt a sequentially increasing method. For example, the bounding box number corresponding to the first detected target fish is set to 1, the bounding box number of the second target fish is set to 2, and so on. When processing a large number of video frames in the future, this number will serve as a key identifier for identifying and associating the position of the same target fish in different frames. The starting position coordinates of the bounding box in the image are accurately determined based on the fish school location data. Generally, the upper left corner of the image is used as the origin of the coordinate system. Assuming that the upper left corner position of a target fish in the image is determined by the data to be (x, y), then (x, y) is the starting coordinate of the bounding box. For example, if the position data of a target fish in an image shows its top-left corner coordinates as (100, 150), this means the starting coordinates of the corresponding recognition box are (100, 150). By precisely setting the starting coordinates, the location range of the target fish in the image can be accurately defined. The width and height of the recognition box are determined based on the actual size range of the fish group in the image. Specifically, this is done by calculating the difference between the boundary points in the fish group's position data. For example, if analysis of the target fish's position data shows that the difference in the horizontal boundary point coordinates is 50 pixels and the difference in the vertical boundary point coordinates is 30 pixels, then the width of the recognition box is 50 pixels and the height is 30 pixels. By reasonably setting the width and height of the recognition box, it is possible to closely match the actual size of the target fish in the image, clearly delineating the outline of the target fish, and providing an accurate initial reference for tracking and analysis of subsequent frames.

[0068] Step S1053: Obtain the j-th video frame image and the (j-1)-th video frame image in the second video frame set. The j-th video frame image is any video frame image in the second video frame set other than the first video frame image.

[0069] Furthermore, for any frame in the second video frame set other than the first video frame, it is necessary to establish a connection with the previous frame and process it.

[0070] Step S1054: Based on the fish location data and fish species data in the j-th video frame image, determine the starting coordinates, width, and height of each recognition box in the j-th video frame image.

[0071] Specifically, based on the fish location and species data in the j-th video frame, the key parameters of each bounding box in that frame are determined, namely the starting coordinates, width, and height. The method for determining these parameters is similar to the annotation method for the first frame. For example, if a target fish is detected in the j-th frame, its position data indicates that its upper left corner coordinates are (120, 160). After analyzing the horizontal and vertical boundary ranges of this target fish in the image, the boundary range differences are found to be 40 and 25, respectively. Therefore, the starting coordinates of this bounding box are (120, 160), its width is 40 pixels, and its height is 25 pixels. These precisely determined parameters provide core data support for subsequent bounding box pairing between the j-th and (j-1)-th frames, helping to accurately determine the positional changes of the same target fish in adjacent frames.

[0072] Step S1055: Based on the starting coordinates, width, and height of each recognition box in the j-th video frame image, and the starting coordinates, width, and height of each recognition box in the (j-1)-th video frame image, recognition boxes are paired, and recognition box annotations are performed on the j-th video frame image based on the pairing results.

[0073] Specifically, step S1055 includes:

[0074] Step b1: Determine a set of paired sequences based on a preset distance threshold. The set of paired sequences includes multiple paired sequences. Each paired sequence includes a bounding box in the j-th video frame and at least 0 bounding boxes in the (j-1)-th video frame.

[0075] Furthermore, to achieve accurate pairing of bounding boxes in the j-th and (j-1)-th video frames, a set of pairing sequences must first be determined based on a preset distance threshold. This preset distance threshold is not arbitrarily set but is derived by comprehensively considering factors such as the actual fish movement speed, video frame rate, and image resolution. In actual fish farming scenarios, different types of fish swim at different speeds. For example, small tropical fish swim relatively fast, and their position changes may be significant between adjacent video frames; while large cold-water fish swim slower, and their position changes are relatively smaller. Simultaneously, the video frame rate determines the time interval between adjacent frames; a higher frame rate results in a shorter time interval, and thus less positional change in the fish population between adjacent frames. Image resolution affects the accuracy of the bounding box coordinates; higher resolution results in higher coordinate accuracy, and a relatively smaller allowable distance threshold can be established. Through multiple experiments and in-depth analysis of a large amount of real-world data, a reasonable preset distance threshold is set. For each bounding box in the j-th video frame, a bounding box within the preset distance threshold is searched in the (j-1)-th video frame, thereby constructing a pairing sequence. Each pairing sequence contains a bounding box from the j-th frame of the video image, and at least 0 bounding boxes from the (j-1)-th frame of the video image. For example, if there are 3 bounding boxes in the j-th frame, in the (j-1)-th frame, bounding box 1 is within a preset distance threshold range from bounding box A in the j-th frame, bounding boxes 2 and 3 are within a preset distance threshold range from bounding box B in the j-th frame, and bounding box C in the j-th frame has no bounding box within the threshold range in the (j-1)-th frame, then 3 pairing sequences are formed, respectively containing bounding box A in the j-th frame with bounding box 1 in the (j-1)-th frame, bounding box B in the j-th frame with bounding boxes 2 and 3 in the (j-1)-th frame, and bounding box C in the j-th frame with 0 bounding boxes from the (j-1)-th frame.

[0076] Step b2: In the k-th pairing sequence, if there are at least two recognition boxes in the (j-1)-th video frame image, calculate the difference between the starting coordinates of the recognition box in the j-th video frame image and the starting coordinates of each recognition box in the (j-1)-th video frame image. The k-th pairing sequence is any pairing sequence in the pairing sequence set.

[0077] Specifically, in the k-th pairing sequence (where k represents any sequence in the pairing sequence set), if there are at least two bounding boxes in the (j-1)-th video frame, it is necessary to accurately calculate the difference between the starting coordinates of the bounding box in the j-th video frame and the starting coordinates of each bounding box in the (j-1)-th video frame. For example, in a certain pairing sequence, the starting coordinates of the bounding box in the j-th frame are (150, 180), and there are two bounding boxes in the (j-1)-th frame with starting coordinates of (140, 175) and (160, 185) respectively. Calculate the difference between the starting coordinates of the bounding box in the j-th frame and the first bounding box in the (j-1)-th frame: the horizontal difference is 150 - 140 = 10, and the vertical difference is 180 - 175 = 5; calculate the difference between the starting coordinates of the bounding box in the j-th frame and the first bounding box in the (j-1)-th frame: the horizontal difference is 150 - 160 = -10, and the vertical difference is 180 - 185 = -5. By accurately calculating these differences, we can initially determine the degree of positional correlation between the recognition box in the j-th frame and different recognition boxes in the (j-1)-th frame, providing an important basis for determining the matching relationship in the future.

[0078] Step b3: If there is a minimum difference, take the identification box number of the identification box in the (j-1)th video frame corresponding to the minimum difference as the identification box number in the jth video frame.

[0079] Specifically, after calculating the difference in starting coordinates, if a minimum difference exists, the identification box number in the (j-1)th frame of the video image corresponding to the minimum difference is used as the identification box number in the j-th frame. Continuing with the example above, calculations show that the sum of the absolute values ​​of the starting coordinate differences between the first (j-1)th frame identification box and the j-th frame identification box (|10|+|5|=15) is less than the sum of the absolute values ​​of the starting coordinate differences between the second (j-1)th frame identification box and the j-th frame identification box (|-10|+|-5|=15). Therefore, the identification box number of the first (j-1)th frame identification box is assigned to the j-th frame identification box. This method of determining the number based on the minimum difference establishes a close correspondence between the two identification boxes, highly suggesting that they represent the same target fish in different frames, providing a crucial identification number for accurately tracking the target fish's movement trajectory.

[0080] Step b4: If there are multiple identical minimum differences, calculate the difference between the sum of the width and height of the recognition box in the j-th video frame and the sum of the width and height of each recognition box in the (j-1)-th video frame. Take the recognition box number of the recognition box in the (j-1)-th video frame corresponding to the minimum difference as the recognition box number of the recognition box in the j-th video frame.

[0081] Specifically, if multiple identical minimum differences occur when calculating the starting point coordinate differences, it is necessary to further calculate the difference between the sum of the width and height of the recognition box in the j-th frame and the sum of the width and height of each recognition box in the (j-1)-th frame. For example, the width of the recognition box in the j-th frame is 40, the height is 30, and the sum is 40+30=70; in the (j-1)-th frame, the sums of the width and height of the two recognition boxes are (35+28=63) and (42+28=70), respectively. The difference between the sum of the width and height of the recognition box in the j-th frame and the first (j-1)-th frame is 70-63=7, and the difference between the sum of the width and height of the recognition box in the j-th frame and the second (j-1)-th frame is 70-70=0. The recognition box number of the recognition box in the (j-1)-th frame corresponding to the minimum difference is taken as the recognition box number of the recognition box in the j-th frame. In this example, the difference in width and height between the second (j-1)th frame image recognition box and the jth frame image recognition box is the smallest, so the number of the second (j-1)th frame image recognition box is assigned to the jth frame image recognition box. By further processing multiple cases with the same minimum difference, and taking size factors into account, the correspondence between recognition boxes can be determined more accurately in complex situations, ensuring the accuracy of target fish tracking.

[0082] Step b5: In the k-th pairing sequence, if there is a recognition box in the (j-1)-th video frame, the recognition box number of the recognition box in the (j-1)-th video frame is used as the recognition box number of the recognition box in the j-th video frame.

[0083] Specifically, in the k-th pairing sequence, if there is only one bounding box in the (j-1)-th video frame, then the bounding box number in the (j-1)-th video frame is directly used as the bounding box number in the j-th video frame. This clearly shows that in this pairing sequence, the bounding box in the j-th frame and the bounding box in the (j-1)-th frame have a very obvious correspondence, and are highly likely to represent the position of the same target fish in adjacent frames. By directly using the numbering, the continuity and accuracy of the target fish tracking number are ensured.

[0084] Step b6: In the k-th pairing sequence, if there are 0 bounding boxes in the (j-1)-th video frame, add a new bounding box number for the bounding box in the j-th video frame.

[0085] Specifically, if no bounding box exists in the (j-1)th frame of the video frame in the k-th pairing sequence, it means that the bounding box in the j-th frame could not find a matching bounding box in the (j-1)th frame. Based on the actual situation, this is likely because a new school of fish has appeared in this frame. In this case, a new bounding box number is added to the bounding box in the j-th frame, and the number can be generated sequentially according to the existing numbering. By adding a new number, the newly appearing target fish can be identified in a timely manner, so that they can be continuously tracked and labeled in subsequent video frames, ensuring the integrity of the entire fish tracking and labeling system.

[0086] Through the above series of rigorous and meticulous steps, each video frame in the second video frame set is comprehensively and accurately labeled with a bounding box, providing a solid and reliable foundation for subsequent calculations of fish swimming speed and in-depth behavioral analysis based on the bounding box data.

[0087] Step S106: Extract the target data of the recognition box in each video frame image, and calculate the swimming speed of the target fish group based on the target data. The target data includes: center point coordinates, recognition box number and video frame image frame number.

[0088] Specifically, step S106 includes:

[0089] Step S1061: For the target fish numbered n in the target fish group, find the recognition box numbered n in each video frame image, sort the recognition boxes based on the video frame image frame number, and determine the time interval between adjacent recognition boxes. n is a positive integer greater than 0, and n is less than or equal to the maximum recognition box number of the target fish group.

[0090] Furthermore, for target fish numbered n within the target fish group, a relational database was chosen to store these video frame image data for efficient and rapid retrieval. Utilizing the database's indexing function, an index was created on the recognition box number field, forming a lookup table indexed by the recognition box number. When searching for the recognition box numbered n, the database can quickly locate the corresponding record based on the index, and then rapidly retrieve the video frame image containing that recognition box. Once a recognition box is successfully found, it is sorted according to the frame number of the video frame image. The frame numbers of the video frame images increase sequentially according to the acquisition time, and their continuity and order accurately reflect the passage of time. By sorting the recognition boxes by frame number, it is ensured that these recognition boxes present a chronological order in the time dimension, thus accurately showing the position of the target fish at different times. Using the frame number as a comparison criterion, the recognition boxes are recursively divided into smaller and larger parts, and these two parts are sorted separately. Finally, the two sorted parts are merged to obtain a sequence of recognition boxes sorted from smallest to largest frame number. To determine the time interval between adjacent recognition boxes, it's necessary to review the time interval parameters set during the equal-time frame unpacking process. Assuming the time interval was set to 0.1 seconds during the equal-time frame unpacking stage, then ideally, the time interval between adjacent recognition boxes would be 0.1 seconds. However, in practical applications, due to factors such as device operating status and data acquisition environment, different time intervals may be used for frame unpacking at different stages. In this case, it's necessary to use the timestamp information of the video frame images to accurately calculate the time interval between adjacent recognition boxes.

[0091] Step S1062: Determine the swimming speed of the target fish numbered n based on the center point coordinates and time interval of each adjacent recognition box.

[0092] Furthermore, assuming the position coordinates of a fish when it appears are P1(x1, y1) and the frame number is t1, and the position coordinates of the next frame are P2(x2, y2) and the frame number is t2, and the frame interval is T, then the swimming speed of this fish can be expressed as:

[0093]

[0094] To calculate the swimming speed of the target fish between multiple adjacent recognition frames, it is necessary to calculate the swimming speed between every two adjacent recognition frames sequentially. Assuming the target fish moves between k consecutive adjacent recognition frames, the swimming speed v1 between the 1st and 2nd adjacent recognition frames, v2 between the 2nd and 3rd adjacent recognition frames, ..., v1 between the (k-1)th and kth adjacent recognition frames is calculated sequentially. k-1 Then, the average swimming speed v of the target fish numbered n during the entire tracking process is obtained by averaging. _avg =(v1+v2+…+v k-1(k-1) / (k-1). By accurately calculating the difference in coordinates between the center points of adjacent recognition boxes and combining it with accurate time intervals, the swimming speed of a single target fish can be determined without error, providing an accurate individual data basis for subsequent calculations of the swimming speed of a group of target fish.

[0095] Step S1063: Calculate the swimming speed of the target fish group based on the swimming speed of each target fish.

[0096] Furthermore, assuming that the total number of target fish involved in the speed calculation in the video image is N, then the average swimming speed of the fish school is:

[0097]

[0098] In some optional embodiments, if the swimming speed of target fish numbered n in the target fish group is 0, target fish numbered n is marked as abnormal data and is not included in the calculation of the swimming speed of the target fish group.

[0099] Specifically, in actual fish school monitoring scenarios, if the calculated swimming speed of target fish numbered n in the target fish school is 0, this is highly likely to indicate an anomaly in the data. Sonar signal interference is one of the common causes of a swimming speed of 0. When sonar equipment operates underwater, it is subject to various interferences, such as impurities in the water, abnormal reflections from other aquatic organisms, and electromagnetic interference in the surrounding environment. These interferences may cause deviations in the image data acquired by the sonar, thus affecting the accurate calculation of the target fish's position coordinates and ultimately leading to an incorrect result of a swimming speed of 0. A brief period of stillness in the target fish can also result in a calculated swimming speed of 0. In the natural behavior of fish schools, target fish may temporarily stop swimming due to feeding, resting, or avoiding predators. Although this is a normal fish school behavior, when calculating the swimming speed of the fish school to assess overall activity and health, the brief stillness of the fish can significantly interfere with the results, making the calculated swimming speed not accurately reflect the dynamic characteristics of the fish school. Furthermore, identification errors can also create the false impression of a target fish swimming speed of 0. When using pre-trained models and a series of image processing techniques to identify target fish, the complexity of sonar images can lead to the model misidentifying other objects as target fish or incorrectly locating the target fish, resulting in inaccurate or even zero calculated swimming speeds. When a target fish with the identifier n is detected swimming at a speed of 0, it needs to be marked as abnormal data to ensure the accuracy of the target fish group's swimming speed calculation. This abnormal data should not be included in subsequent calculations. In the data storage and processing system, a dedicated flag or field can be used to mark abnormal data. For example, in the database table storing target fish swimming speed data, a boolean field named "is_abnormal" can be added. When a swimming speed of 0 is detected, this field can be set to "true," indicating that the target fish's data is abnormal. When calculating the target fish group's swimming speed, the query can filter data where the "is_abnormal" field is "false" to exclude abnormal data with a swimming speed of 0, thus ensuring that the calculation results more accurately reflect the actual swimming speed of the fish group.

[0100] In some optional embodiments, the twist angle of the target fish numbered n in the target fish group between each adjacent identification frame is calculated. If the difference of the twist angle in three consecutive frames is greater than or equal to 90°, the target fish numbered n is marked as abnormal data and is not included in the calculation of the swimming speed of the target fish group.

[0101] Specifically, the formula for calculating the torsion angle is:

[0102]

[0103] If the difference in torsion angle across three consecutive frames is greater than or equal to 90°, it indicates an abnormal change in the target fish's movement. In natural environments, fish typically swim with relatively stable postures and directions, and the torsion angle changes relatively gradually under normal circumstances. However, when a torsion angle difference of 90° or more occurs, it may mean that the target fish has encountered a special situation. For example, the target fish may have collided with another object, and its direction of movement would change drastically at the moment of collision, resulting in a large change in the torsion angle. Alternatively, the target fish may have been startled by external factors, such as a sudden bright light or a loud noise. To avoid danger, the fish will quickly change its swimming direction, causing a significant increase in the difference in torsion angle across three consecutive frames. Once this abnormal torsion angle is detected in the target fish numbered n, it is also marked as abnormal data and will not be included in the calculation of the target fish's swimming speed. Judgment logic can be added to the torsion angle calculation module in the data processing flow. After calculating the torsion angle difference over three consecutive frames, if the difference is greater than or equal to 90°, the abnormal flag (such as the "is_abnormal" field mentioned above) is set to "true" in the record storing the target fish's relevant data. When subsequently calculating the target fish's swimming speed, normal data is selected based on this flag, avoiding interference from abnormally moving target fish data. This provides more reliable data support for analyzing fish behavior and health in aquaculture.

[0104] In some alternative embodiments, a fish health analysis report is generated based on the swimming speed and water quality data of the target fish population. The water quality data is collected by sensors installed in the target fish pond and includes: dissolved oxygen data, ammonia nitrogen content data, electrolyte data, pH data, and turbidity data.

[0105] Specifically, in aquaculture, accurately monitoring the health status of fish populations is crucial for improving aquaculture efficiency and ensuring the quality of aquatic products. In the target fishpond, various sensors are scientifically and rationally deployed according to the pond's size, shape, and water flow characteristics. Dissolved oxygen sensors, ammonia nitrogen sensors, electrolyte sensors, pH sensors, and turbidity sensors are installed at the inlet, outlet, central area, and areas where fish frequently congregate, ensuring comprehensive and accurate collection of water quality data from different areas. These sensors transmit the real-time data to a data processing center, integrating it with the target fish population's swimming speed data. For example, data is collected at regular intervals to ensure timeliness and continuity. Multi-dimensional analysis is performed on the target fish population's swimming speed data, calculating parameters such as average swimming speed, maximum swimming speed, minimum swimming speed, and swimming speed trends. If the average swimming speed is significantly lower than the normal range, it may indicate poor overall health and decreased vitality in the fish population; while frequent and irregular changes in swimming speed may suggest that the fish population is experiencing external disturbances or is in an unsuitable environment. By combining the normal swimming speed range of different fish species, determine whether the current swimming speed of the fish school is within a reasonable range. For example, for carp, the normal swimming speed fluctuates within a certain range. If the actual monitored swimming speed of the carp school deviates from this range, further analysis is needed to determine the cause. Dissolved oxygen is one of the key indicators for fish survival. When the dissolved oxygen content is below a certain threshold, the fish will experience difficulty breathing and may exhibit surface breathing. Long-term exposure to a low dissolved oxygen environment will lead to slow growth, decreased immunity, and even mass mortality. Analyze the trend of dissolved oxygen data. If the dissolved oxygen content continues to decline, timely oxygenation measures should be taken, such as turning on an aerator. Ammonia nitrogen is a harmful substance produced by the decomposition of fish metabolic products and uneaten feed and feces. When the ammonia nitrogen content is too high, it will damage the gills, liver, and other organs of the fish, affecting the normal physiological functions of the fish, leading to decreased appetite, stunted growth, and potentially causing disease. Monitor ammonia nitrogen content data. Once it is found to be excessive, measures such as water changes and the introduction of beneficial microorganisms should be taken to reduce the ammonia nitrogen content. Electrolytes such as calcium, magnesium, and potassium ions in fishpond water play a crucial role in maintaining the physiological balance of fish. Electrolyte imbalances can affect physiological processes such as osmotic pressure regulation, nerve conduction, and muscle contraction. Analyzing electrolyte data ensures that the concentrations of various ions are within appropriate ranges. If the concentration of one or more ions is abnormal, it may be necessary to adjust it by adding appropriate minerals or adjusting the feed formula. The pH of fishpond water should generally be maintained between 6.5 and 8.5. Excessively acidic (pH below 6.5) or excessively alkaline (pH above 8.5) water will adversely affect the growth and health of fish, such as corroding gill tissue and affecting the activity of digestive enzymes. Closely monitor pH data; when the pH deviates from the normal range, the pH can be adjusted by adding lime or acidifiers. Turbidity reflects the amount of suspended particles in the water.Excessive turbidity (e.g., exceeding 20 NTU) affects water transparency, hinders phytoplankton photosynthesis, may cause breathing difficulties in fish, and provides a breeding ground for pathogens such as bacteria and parasites, increasing the risk of fish disease. Monitoring turbidity data is crucial; if it is too high, sedimentation and filtration can be used to reduce it. A comprehensive analysis of the target fish population's swimming speed data along with various water quality data is necessary. If abnormal swimming speed is observed, and one or more water quality indicators deviate from the normal range, this indicator may be a contributing factor to the fish's health problems. For example, if the fish's swimming speed slows down and dissolved oxygen levels are low, low dissolved oxygen may be a key factor affecting fish health. Based on the comprehensive analysis results, a detailed fish health analysis report should be generated. The report should include an assessment of the current fish health status, potential health risks, an analysis of the impact of water quality indicators on fish health, and corresponding improvement suggestions. For example, it is recommended to increase the operating time of aeration equipment when the dissolved oxygen content is low, and to change the water and adjust the feed amount in time when the ammonia nitrogen content exceeds the standard. This provides fish farmers with targeted and actionable decision-making basis, helps them to adjust their farming strategies in a timely manner, and ensures the healthy growth of fish.

[0106] The sonar image-based fish target tracking and swimming speed calculation method provided in this embodiment firstly acquires a sonar video set and parameter set by installing a sonar sensor array within the target fishpond area. This allows for comprehensive and real-time capture of dynamic images and related environmental parameters of fish in complex aquatic environments, providing a rich and accurate data foundation for subsequent analysis and solving the problems of incomplete and inaccurate data acquisition in traditional methods. Secondly, by performing equal-time-interval frame de-framing on the sonar video set based on a preset time interval set, a first video frame set is obtained. This allows for the acquisition of a series of discrete and representative image frames from a continuous sonar video stream, facilitating detailed analysis of the fish's state. Furthermore, based on the frame de-framing effect at different time intervals, the most suitable video frames for subsequent processing can be selected, improving the targeting and effectiveness of data processing. Then, by filtering the first set of video frames based on sharpness indicators and using median filtering to denoise, a second set of video frames was obtained. This removed blurry, low-quality image frames and noise interference, significantly improving image sharpness and quality. This provided a solid image foundation for accurately identifying fish targets and determining their location and species, effectively avoiding misjudgments and missed detections due to image quality issues. Next, by associating the second set of video frames with a parameter set and inputting it into a pre-trained recognition model, the model's initial output data was optimized using the complementarity of multimodal data and parameters such as temperature and pressure. This greatly improved the accuracy of determining the location and species of target fish, enabling precise identification of different species of fish in the pond and providing a reliable basis for subsequent tracking and analysis of fish behavior. Subsequently, by first labeling the first video frame with bounding boxes and then pairing and labeling adjacent frame bounding boxes with their parameters, a correspondence between target fish groups in different video frames was established. This enabled continuous and accurate tracking of the fish group, clearly distinguishing the movement trajectory of each individual fish and solving the target loss and misjudgment problems that easily occur in multi-target tracking. Finally, by extracting target data such as the center point coordinates of the recognition box, the recognition box number, and the video frame number, the swimming speed of each fish is accurately calculated. By labeling and eliminating abnormal data, the accuracy and reliability of the calculation results are ensured. This provides precise swimming speed data support for the quantitative analysis of the health status, behavioral characteristics, and activity of fish populations, helping fish farmers to keep abreast of fish dynamics and formulate reasonable aquaculture strategies.The sonar image-based fish target tracking and swimming speed calculation method provided by this invention overcomes the limitations of traditional monitoring methods in complex fishpond environments. It achieves comprehensive and accurate data acquisition, high-quality image processing, precise target identification and tracking, and finally obtains reliable fish swimming speed data. This provides an efficient and intelligent monitoring solution for the aquaculture industry, greatly improving the ability to analyze fish health and behavior. It solves the problem that related technologies, which are specifically designed for fish target tracking and swimming speed calculation and are suitable for complex fishpond environments, are not perfect and are prone to target loss or misjudgment when handling multi-target tracking, thus failing to meet the need for accurate measurement of fish speed.

[0107] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for fish target tracking and swimming speed calculation based on sonar images, characterized in that, The method comprises: acquiring a sonar video set and a parameter set collected by a sonar sensor array installed in a collection area of a target fish pond; performing isochronal frame processing on the sonar video set based on a preset time interval set to obtain a first video frame set; performing screening and denoising on the first video frame set to obtain a second video frame set; based on the second video frame set, the parameter set, a pre-trained identification model, determining target fish school position data and fish school category data; based on the fish school position data and the fish school category data, performing identification frame labeling on each video frame image in the second video frame set; extracting target data of the identification frame in each video frame image, and calculating the swimming speed of the target fish school based on the target data, wherein the target data comprises: center point coordinates, identification frame number, and video frame image frame number; the identification frame labeling on each video frame image in the second video frame set based on the fish school position data and the fish school category data comprises: based on the identification frame start point coordinates, identification frame width, and identification frame height of each identification frame in the jth video frame image in the second video frame set, and the identification frame start point coordinates, identification frame width, and identification frame height of each identification frame in the (j-1)th video frame image, performing identification frame pairing, and based on the pairing result, performing identification frame labeling on the jth video frame image; based on the identification frame start point coordinates, identification frame width, and identification frame height of each identification frame in the jth video frame image in the second video frame set, and the identification frame start point coordinates, identification frame width, and identification frame height of each identification frame in the (j-1)th video frame image, performing identification frame pairing, and based on the pairing result, performing identification frame labeling on the jth video frame image, comprising: determining a preset distance threshold to determine a pairing sequence set, wherein the pairing sequence set comprises a plurality of pairing sequences, each pairing sequence comprises an identification frame in the jth video frame image, and at least 0 identification frames in the (j-1)th video frame image; in the kth pairing sequence, if there are at least two identification frames in the (j-1)th video frame image, calculating the difference between the identification frame start point coordinates of the identification frame in the jth video frame image and the identification frame start point coordinates of each identification frame in the (j-1)th video frame image, and the kth pairing sequence is any pairing sequence in the pairing sequence set; if there is a minimum difference value, the identification frame number of the identification frame in the (j-1)th video frame image corresponding to the minimum difference value is taken as the identification frame number of the identification frame in the jth video frame image; if there are a plurality of same minimum difference values, calculating the difference between the sum of the identification frame width and the identification frame height of the identification frame in the jth video frame image and the sum of the identification frame width and the identification frame height of each identification frame in the (j-1)th video frame image, and the identification frame number of the identification frame in the (j-1)th video frame image corresponding to the minimum difference value is taken as the identification frame number of the identification frame in the jth video frame image. In the kth pairing sequence, if there is 1 recognition box in the j-1th video frame image, the recognition box number of the recognition box in the j-1th video frame image is taken as the recognition box number of the recognition box in the jth video frame image; In the kth pairing sequence, if there are 0 recognition boxes in the j-1th video frame image, a recognition box number is added to the recognition box in the jth video frame image; The target data of the recognition box in each video frame image is extracted, and the swimming speed of the target fish school is calculated based on the target data, comprising: For a target fish numbered n in the target fish school, find the recognition box numbered n in each video frame image, sort the recognition boxes based on the video frame image frame number, and determine the time interval between adjacent recognition boxes, n is a positive integer greater than 0, and n is less than or equal to the maximum recognition box number of the target fish school; Based on the center point coordinates of each adjacent recognition box and the time interval, the swimming speed of the target fish numbered n is determined; Based on the swimming speed of each target fish, the swimming speed of the target fish school is calculated; The swimming speed of the target fish school is calculated based on the swimming speed of each target fish, comprising: The twist angle of the target fish numbered n between each adjacent recognition box is calculated, if the difference in twist angle in consecutive three frames is greater than or equal to 90°, the target fish numbered n is marked as abnormal data and does not participate in the calculation of the swimming speed of the target fish school, the abnormal flag bit is set to true in the record of storing the target fish related data, and the normal data is selected based on the abnormal flag bit when calculating the swimming speed of the target fish school.

2. The method of claim 1, wherein, The isochronal frame processing of the sonar video set based on the preset time interval set is performed to obtain a first video frame set, comprising: Obtain a preset time interval set, the preset time interval set includes at least two time intervals; Based on each time interval, each sonar video in the sonar video set is processed by isochronal frame processing to obtain a first video frame set.

3. The method of claim 2, wherein, The first video frame set is filtered and denoised to obtain a second video frame set, comprising: The first video frame set is filtered based on the definition index to obtain a filtered first video frame set; The video frame images in the filtered first video frame set are sorted in time sequence to obtain a sorted first video frame set; Each video frame image is denoised using a median filtering method to obtain a second video frame set.

4. The method of claim 3, wherein, The median filtering method is used to denoise each video frame image to obtain a second video frame set, comprising: A fixed window is used to slide on the ith video frame image in the order of row first and column second to obtain the RGB values of each pixel point in the current fixed window to obtain a first RGB value sequence, the ith video frame image is any video frame image in the second video frame set; The first RGB value sequence is sorted to obtain a second RGB value sequence; The median value of the second RGB value sequence is used to replace the RGB value of the pixel point at the center position of the current fixed window; Regions with an area less than a preset area in the ith video frame image are marked as background color regions; The second video frame set is obtained by traversing each video frame image in the second video frame set.

5. The method of claim 4, wherein, The position data and the fish school category data of the target fish school are determined based on the second video frame set, the parameter set, the pre-trained identification model. The second video frame set and the parameter set are associated and processed according to time to obtain a multi-modal data set, the parameter set includes temperature parameters and pressure parameters of different regions of the target fish tank at each collection time, and the multi-modal data set includes a plurality of multi-modal data groups, each multi-modal data group includes a video frame image, corresponding temperature parameters and corresponding pressure parameters. The multi-modal data set is input into the pre-trained identification model, target extraction and classification are performed on each video frame image by using the pre-trained identification model, and preliminary position data and preliminary fish school category data of the target fish school in each video frame image are determined. The preliminary position data and the preliminary fish school category data of the target fish school are optimized by using the corresponding temperature parameters and the corresponding pressure parameters of each video frame image, and the position data and the fish school category data of the target fish school are obtained.

6. The method of claim 5, wherein, The method further comprises: The first frame video frame image in the second video frame set is obtained, and the first frame video frame image is the video frame image collected earliest in the second video frame set. The first frame video frame image is annotated with the recognition frame based on the fish school position data and the fish school category data, and the recognition frame annotation includes recognition frame number, recognition frame start point coordinates, recognition frame width and recognition frame height of each recognition frame, and one recognition frame number corresponds to one target fish in the target fish school. The jth frame video frame image and the (j-1)th frame video frame image in the second video frame set are obtained, and the jth frame video frame image is any video frame image in the second video frame set except the first frame video frame image. The recognition frame start point coordinates, the recognition frame width and the recognition frame height of each recognition frame in the jth frame video frame image are determined based on the fish school position data and the fish school category data in the jth frame video frame image. The recognition frame start point coordinates, the recognition frame width and the recognition frame height of each recognition frame in the jth frame video frame image are determined based on the fish school position data and the fish school category data in the jth frame video frame image.

7. The method of claim 1, wherein, The recognition frame start point coordinates, the recognition frame width and the recognition frame height of each recognition frame in the jth frame video frame image are determined based on the fish school position data and the fish school category data in the jth frame video frame image. The swimming speed of the target fish school is calculated based on the swimming speed of each target fish.

8. The method of claim 7, wherein, If the swimming speed of the target fish numbered n in the target fish school is 0, the target fish numbered n is marked as abnormal data and does not participate in the calculation of the swimming speed of the target fish school. The method further comprises: A fish school health analysis report is generated based on the swimming speed of the target fish school and water quality data collected by sensors installed in the target fish tank, including dissolved oxygen data, ammonia nitrogen content data, electrolyte data, pH data and turbidity data.

Images