Fish school target tracking and cruising speed calculation method based on sonar image

Abstract

The invention relates to the technical field of image processing, and discloses a fish school target tracking and cruise calculation method based on a sonar image, and the method comprises the steps: obtaining a sonar video set and a parameter set collected by a sonar sensor array, and carrying out the equal-time-interval frame decoding processing, and obtaining a first video frame set; performing screening and denoising to obtain a second video frame set; based on the second video frame set, the parameter set and a pre-trained identification model, determining position data and fish school type data of a target fish school; performing identification frame labeling on each video frame image in the second video frame set; and extracting target data of the recognition frame, and calculating the swimming speed of the target fish school. Through the implementation of the method and the device, comprehensive and accurate data acquisition, high-quality image processing and accurate target identification and tracking are realized, and the problems that related technologies are not perfect, targets are easy to lose or misjudge when multi-target tracking is processed, and the requirement of accurate measurement of the fish school speed cannot be met are solved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly to a method for tracking fish school targets and calculating swimming speeds based on sonar images. Background Art

[0002] At present, with the booming development of the modern aquaculture industry, achieving refined and scientific aquaculture management has become the key to improving aquaculture efficiency and ensuring the quality of aquatic products. Monitoring the health status and behavioral characteristics of fish schools plays a crucial role in formulating aquaculture management decisions, and the fish school speed, as a core indicator, has received much attention.

[0003] The speed of a fish school directly reflects its own behavioral characteristics. For a healthy and active fish school, the swimming speed is usually relatively stable and within a certain reasonable range. When a fish school is attacked by diseases, is uncomfortable in the environment, or is disturbed by the outside world, its swimming speed will change significantly. For example, fish infected with parasites may swim sluggishly; when the dissolved oxygen content in the fish pond water is insufficient, the fish school will increase its swimming speed in an attempt to find an area with more sufficient dissolved oxygen. Therefore, through the accurate monitoring and analysis of the fish school speed, the behavioral patterns of the fish school can be deeply understood, and thus a strong basis can be provided for judging the health status of the fish school. Quantitative analysis of the fish school speed also has irreplaceable value in evaluating the activity of the fish school in the entire fish pond. The activity of the fish school is an important indicator for measuring the quality of the aquaculture environment and the overall health of the fish school. A fish school with high activity is more active in aspects such as feeding and growth, which helps to increase aquaculture production. On the contrary, a fish school with low activity may face various potential problems, such as water quality deterioration and unpalatable feed. By tracking fish school targets to obtain the speed of each fish and then conducting comprehensive analysis, the activity of the fish school can be accurately quantified, providing scientific guidance for aquaculture personnel to adjust aquaculture strategies in a timely manner. However, in the actual fish pond environment, obtaining accurate fish school images faces many challenges. The water quality conditions in fish ponds are complex and changeable, and in most cases, they are not clear. Suspended solids, algae, and microorganisms in the water will seriously affect the propagation of light, making it difficult for traditional cameras to clearly obtain fish school pictures. Even in some fish ponds with relatively good water quality, the camera is easily restricted by lighting conditions. For example, when the light is weak in the early morning and evening or when the sun shines directly on the water surface and causes reflection, the captured fish school images are blurred and cannot meet the requirements for accurately identifying and tracking fish schools. This results in the ineffectiveness of the fish school speed analysis method based on camera images and the inability to provide accurate and reliable data support for aquaculture management.

[0004] Currently, although there are some technologies for underwater target monitoring, the methods specifically for fish target tracking and swimming speed calculation and applicable to complex fish pond environments are not perfect. When dealing with multi-target tracking, it is easy to lose or misjudge targets; there are also some technologies with low calculation accuracy, which cannot meet the requirements for accurate measurement of fish speed. These technical bottlenecks limit the pace of the aquaculture industry towards intelligent and efficient development. Summary of the Invention

[0005] In view of this, the present invention provides a method for fish target tracking and swimming speed calculation based on sonar images to solve the problems in the related technologies that the methods specifically for fish target tracking and swimming speed calculation and applicable to complex fish pond environments are not perfect, it is easy to lose or misjudge targets when dealing with multi-target tracking, and the requirements for accurate measurement of fish speed cannot be met.

[0006] The present invention provides a method for fish target tracking and swimming speed calculation based on sonar images, and the method includes: obtaining a sonar video set and a parameter set collected by a sonar sensor array, where the sonar sensor array is installed in the acquisition area of the target fish pond; performing equal-time interval frame decomposition processing on the sonar video set based on a preset time interval set to obtain a first video frame set; screening and denoising the first video frame set to obtain a second video frame set; determining the position data and fish species data of the target fish group based on the second video frame set, the parameter set, and a pre-trained recognition model; performing recognition frame annotation on each video frame image in the second video frame set based on the fish group position data and fish species data; extracting the target data of the recognition frame in each video frame image, and calculating the swimming speed of the target fish group based on the target data, where the target data includes: center point coordinates, recognition frame number, and video frame image number.

[0007] In an optional implementation manner, the performing equal-time interval frame decomposition processing on the sonar video set based on a preset time interval set to obtain a first video frame set includes: obtaining a preset time interval set, where the preset time interval set includes at least two time intervals; performing equal-time interval frame decomposition processing on each sonar video in the sonar video set based on each time interval to obtain a first video frame set.

[0008] In an optional implementation manner, the screening and denoising the first video frame set to obtain a second video frame set includes: screening the first video frame set based on a clarity index to obtain a screened first video frame set; sorting each video frame image in the screened first video frame set in chronological order to obtain a sorted first video frame set; performing denoising processing on each video frame image by using a median filtering method to obtain a second video frame set.

[0009] In an alternative embodiment, the above method of using median filtering to denoise each video frame image to obtain a second video frame set includes: successively sliding a fixed window on the i-th video frame image in the order of row first and then column to obtain the RGB values of each pixel point within the current fixed window, resulting in a first RGB value sequence, where the i-th video frame image is any video frame image in the second video frame set; sorting the first RGB value sequence to obtain a second RGB value sequence; replacing the RGB value of the pixel point at the center position of the current fixed window with the median value of the second RGB value sequence; labeling the regions smaller than a preset area in the i-th video frame image as background color regions; traversing each video frame image in the second video frame set to obtain the second video frame set.

[0010] In an alternative embodiment, the above method of determining the position data and fish species data of the target fish school based on the second video frame set, the parameter set, the pre-trained recognition model includes: associating the second video frame set and the parameter set according to time to obtain a multi-modal data set, where the parameter set includes temperature parameters and pressure parameters corresponding to different regions of the target fish pond at each acquisition moment, and the multi-modal data set contains multiple multi-modal data groups, and each multi-modal data group contains a video frame image, the corresponding temperature parameter, and the corresponding pressure parameter; inputting the multi-modal data set into the pre-trained recognition model, and using the pre-trained recognition model to perform target extraction and classification on each video frame image to determine the preliminary position data and preliminary fish species data of the target fish school in each video frame image; optimizing the preliminary position data and preliminary fish species data of the target fish school by using the temperature parameters and pressure parameters corresponding to each video frame image to obtain the position data and fish species data of the target fish school.

[0011] In an alternative embodiment, the above-mentioned identification box annotation of each video frame image in the second video frame set based on the fish school position data and fish school species data includes: obtaining the first video frame image in the second video frame set, where the first video frame image is the video frame image with the earliest acquisition time in the second video frame set; performing identification box annotation on the first video frame image based on the fish school position data and fish school species data, and the identification box annotation includes annotating the identification box number, identification box starting point coordinates, identification box width, and identification box height of each identification box, and one identification box number corresponds to a target fish in a target fish school; obtaining the j-th video frame image and the (j - 1)-th video frame image in the second video frame set, where the j-th video frame image is any video frame image in the second video frame set except the first video frame image; determining the identification box starting point coordinates, identification box width, and identification box height of each identification box in the j-th video frame image based on the fish school position data and fish school species data in the j-th video frame image; performing identification box pairing based on the identification box starting point coordinates, identification box width, and identification box height of each identification box in the j-th video frame image and the identification box starting point coordinates, identification box width, and identification box height of each identification box in the (j - 1)-th video frame image, and performing identification box annotation on the j-th video frame image based on the pairing result.

[0012] In an alternative embodiment, the above-mentioned identification box pairing is performed based on the identification box starting coordinates, identification box width, and identification box height of each identification box in the j-th video frame image, and the identification box starting coordinates, identification box width, and identification box height of each identification box in the (j - 1)-th video frame image, and the j-th video frame image is labeled with identification boxes based on the pairing result, including: determining a set of pairing sequences based on a preset distance threshold, where the set of pairing sequences includes multiple pairing sequences, and each pairing sequence includes one identification box in the j-th video frame image and at least 0 identification boxes in the (j - 1)-th video frame image; in the k-th pairing sequence, if there are at least 2 identification boxes in the (j - 1)-th video frame image, calculate the difference between the identification box starting coordinates of the identification box in the j-th video frame image and the identification box starting coordinates of each identification box in the (j - 1)-th video frame image, and the k-th pairing sequence is any pairing sequence in the set of pairing sequences; if there is a minimum difference, use the identification box number of the identification box in the (j - 1)-th video frame image corresponding to the minimum difference as the identification box number of the identification box in the j-th video frame image; if there are multiple identical minimum differences, calculate the difference between the sum of the identification box width and identification box height of the identification box in the j-th video frame image and the sum of the identification box width and identification box height of each identification box in the (j - 1)-th video frame image, and use the identification box number of the identification box in the (j - 1)-th video frame image corresponding to the minimum difference as the identification box number of the identification box in the j-th video frame image; in the k-th pairing sequence, if there is 1 identification box in the (j - 1)-th video frame image, use the identification box number of the identification box in the (j - 1)-th video frame image as the identification box number of the identification box in the j-th video frame image; in the k-th pairing sequence, if there are 0 identification boxes in the (j - 1)-th video frame image, assign a new identification box number to the identification box in the j-th video frame image.

[0013] In an alternative embodiment, the above-mentioned extraction of the target data of the identification boxes in each video frame image and the calculation of the swimming speed of the target fish school based on the above-mentioned target data include: for the target fish numbered n in the target fish school, find the identification boxes with the identification box number n in each video frame image, sort the identification boxes based on the video frame image number, and determine the time interval between adjacent identification boxes, where n is a positive integer greater than 0 and n is less than or equal to the maximum identification box number of the target fish school; determine the swimming speed of the target fish numbered n based on the center point coordinates and time interval of each adjacent identification box; calculate the swimming speed of the target fish school based on the swimming speeds of each target fish.

[0014] In an alternative embodiment, the above-mentioned calculation of the swimming speed of the target fish school based on the swimming speeds of each target fish includes: if the swimming speed of the target fish numbered n in the target fish school is 0, label the target fish numbered n as abnormal data and do not participate in the calculation of the swimming speed of the target fish school.

[0015] In an alternative embodiment, calculating the swimming speed of the target fish school based on the swimming speeds of the respective target fish includes: calculating the torsion angle of the target fish numbered n in the target fish school between each adjacent recognition frame, and if the difference in the torsion angle in three consecutive frames is greater than or equal to 90°, labeling the target fish numbered n as abnormal data and not participating in the calculation of the swimming speed of the target fish school.

[0016] In an alternative embodiment, the method further includes: generating a fish school health analysis report based on the swimming speed of the target fish school and water quality data, where the water quality data is collected by sensors installed in the target fish pond and includes: dissolved oxygen data, ammonia nitrogen content data, electrolyte data, pH data, and turbidity data.

[0017] The method for fish school target tracking and swimming speed calculation based on sonar images provided in this embodiment. First, by installing a sonar sensor array in the acquisition area of the target fish pond, a sonar video set and a parameter set are obtained, which can comprehensively and real-time capture the dynamic images and relevant environmental parameters of the fish school in the complex water environment, providing a rich and accurate data basis for subsequent analysis, and solving the problems of incomplete and inaccurate data acquisition in traditional methods. Secondly, by performing equidistant frame decomposition processing on the sonar video set based on a preset time interval set to obtain a first video frame set, a series of discrete and representative image frames can be obtained from the continuous sonar video stream, facilitating subsequent detailed analysis of the fish school state. At the same time, according to the frame decomposition effect at different time intervals, the most suitable video frames for subsequent processing can be selected, improving the pertinence and effectiveness of data processing. Then, by screening the first video frame set based on the clarity index and using the median filtering method to denoise to obtain a second video frame set, the blurred and low-quality image frames and noise interference are removed, significantly improving the clarity and quality of the images, providing a good image basis for accurately identifying the fish school target and determining its position, species and other information, and effectively avoiding misjudgment and missed judgment caused by image quality problems. After that, by associating the second video frame set and the parameter set and inputting them into a pre-trained recognition model, using the complementarity of multi-modal data, and optimizing the preliminary data output by the model in combination with parameters such as temperature and pressure, the accuracy of determining the position data and species data of the target fish school is greatly improved, and the positions of different species of fish schools in the fish pond can be accurately identified, providing a reliable basis for subsequent tracking and analysis of fish school behavior. Thereafter, by first annotating the recognition frame on the first frame of the video frame image and then pairing and annotating according to the parameters of the adjacent frame recognition frames, the corresponding relationship of the target fish school between different video frames is established, realizing continuous and accurate tracking of the fish school, and being able to clearly distinguish the movement trajectories of each individual in the fish school, solving the problems of target loss and misjudgment that are prone to occur in multi-target tracking. Finally, by extracting target data such as the center point coordinates of the recognition frame, the recognition frame number, and the video frame image frame number, the swimming speed of each fish is accurately calculated, and by annotating and excluding abnormal data, the accuracy and reliability of the calculation results are ensured, providing accurate swimming speed data support for quantitatively analyzing the health status, behavior characteristics, and activity level of the fish school, helping the breeding personnel to timely master the fish school dynamics and formulate reasonable breeding strategies.The method for tracking fish school targets and calculating swimming speeds based on sonar images provided by the present invention overcomes the limitations of traditional monitoring means in complex fishpond environments, realizes comprehensive and accurate data collection, high-quality image processing, precise target recognition and tracking, and finally obtains reliable fish school swimming speed data, providing an efficient and intelligent monitoring solution for the aquaculture field, greatly enhancing the ability to analyze the health and behavior of fish schools, and solving the problem in related technologies that the method specifically for fish school target tracking and swimming speed calculation and applicable to complex fishpond environments is not perfect, and it is easy to lose or misjudge targets when dealing with multi-target tracking, unable to meet the requirement of accurately measuring the speed of fish schools. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a schematic flowchart of the method for tracking fish school targets and calculating swimming speeds based on sonar images according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0020] At present, with the booming development of the modern aquaculture industry, realizing refined and scientific aquaculture management has become the key to improving aquaculture efficiency and ensuring the quality of aquatic products. Monitoring the health status and behavioral characteristics of fish schools plays a crucial role in formulating aquaculture management decisions, and the fish school speed, as a core indicator, has attracted much attention.

[0021] The speed of a fish school directly reflects its own behavioral characteristics. For a healthy and active fish school, the swimming speed is usually relatively stable and within a certain reasonable range. When the fish school is attacked by diseases, uncomfortable with the environment or disturbed by the outside world, its swimming speed will change significantly. For example, fish infected with parasites may swim slowly; when the dissolved oxygen content in the fish pond water is insufficient, the fish school will speed up its swimming speed in an attempt to find an area with more sufficient dissolved oxygen. Therefore, through the precise monitoring and analysis of the fish school speed, the behavioral patterns of the fish school can be deeply understood, and thus a strong basis can be provided for judging the health status of the fish school. Quantitative analysis of the fish school speed also has an irreplaceable value in evaluating the activity of the fish school in the entire fish pond. The activity of the fish school is an important indicator for measuring the quality of the aquaculture environment and the overall health of the fish school. A fish school with high activity is more active in feeding, growth, etc., which helps to increase the aquaculture output. On the contrary, a fish school with low activity may face various potential problems, such as water quality deterioration and unpalatable feed. By tracking the fish school target to obtain the speed of each fish and then conducting comprehensive analysis, the activity of the fish school can be accurately quantified, providing scientific guidance for aquaculture personnel to adjust aquaculture strategies in a timely manner. However, in the actual fish pond environment, obtaining accurate fish school images faces many challenges. The water quality of the fish pond is complex and changeable, and in most cases it is not clear. Suspended solids, algae, and microorganisms in the water will seriously affect the transmission of light, making it difficult for traditional cameras to clearly obtain fish school pictures. Even in some fish ponds with relatively good water quality, the camera is easily restricted by lighting conditions. For example, when the light is weak in the early morning and evening or when the sun shines directly on the water surface and causes reflection, the captured fish school images are blurred and cannot meet the requirements for accurate identification and tracking of the fish school. This results in the difficulty of effectively implementing the fish school speed analysis method based on camera images and cannot provide accurate and reliable data support for aquaculture management.

[0022] Currently, although there are some technologies for underwater target monitoring, the methods specifically for fish school target tracking and swimming speed calculation and applicable to complex fish pond environments are not perfect. When dealing with multi-target tracking, it is easy to lose or misjudge the target; there are also some technologies with low calculation accuracy and cannot meet the requirements for accurate measurement of fish school speed. These technical bottlenecks limit the pace of the development of the aquaculture industry towards intelligence and high efficiency.

[0023] The method for tracking fish school targets and calculating swimming speeds based on sonar images provided in this embodiment first installs a sonar sensor array in the acquisition area of the target fish pond to obtain a sonar video set and a parameter set, which can comprehensively and real-time capture the dynamic images and related environmental parameters of the fish school in a complex water environment, providing a rich and accurate data basis for subsequent analysis and solving the problems of incomplete and inaccurate data acquisition in traditional methods. Secondly, by performing equal-time interval frame decomposition processing on the sonar video set based on a preset time interval set to obtain a first video frame set, a series of discrete and representative image frames can be obtained from the continuous sonar video stream, facilitating subsequent detailed analysis of the fish school state. At the same time, according to the frame decomposition effect at different time intervals, the most suitable video frames for subsequent processing can be selected, improving the pertinence and effectiveness of data processing. Then, by screening the first video frame set based on the clarity index and using the median filtering method to remove noise to obtain a second video frame set, blurred and low-quality image frames and noise interference are removed, significantly improving the clarity and quality of the images, providing a good image basis for accurately identifying fish school targets and determining their positions, species, etc., and effectively avoiding misjudgment and missed judgment caused by image quality problems. After that, by associating the second video frame set and the parameter set and inputting them into a pre-trained recognition model, using the complementarity of multi-modal data, and optimizing the preliminary data output by the model in combination with parameters such as temperature and pressure, the accuracy of determining the position data and species data of the target fish school is greatly improved, and the positions of different species of fish schools in the fish pond can be accurately identified, providing a reliable basis for subsequent tracking and analysis of fish school behavior. Thereafter, by first annotating the recognition frame on the first video frame image and then pairing and annotating according to the parameters of the adjacent frame recognition frames, the corresponding relationship of the target fish school between different video frames is established, realizing continuous and accurate tracking of the fish school, and the movement trajectories of each individual in the fish school can be clearly distinguished, solving the problems of target loss and misjudgment that are prone to occur in multi-target tracking. Finally, by extracting target data such as the center point coordinates of the recognition frame, the recognition frame number, and the video frame image number, the swimming speed of each fish is accurately calculated, and by annotating and excluding abnormal data, the accuracy and reliability of the calculation results are ensured, providing accurate swimming speed data support for quantitatively analyzing the health status, behavior characteristics, and activity of the fish school, helping the breeding personnel to timely master the fish school dynamics and formulate reasonable breeding strategies.The method for tracking fish school targets and calculating swimming speeds based on sonar images provided by the present invention overcomes the limitations of traditional monitoring means in complex fishpond environments, realizes comprehensive and accurate data collection, high-quality image processing, precise target recognition and tracking, and finally obtains reliable fish school swimming speed data, providing an efficient and intelligent monitoring solution for the aquaculture field, greatly enhancing the ability to analyze the health and behavior of fish schools, and solving the problem in related technologies that the method specifically for fish school target tracking and swimming speed calculation and applicable to complex fishpond environments is not perfect, and it is easy to lose or misjudge targets when dealing with multi-target tracking, unable to meet the requirement of accurate measurement of fish school speeds.

[0024] According to an embodiment of the present invention, an embodiment of a method for tracking fish school targets and calculating swimming speeds based on sonar images is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0025] In this embodiment, a method for tracking fish school targets and calculating swimming speeds based on sonar images is provided. Figure 1 It is a flowchart of the method for tracking fish school targets and calculating swimming speeds based on sonar images according to an embodiment of the present invention, as Figure 1 shown, and this process includes the following steps: Step S101, obtain a sonar video set and a parameter set collected by a sonar sensor array, and the above sonar sensor array is installed in the acquisition area of the target fishpond.

[0026] Specifically, fix the sonar fish finder sensor at the edge of the target fish pond, place it underwater, with the beam emission direction perpendicular to the shore and the sonar lens in a horizontal posture. Set the sonar usage recognition mode, i.e., the detection distance, according to the size of the fish school and the depth of the fish pond, set the sonar operating frequency, and collect sonar videos. For example, select sonar fish finder sensors with high resolution, wide beam and anti-interference ability to construct an array. Scientifically layout the sensors according to the shape, area of the target fish pond and the expected monitoring accuracy. For a rectangular fish pond, sensors can be installed at the four corners and the center of the fish pond to ensure that the entire fish pond area can be covered and the monitoring blind area can be reduced. If the shape of the fish pond is irregular, then according to the actual terrain, deploy sensors densely in key areas and positions where fish schools may gather to achieve all-round and dead-angle-free monitoring of the fish school. Combine factors such as the size, density of the fish school and the depth of the fish pond, and reasonably adjust the working parameters of the sonar sensor. Set an appropriate operating frequency. For example, when monitoring small fish schools, a higher frequency can be selected to improve the resolution; when monitoring large fish schools or deeper fish ponds, a lower frequency is selected to increase the detection distance. At the same time, determine an appropriate detection distance. Generally, according to the maximum length and depth of the fish pond, set the detection distance to be slightly greater than the diagonal length of the fish pond to ensure that the activity range of the fish school in the fish pond can be completely captured. Set the acquisition frame rate within the range that can ensure capturing the continuous dynamics of the fish school without generating too much redundant data, such as 25 frames per second - 50 frames per second, to ensure that the collected sonar videos can clearly reflect the movement state of the fish school.

[0027] The parameter set covers the temperature parameters and pressure parameters corresponding to different regions of the target fish pond at each acquisition moment. Install high-precision temperature sensors and pressure sensors at different depths and horizontal positions in the fish pond. Select temperature sensors with waterproof and fast response speed, and evenly distribute them in different depth layers of the fish pond, such as installing one every 0.5 meters, to collect water temperature data in real time, because water temperature has a significant impact on the distribution and activities of fish schools. The pressure sensor is used to measure the water pressure at the depth where the fish school is located. Its installation position corresponds to that of the temperature sensor. By accurately measuring the change in water pressure and combining with the sonar video data, the vertical position of the fish school in the water body can be determined more accurately. These sensors work synchronously with the sonar sensor, record the collected temperature and pressure data according to the time stamp, and correspond one by one with the sonar video data, providing comprehensive and accurate environmental parameter information for subsequent multi-modal data analysis.

[0028] Step S102, perform equal-time interval frame decomposition processing on the above sonar video set based on a preset time interval set to obtain a first video frame set.

[0029] Specifically, the above step S102 includes: Step S1021, obtain a preset time interval set, and at least two time intervals are included in the preset time interval set.

[0030] Further, the time interval range of the preset time interval set should consider the swimming speed and behavioral characteristics of the fish school. For fish schools with a relatively fast swimming speed and frequent behavioral changes, the time interval should be relatively short, such as 0.1 - 0.3 seconds, in order to capture their rapid action details; while for fish schools with a slow swimming speed and relatively stable behavior, the time interval can be appropriately increased, such as 0.5 - 1 second. At the same time, combine the frame rate of the sonar video and the computing resources and time costs of subsequent data processing. If the video frame rate is high, the time interval can be appropriately increased to reduce the data volume; if the computing resources are limited, it is also necessary to avoid too small a time interval resulting in too large a data volume and affecting the processing efficiency. For example, in a fish pond for cultivating sea bass, the sea bass has a relatively fast swimming speed. After testing and empirical judgment, it is more appropriate to select 0.1 second and 0.2 seconds as the preset time intervals; while in a fish pond for cultivating grass carp, the grass carp swims relatively slowly, and 0.3 seconds and 0.5 seconds can be selected as the preset time intervals. The preset time interval set includes at least two time intervals. The number is set to find the optimal frame extraction time interval by comparing the frame extraction effects at different time intervals. Generally, it is more appropriate to select 2 - 4 time intervals. If the number of time intervals is too small, the best time interval may not be comprehensively compared; if the number of time intervals is too large, the computing cost and data processing complexity will increase. In actual operation, it can be flexibly adjusted according to the complexity of the fish school species and the requirements for frame extraction accuracy. For a fish school of a single variety with a relatively consistent behavior pattern, 2 - 3 time intervals may be sufficient; for a fish school of multiple species with large behavioral differences, it can be appropriately increased to 3 - 4 time intervals.

[0031] Step S1022: Perform equal-time-interval frame extraction processing on each sonar video in the sonar video set based on each time interval to obtain a first video frame set.

[0032] Further, for each sonar video in the sonar video set, perform frame extraction operations according to each time interval in the preset time interval set. Taking a set containing multiple sonar videos as an example, assume that one of the sonar videos has a duration of 60 seconds and the preset time interval set is {0.2 seconds, 0.4 seconds}. When performing frame extraction according to the time interval of 0.2 seconds, starting from the beginning of the video, extract one frame image every 0.2 seconds, and a total of 60÷0.2 = 300 frame images can be extracted; when performing frame extraction according to the time interval of 0.4 seconds, extract one frame image every 0.4 seconds, and 60÷0.4 = 150 frame images can be extracted. For other sonar videos in the sonar video set, the same method is used to perform frame extraction according to different time intervals in the preset time interval set to obtain a first video frame set.

[0033] Step S103: Screen and denoise the above first video frame set to obtain a second video frame set.

[0034] Specifically, the above step S103 includes: Step S1031: Screen the above first video frame set based on the clarity index to obtain the screened first video frame set.

[0035] Furthermore, use an edge detection tool to process the first video frame set, detect the edge change conditions of each area in the video frame image in the horizontal and vertical directions, and summarize these changes to calculate the overall clarity value. Set a clarity standard value based on practical experience and a large number of tests. When the overall clarity value of the image is less than this standard value, screen out this video frame image, thereby obtaining the screened first video frame set.

[0036] Step S1032: Sort the video frame images in the screened first video frame set in chronological order to obtain the sorted first video frame set.

[0037] Specifically, arrange the video frame images in the screened first video frame set in the order of shooting time.

[0038] Step S1033: Denoise each video frame image using the median filtering method to obtain the second video frame set.

[0039] Specifically, the above Step S1033 includes: Step a1: Use a fixed window to slide successively on the i-th video frame image in the order of rows first and then columns, obtain the RGB values of each pixel point within the current fixed window to get the first RGB value sequence, where the i-th video frame image is any video frame image in the second video frame set.

[0040] Furthermore, determine a fixed-size window according to the picture size, and the window slides successively on the picture in the order of rows first and then columns.

[0041] Step a2: Sort the above first RGB value sequence to obtain the second RGB value sequence.

[0042] Furthermore, after each slide, sort the RGB values of each point within the window. For example, for a 3×3 window, at a certain position, there are 9 pixel points within the window, and extract and sort the RGB values of these 9 pixel points.

[0043] Step a3: Replace the RGB value of the pixel point at the center position of the current fixed window with the median value of the second RGB value sequence.

[0044] Further, after sorting is completed, the median value obtained from the sorting is used to replace the original value at the center position of the window. Suppose the RGB values of the pixel points within the window are sorted as [10, 20, 30, 40, 50, 60, 70, 80, 90], then the median value is 50, and the original RGB value at the center position of the window is replaced with 50. Continuously slide the window in the order from left to right and from top to bottom, and repeat the operations of sorting the RGB values of the pixel points within the window and replacing the center value of the window.

[0045] Step a4, label the regions smaller than the preset area in the i-th video frame image as background color regions.

[0046] Further, perform binarization processing on the image after preliminary median filtering, and set the threshold to 1000. Compare the area of each region in the image with the preset threshold, mark the small area regions with an area smaller than the threshold, set these small area regions as the background color, and remove the minimum area. This step further optimizes the image and reduces small noise points or irrelevant small regions that may interfere with subsequent fish school recognition.

[0047] Step a5, traverse each video frame image in the second video frame set to obtain the second video frame set.

[0048] Further, continue until all the image data has been traversed. In this way, the preliminary median filtering process for the entire image is completed, and most of the noise is removed.

[0049] Step S104, based on the above-mentioned second video frame set, parameter set, pre-trained recognition model, determine the position data and fish school type data of the target fish school Specifically, the above-mentioned step S104 includes: Step S1041, perform correlation processing on the above-mentioned second video frame set and parameter set according to time to obtain a multi-modal data set. The above-mentioned parameter set includes temperature parameters and pressure parameters corresponding to different regions of the target fish pond at each acquisition moment. The above-mentioned multi-modal data set contains multiple multi-modal data groups, and each multi-modal data group contains a video frame image, the corresponding temperature parameter, and the corresponding pressure parameter.

[0050] Furthermore, each video frame image has a corresponding precise time record during acquisition, and the temperature and pressure parameters in the parameter set also record the acquisition time. Traverse each video frame image in the second video frame set and extract its time information. Then, search in the parameter set for the temperature and pressure parameters of different areas of the target fish pond collected at the same time point. For example, there is a video frame image with an acquisition time of 10:15:05 on March 19, 2025, and the acquisition location is Area A. Search for the temperature and pressure data recorded at this moment in the parameter set. Assume that the temperature at Area A of the fish pond at this moment is 25°C and the pressure is 1013 hPa. Then, combine this video frame image with this set of temperature and pressure parameters of Area A into a multi-modal data group. Adjust the image size to the fixed size required by the YOLO model, such as 416×416 pixels, and normalize the brightness, contrast, etc. of the image to make the image data conform to the input specifications of the model. The YOLO (You Only Look Once) model is an efficient deep learning model for object detection. It regards object detection as a regression problem and can predict the bounding boxes and class probabilities of multiple objects simultaneously with only one scan of the image.

[0051] Step S1042: Input the above multi-modal data set into the pre-trained recognition model, and use the above pre-trained recognition model to perform object extraction and classification on each video frame image to determine the preliminary position data and preliminary fish group type data of the target fish group in each video frame image.

[0052] Furthermore, the pre-trained YOLO model is trained on a dataset containing a large number of sonar images of different fish schools and related environmental parameter data. The constructed and pre-processed multi-modal data set is input into this model. In the feature extraction stage, the backbone network part of the YOLO model extracts features from the input video frame images through a series of convolutional layers. The convolutional kernels in these convolutional layers slide on the image, and each convolutional kernel is responsible for extracting specific types of local features. For example, some convolutional kernels capture the edge contours of the fish school, and others are sensitive to the texture details of the fish school. Due to the different imaging characteristics of different regions in the sonar image, in the fish school aggregation area, the gray value and texture change of the image present a specific pattern. After multiple convolutional operations, the model can extract deep feature maps reflecting the characteristics of the fish school. In the object detection and classification stage, the YOLO model divides the feature map into multiple grids. For each grid, the model predicts whether there is a fish school target in the grid, as well as the category and bounding box position of the fish school. The model judges the prediction results of each grid according to the different fish school feature patterns learned during the training process. For example, if the predicted features of a certain grid highly match the features of the carp school in the training data, showing characteristics such as the unique streamlined body contour and regular scale texture of the carp, the model will initially determine that there is a carp school in this grid and output the preliminary position data of the fish school, represented by the offset relative to the grid and the length and width of the bounding box. After conversion, the approximate coordinate range in the image is obtained, such as the area from coordinates (100, 150) to (200, 250), and at the same time, the preliminary fish school type data is output, judged as a carp school. However, due to the noise interference in the sonar image and the complexity of the actual movement and aggregation of the fish school, these preliminarily output data may have errors and need to be further optimized.

[0053] Step S1043, optimize the preliminary position data and preliminary fish school type data of the target fish school by using the temperature parameter and the pressure parameter corresponding to each video frame image to obtain the position data and fish school type data of the target fish school.

[0054] Furthermore, there are significant differences in the adaptability of fish of the same species to temperature and pressure. Utilize this characteristic to optimize the preliminary fish population data. Suppose the model initially determines that the fish population in a certain video frame image is a carp population. The temperature parameter corresponding to this image shows that the temperature of the fish pond is 28°C, and the pressure parameter indicates that the fish population is in a relatively shallow water area. It is known that carp are suitable to live in a water temperature environment of 20°C - 30°C and prefer relatively shallow water areas. The current temperature and pressure parameters are consistent with the habits of carp. Considering the correspondence between the imaging area of the sonar image and the actual area of the fish pond, in terms of optimizing the position data, if the preliminary position data shows that the fish population is at the edge of the image, and according to the suitable activity area of carp reflected by the temperature and pressure, in the actual fish pond environment, the temperature distribution is usually more stable in the center of the fish pond, and the pressure change in the shallow water area is relatively small. The fish population tends to gather in such areas, and the probability of the fish population appearing at the corresponding positions in the image in these areas is higher. At this time, combine the temperature and pressure parameters to correct the preliminary position data output by the YOLO model. For example, if the center coordinates of the bounding box in the preliminary position data are near the edge of the image, according to the analysis of the environmental parameters, adjust the center coordinates of the bounding box to the corresponding position in the image that is more in line with the suitable activity area of carp to make it more in line with the actual situation. Another example is that if a certain fish population is initially judged to be a tropical fish population, but the current temperature parameter shows that the water temperature is only 18°C, which seriously does not match the high-temperature environment suitable for tropical fish. At this time, re-examine the features and category judgments output by the YOLO model. By comparing the features of different fish populations in the training data under low-temperature environments, it may be found that this fish population is actually a local cold-water fish population with stronger temperature adaptability, such as a crucian carp population. Crucian carp can still maintain a certain level of activity in a low-temperature environment, and the performance of its body characteristics in the sonar image is more consistent with the current image characteristics. Therefore, correct the preliminary fish population type data and change the category from tropical fish population to crucian carp population.

[0055] By optimizing the preliminary data by combining temperature and pressure parameters, more accurate and reliable target fish population position data and fish population type data can be obtained, providing a precise basis for subsequent fish population tracking, behavior analysis, and aquaculture decision-making.

[0056] Step S105, perform recognition box annotation on each video frame image in the second video frame set based on the above fish population position data and fish population type data.

[0057] Specifically, the above step S105 includes: Step S1051, obtain the first video frame image in the second video frame set, and the above first video frame image is the video frame image with the earliest acquisition time in the second video frame set.

[0058] Further, after constructing the multi-modal data set and preliminarily determining the fish school position and species data using the pre-trained model, start the recognition box annotation for the images in the second video frame set. First, obtain the video frame image with the earliest acquisition time from the second video frame set and define it as the first video frame image. This frame of image is crucial as the starting reference for subsequent continuous frame annotation and for establishing the initial state of fish school tracking. Through the timestamp information, the first video frame image can be accurately located to ensure the coherence and accuracy of the entire annotation process.

[0059] Step S1052, perform recognition box annotation on the first video frame image based on the above fish school position data and fish school species data. The above recognition box annotation includes annotating the recognition box number, recognition box starting coordinates, recognition box width, and recognition box height of each recognition box. One recognition box number corresponds to one target fish in a target fish school.

[0060] Further, according to the previously determined fish school position data and fish school species data, carry out the recognition box annotation operation for the first video frame image. For each target fish in each detected target fish school in this frame of image, a dedicated recognition box needs to be generated for it, and comprehensive and key annotation information needs to be given to the recognition box: To achieve accurate tracking of different target fish, each recognition box is assigned a unique number. The numbering rule can adopt the method of increasing sequentially. For example, the recognition box number corresponding to the first detected target fish is set to 1, and the recognition box number of the second target fish following it is 2, and so on. In subsequent processing of a large number of video frame images, this number will be used as the key identifier for recognizing and associating the positions of the same target fish in different frames. Accurately determine the starting position coordinates of the recognition box in the image according to the fish school position data. Generally, the upper left corner of the image is taken as the coordinate origin. Assuming that the upper left corner position of a certain target fish in the image is determined as (x, y) through data, then (x, y) is the starting coordinate of the recognition box. For example, if the position data of a certain target fish in the image shows that its upper left corner coordinate is (100, 150), it means that the starting coordinate of the corresponding recognition box is (100, 150). By setting the accurate starting coordinates, the position range of the target fish can be accurately framed in the image. Determine the width and height of the recognition box according to the actual size range of the fish school in the image. The specific method is to obtain it by calculating the difference between the boundary points in the fish school position data. For example, through the analysis of the position data of a certain target fish, the difference in the boundary point coordinates in the horizontal direction is 50 pixels, and the difference in the vertical direction is 30 pixels. Then the width of the recognition box is 50 pixels, and the height is 30 pixels. By reasonably setting the width and height of the recognition box, it can closely fit the actual size of the target fish in the image, clearly outline the contour range of the target fish, and provide an accurate initial reference for subsequent frame tracking and analysis.

[0061] Step S1053: Obtain the j-th video frame image and the (j - 1)-th video frame image in the second video frame set. The j-th video frame image is any video frame image in the second video frame set except the first video frame image.

[0062] Further, for any frame image in the second video frame set except the first video frame image, it is necessary to establish an association with the previous frame and perform processing.

[0063] Step S1054: Based on the fish school position data and fish school type data in the j-th video frame image, determine the starting coordinates, width, and height of each recognition box in the j-th video frame image.

[0064] Specifically, according to the fish school position data and fish school type data in the j-th video frame image, determine the key parameters of each recognition box in this frame image, namely the starting coordinates, width, and height. The determination method is similar to the annotation method of the first frame image. For example, if a target fish is detected in the j-th frame image, and its position data indicates that the upper left corner coordinates are (120, 160), after analyzing the horizontal and vertical boundary ranges of the target fish in the image, the boundary range differences are obtained as 40 and 25 respectively. Then the starting coordinates of this recognition box are (120, 160), the width is 40 pixels, and the height is 25 pixels. These precisely determined parameters provide core data support for subsequent pairing of recognition boxes between the j-th and (j - 1)-th frame images, and help accurately judge the position change of the same target fish in adjacent two frames.

[0065] Step S1055: Perform recognition box pairing based on the starting coordinates, width, and height of each recognition box in the j-th video frame image, and the starting coordinates, width, and height of each recognition box in the (j - 1)-th video frame image, and perform recognition box annotation on the j-th video frame image based on the pairing result.

[0066] Specifically, the above Step S1055 includes: Step b1: Determine a pairing sequence set based on a preset distance threshold. The pairing sequence set includes multiple pairing sequences, and each pairing sequence includes a recognition box in the j-th video frame image and at least 0 recognition boxes in the (j - 1)-th video frame image.

[0067] Further, to achieve the precise pairing of the recognition boxes in the j-th video frame image and the (j - 1)-th video frame image, first, a pairing sequence set needs to be determined based on a preset distance threshold. This preset distance threshold is not randomly set but is obtained by comprehensively considering various factors such as the actual swimming speed of the fish school, the video frame rate, and the image resolution. In the actual fish farming scenario, different species of fish have different swimming speeds. For example, small tropical fish swim relatively fast, and their position changes between adjacent video frames may be relatively large; while large cold-water fish swim slowly, and their position changes are relatively small. At the same time, the video frame rate determines the time interval between adjacent frames. The higher the frame rate, the shorter the time interval, and the smaller the position change of the fish school between adjacent frames. The image resolution affects the accuracy of the recognition box coordinates. The higher the resolution, the higher the coordinate accuracy, and the relatively smaller the allowable distance threshold can be. Through multiple experiments and in-depth analysis of a large amount of actual data, a reasonable preset distance threshold is set. For each recognition box in the j-th video frame image, search for the recognition boxes in the (j - 1)-th video frame image whose distances are within the preset distance threshold range, so as to construct a pairing sequence. Each pairing sequence contains one recognition box in the j-th video frame image and at least 0 recognition boxes in the (j - 1)-th video frame image. For example, if there are 3 recognition boxes in the j-th image, in the (j - 1)-th image, recognition box 1 is within the preset distance threshold range from recognition box A in the j-th image, recognition boxes 2 and 3 are within the preset distance threshold range from recognition box B in the j-th image, and there is no recognition box in the (j - 1)-th image whose distance from recognition box C in the j-th image is within the threshold range, then 3 pairing sequences are formed, which respectively contain recognition box A in the j-th image and recognition box 1 in the (j - 1)-th image, recognition box B in the j-th image and recognition boxes 2 and 3 in the (j - 1)-th image, and recognition box C in the j-th image and 0 recognition boxes in the (j - 1)-th image.

[0068] Step b2, in the k-th pairing sequence, if there are at least 2 recognition boxes in the (j - 1)-th video frame image, calculate the differences between the starting coordinates of the recognition box in the j-th video frame image and the starting coordinates of each recognition box in the (j - 1)-th video frame image, where the k-th pairing sequence is any pairing sequence in the pairing sequence set.

[0069] Specifically, in the k-th pairing sequence (where k represents any sequence in the set of pairing sequences), if there are at least two recognition frames in the video frame image of the (j - 1)-th frame, it is necessary to accurately calculate the difference between the starting coordinates of the recognition frame in the video frame image of the j-th frame and the starting coordinates of each recognition frame in the video frame image of the (j - 1)-th frame. For example, in a certain pairing sequence, the starting coordinates of the recognition frame in the j-th image are (150, 180), and there are two recognition frames in the (j - 1)-th image, with starting coordinates of (140, 175) and (160, 185) respectively. Calculate the difference between the starting coordinates of the recognition frame in the j-th image and the first recognition frame in the (j - 1)-th image. The horizontal difference is 150 - 140 = 10, and the vertical difference is 180 - 175 = 5; the difference between the starting coordinates of the recognition frame in the j-th image and the second recognition frame in the (j - 1)-th image, the horizontal difference is 150 - 160 = -10, and the vertical difference is 180 - 185 = -5. By accurately calculating these differences, it is possible to initially judge the degree of positional association between the recognition frame in the j-th image and different recognition frames in the (j - 1)-th image, providing an important basis for determining the matching relationship subsequently.

[0070] Step b3, if there is a minimum difference, use the recognition frame number of the recognition frame in the video frame image of the (j - 1)-th frame corresponding to the minimum difference as the recognition frame number of the recognition frame in the video frame image of the j-th frame.

[0071] Specifically, after calculating the difference in starting coordinates, if there is a minimum difference, then use the recognition frame number of the recognition frame in the video frame image of the (j - 1)-th frame corresponding to the minimum difference as the recognition frame number of the recognition frame in the video frame image of the j-th frame. Continuing with the above example, after calculation, it is found that the sum of the absolute values of the differences in starting coordinates between the first recognition frame in the (j - 1)-th image and the recognition frame in the j-th image (|10| + |5| = 15) is less than the sum of the absolute values of the differences in starting coordinates between the second recognition frame in the (j - 1)-th image and the recognition frame in the j-th image (|-10| + |-5| = 15). Then, assign the number of the first recognition frame in the (j - 1)-th image to the recognition frame in the j-th image. This method of determining the number based on the minimum difference establishes a close correspondence between the two recognition frames, highly likely indicating that they represent the positions of the same target fish in different frames, providing a key number identifier for accurately tracking the movement trajectory of the target fish.

[0072] Step b4, if there are multiple identical minimum differences, calculate the difference between the sum of the recognition frame width and recognition frame height of the recognition frame in the video frame image of the j-th frame and the sum of the recognition frame width and recognition frame height of each recognition frame in the video frame image of the (j - 1)-th frame, and use the recognition frame number of the recognition frame in the video frame image of the (j - 1)-th frame corresponding to the minimum difference as the recognition frame number of the recognition frame in the video frame image of the j-th frame.

[0073] Specifically, if there are multiple identical minimum differences when calculating the difference in starting coordinates, it is necessary to further calculate the difference between the sum of the width and height of the recognition box in the j-th video frame image and the sum of the width and height of each recognition box in the (j - 1)-th video frame image. For example, the width of the recognition box in the j-th image is 40, the height is 30, and the sum value is 40 + 30 = 70; in the (j - 1)-th image, the sum values of the width and height of the two recognition boxes are (35 + 28 = 63) and (42 + 28 = 70) respectively. Calculate the difference between the sum of the width and height of the recognition box in the j-th image and the first recognition box in the (j - 1)-th image as 70 - 63 = 7, and the difference between the sum of the width and height of the recognition box in the j-th image and the second recognition box in the (j - 1)-th image as 70 - 70 = 0. Take the recognition box number of the recognition box in the (j - 1)-th video frame image corresponding to the minimum difference as the recognition box number of the recognition box in the j-th video frame image. In this example, the difference between the sum of the width and height of the second recognition box in the (j - 1)-th image and the recognition box in the j-th image is the smallest, so the number of the second recognition box in the (j - 1)-th image is assigned to the recognition box in the j-th image. Through this further processing of multiple identical minimum difference situations and taking into account the size factor, the corresponding relationship between recognition boxes can be determined more accurately in complex situations, ensuring the accuracy of target fish tracking.

[0074] Step b5, in the k-th pairing sequence, if there is 1 recognition box in the (j - 1)-th video frame image, take the recognition box number of the recognition box in the (j - 1)-th video frame image as the recognition box number of the recognition box in the j-th video frame image.

[0075] Specifically, in the k-th pairing sequence, if there is only 1 recognition box in the (j - 1)-th video frame image, in this case, directly take the recognition box number of the recognition box in the (j - 1)-th video frame image as the recognition box number of the recognition box in the j-th video frame image. This clearly indicates that in this pairing sequence, there is an extremely obvious corresponding relationship between the recognition box in the j-th image and the recognition box in the (j - 1)-th image, and it is very likely to represent the positions of the same target fish in adjacent frames. By directly using the number, the continuity and accuracy of the target fish tracking number are ensured.

[0076] Step b6, in the k-th pairing sequence, if there are 0 recognition boxes in the (j - 1)-th video frame image, add a new recognition box number for the recognition box in the j-th video frame image.

[0077] Specifically, in the k-th pairing sequence, if there is no recognition box in the video frame image of the (j - 1)-th frame, it means that the recognition box in the video frame image of the j-th frame fails to find a matching recognition box in the (j - 1)-th frame. Analyzing the actual situation, this is likely because a school of fish newly appears in this frame. At this time, a new recognition box number is added to the recognition box in the video frame image of the j-th frame, and the number can be generated incrementally according to the existing number sequence. By adding the number, the newly emerged target fish can be marked in time, so that it can be continuously tracked and labeled in subsequent video frames, ensuring the integrity of the entire fish school tracking and labeling system. Through the above series of rigorous and detailed steps, comprehensive and accurate recognition box labeling is performed on each video frame image in the second video frame set, providing a solid and reliable foundation for subsequent calculation of the swimming speed of the fish school and in-depth behavior analysis based on the recognition box data.

[0078] Step S106: Extract the target data of the recognition boxes in each video frame image, and calculate the swimming speed of the target fish school based on the above target data. The above target data includes: the center point coordinates, the recognition box number, and the video frame image number.

[0079] Specifically, the above step S106 includes: Step S1061: For the target fish with the number n in the target fish school, find the recognition box with the recognition box number n in each video frame image, sort the recognition boxes based on the video frame image number, and determine the time interval between adjacent recognition boxes, where n is a positive integer greater than 0 and n is less than or equal to the maximum recognition box number of the above target fish school.

[0080] Further, for the target fish numbered n in the target fish group, in order to achieve efficient and rapid searching, a relational database is selected to store these video frame image data. Using the indexing function of the database, an index is established on the identification box number field to form a query table indexed by the identification box number. When it is necessary to search for the identification box numbered n, the database can quickly locate the corresponding record based on the index, and then quickly retrieve the video frame image containing the identification box. After successfully finding the identification box, these identification boxes are sorted according to the frame numbers of the video frame images. The frame numbers of the video frame images increase sequentially according to the order of acquisition time, and their continuity and order can accurately reflect the passage of time. By sorting the identification boxes according to the frame numbers, it can be ensured that these identification boxes show a sequential order in the time dimension, so as to accurately display the positions of the target fish at different times. Taking the frame numbers as the comparison basis for the identification boxes, the identification boxes are recursively divided into two smaller and larger parts, and these two parts are sorted respectively. Finally, the two sorted parts are merged to obtain a sequence of identification boxes sorted from smallest to largest frame number. When determining the time interval between adjacent identification boxes, it is necessary to recall the time interval parameter set during the equal-time interval frame decomposition process before. Suppose the set time interval is 0.1 second during the equal-time interval frame decomposition stage. Then, in an ideal situation, the time interval between adjacent identification boxes is 0.1 second. However, in actual applications, due to the influence of various factors such as the operating state of the device and the data acquisition environment, different time intervals may be used for frame decomposition at different stages. At this time, it is necessary to rely on the timestamp information of the video frame images to accurately calculate the time interval between adjacent identification boxes.

[0081] Step S1062: Based on the center point coordinates and time intervals of each pair of adjacent identification boxes, determine the swimming speed of the target fish numbered n.

[0082] Further, assume that the position coordinates of a fish when it appears are P1(x1, y1), the frame number is t1, the position coordinates of the next frame are P2(x2, y2), the frame number is t2, and the frame decomposition interval time is T. Then the swimming speed of this fish can be expressed as:

[0083] For the calculation of the swimming speed of the target fish between multiple adjacent identification boxes, it is necessary to calculate the swimming speed between each pair of adjacent identification boxes in turn. Suppose the target fish moves between k consecutive adjacent identification boxes. Calculate the swimming speed v1 between the 1st and 2nd adjacent identification boxes, the swimming speed v2 between the 2nd and 3rd adjacent identification boxes, ……, the swimming speed v between the (k - 1)th and kth adjacent identification boxes k-1 . Then, by taking the average value, the average swimming speed v of the target fish numbered n during the entire tracking process is obtained _avg =(v1 + v2 + … + v k-1) / (k - 1). In this way, by accurately calculating the coordinate difference of the central points of adjacent recognition frames and combining the accurate time interval, the swimming speed of a single target fish can be accurately obtained, providing an accurate individual data basis for calculating the swimming speed of the target fish school in the subsequent process.

[0084] Step S1063: Calculate the swimming speed of the target fish school based on the swimming speeds of each target fish.

[0085] Furthermore, assuming that the total number of target fish participating in the speed calculation in the video image is N, the average swimming speed of the fish school is:

[0086] In an optional manner of some embodiments, if the swimming speed of the target fish numbered n in the above target fish school is 0, the target fish numbered n is marked as abnormal data and does not participate in the calculation of the swimming speed of the target fish school.

[0087] Specifically, in an actual fish school monitoring scenario, if it is calculated that the swimming speed of the target fish numbered n in the target fish school is 0, this is very likely to imply that there is an abnormal situation in the data. Sonar signal interference is one of the common reasons for the swimming speed to be 0. When the sonar device works underwater, it will be interfered by various factors, such as impurities in the water body, abnormal reflections of other aquatic organisms, electromagnetic interference in the surrounding environment, etc. These interferences may cause deviations in the image data collected by the sonar, thereby affecting the accurate calculation of the position coordinates of the target fish and ultimately resulting in an incorrect result that the swimming speed is 0. The target fish being stationary for a short time may also cause the calculated swimming speed to be 0. In the natural behavior of the fish school, the target fish may temporarily stop swimming due to reasons such as feeding, resting, and avoiding natural enemies. Although this situation belongs to the normal behavior performance of the fish school, when calculating the swimming speed of the fish school to evaluate the overall activity level and health status, the fish that are stationary for a short time may have a greater interference on the result, making the calculated swimming speed of the fish school unable to truly reflect the dynamic characteristics of the fish school. In addition, misidentification may also create the false appearance that the swimming speed of the target fish is 0. When using a pre-trained model and a series of image processing techniques to identify the target fish, due to the complexity of the sonar image itself, the model may misidentify other objects as the target fish or mislocate the position of the target fish, resulting in a deviation or even 0 in the calculated swimming speed. When it is detected that the swimming speed of the target fish numbered n is 0, to ensure the accuracy of calculating the swimming speed of the target fish school, it needs to be marked as abnormal data and not participate in the subsequent calculation of the swimming speed of the target fish school. In the data storage and processing system, a flag bit or field can be specifically set up to mark abnormal data. For example, in the database table storing the swimming speed data of the target fish, add a Boolean field named "is_abnormal". When it is detected that the swimming speed is 0, set the value of this field to "true", indicating that the data of this target fish is abnormal data. When calculating the swimming speed of the target fish school, the query statement can exclude the abnormal data with a swimming speed of 0 by filtering the data with the "is_abnormal" field being "false", so as to ensure that the calculation result can more accurately reflect the actual swimming speed of the fish school.

[0088] In an alternative of some embodiments, calculate the torsion angle of the target fish numbered n in the above-mentioned target fish school between each adjacent recognition frame. If the difference in the torsion angle in three consecutive frames is greater than or equal to 90°, mark the target fish numbered n as abnormal data and do not participate in the calculation of the swimming speed of the target fish school.

[0089] Specifically, the torsion angle calculation formula:

[0090] If the difference in the twist angle between three consecutive frames is greater than or equal to 90°, it indicates that an abnormal change has occurred in the motion state of the target fish. In the natural environment, fish schools usually swim with relatively stable postures and directions, and the twist angle changes relatively gently under normal circumstances. When the difference in the twist angle is greater than or equal to 90°, it may mean that the target fish has encountered a special situation. For example, the target fish may have collided with other objects, and its motion direction will change sharply at the moment of collision, resulting in a large change in the twist angle. Or the target fish has been frightened by external factors such as sudden strong light or loud noise. To avoid danger, the fish will quickly change its swimming direction, resulting in a significant increase in the difference in the twist angle between three consecutive frames. Once this abnormal twist angle situation is detected for the target fish numbered n, it is also marked as abnormal data and does not participate in the calculation of the swimming speed of the target fish school. In the data processing flow, judgment logic can be added to the module for calculating the twist angle. When the difference in the twist angle between three consecutive frames is calculated, if it is found that the difference is greater than or equal to 90°, then in the record storing the relevant data of the target fish, the abnormal flag bit (such as the "is_abnormal" field mentioned above) is set to "true". When calculating the swimming speed of the target fish school subsequently, normal data is filtered based on this flag bit for calculation, avoiding interference from the data of these target fish with abnormal motion states on the calculation result of the swimming speed of the fish school, so as to provide more reliable data support for the analysis of the behavior and health status of fish schools in aquaculture.

[0091] In an alternative of some embodiments, a fish school health analysis report is generated based on the swimming speed of the target fish school and water quality data, and the water quality data is collected by sensors installed in the target fish pond, including: dissolved oxygen data, ammonia nitrogen content data, electrolyte data, pH data, and turbidity data.

[0092] Specifically, during the aquaculture process, accurately grasping the health status of fish schools is crucial for improving aquaculture efficiency and ensuring the quality of aquatic products. In the target fish pond, according to the size, shape, and water flow characteristics of the fish pond, a variety of sensors are scientifically and reasonably arranged. Dissolved oxygen sensors, ammonia nitrogen content sensors, electrolyte sensors, pH sensors, and turbidity sensors are installed at the inlet, outlet, central area, and areas where fish schools often gather in the fish pond respectively to ensure that water quality data in different areas can be comprehensively and accurately collected. These sensors transmit the real-time collected data to the data processing center and integrate it with the swimming speed data of the target fish school. For example, data is collected at regular intervals to ensure the timeliness and continuity of the data. Conduct multi-dimensional analysis on the swimming speed data of the target fish school. Calculate parameters such as the average swimming speed, maximum swimming speed, minimum swimming speed, and the change trend of the swimming speed of the fish school. If the average swimming speed of the fish school is significantly lower than the normal range, it may mean that the overall health status of the fish school is poor and the vitality has declined; while frequent and irregular changes in the swimming speed may imply that the fish school is affected by external interference or is in an unsuitable environment. Combine with the normal swimming speed range of different species of fish to judge whether the current swimming speed of the fish school is within a reasonable range. For example, for carp, the normal swimming speed fluctuates within a certain speed range. If the actual monitored swimming speed of the carp school deviates from this range, the reason needs to be further analyzed. Dissolved oxygen is one of the key indicators for the survival of fish. When the dissolved oxygen content is below a certain threshold, the fish school will feel difficulty in breathing and may show the phenomenon of floating heads. Long-term exposure to a low dissolved oxygen environment will lead to slow growth, decreased immunity, and even mass death of the fish school. Analyze the change trend of the dissolved oxygen data. If the dissolved oxygen content continues to decline, oxygen-increasing measures such as turning on the aerator need to be taken in a timely manner. Ammonia nitrogen is a harmful substance produced by the metabolism of fish, as well as the decomposition of residual bait, feces, etc. When the ammonia nitrogen content is too high, it will damage the gills, liver and other organs of the fish school, affect the normal physiological functions of the fish school, lead to reduced appetite and growth inhibition of the fish school, and may also cause diseases. Monitor the ammonia nitrogen content data. Once it is found to exceed the standard, measures such as changing water and adding beneficial microorganisms need to be taken to reduce the ammonia nitrogen content. Electrolytes in the fish pond water, such as calcium, magnesium, potassium ions, etc., play an important role in maintaining the physiological balance of the fish school. Electrolyte imbalance may affect physiological processes such as osmotic regulation, nerve conduction, and muscle contraction of the fish school. Analyze the electrolyte data to ensure that the concentrations of various ions are within the appropriate range. If the concentration of one or several ions is abnormal, it may be necessary to adjust by adding the corresponding minerals or adjusting the feed formula. The pH value of the fish pond water generally should be maintained between 6.5 - 8.5. Water quality that is too acidic (pH value below 6.5) or too alkaline (pH value above 8.5) will have an adverse impact on the growth and health of the fish school, such as corroding the gill tissues of the fish and affecting the activity of digestive enzymes of the fish. Closely monitor the pH data. When the pH value deviates from the normal range, the water quality pH can be adjusted by adding lime or acidic regulators. Turbidity reflects the amount of suspended particles in the water body.Excessive turbidity (such as exceeding 20 NTU) can affect the light transmittance of water bodies, hinder the photosynthesis of phytoplankton, and may also cause difficulty in breathing for fish populations. At the same time, it provides a breeding ground for pathogens such as bacteria and parasites, increasing the risk of fish getting sick. Monitor the turbidity data. If the turbidity is too high, methods such as sedimentation and filtration can be used to reduce the turbidity of the water body. Conduct a comprehensive analysis of the swimming speed data of the target fish population and various water quality data. If the swimming speed of the fish population shows abnormalities and one or more indicators in the water quality data also deviate from the normal range, it can be judged that this water quality indicator may be one of the reasons for the fish health problems. For example, when the swimming speed of the fish population slows down and the dissolved oxygen content is low, it can be considered that low dissolved oxygen may be a key factor affecting the health of the fish population. Generate a detailed fish health analysis report based on the comprehensive analysis results. The report content includes an assessment of the current health status of the fish population, possible health risks, an analysis of the impact of water quality indicators on fish health, and corresponding improvement suggestions, etc. For example, it is recommended to increase the operating time of oxygenation equipment when the dissolved oxygen content is low, and change the water in time and adjust the feed feeding amount when the ammonia nitrogen content exceeds the standard, etc., providing targeted and operable decision-making basis for farmers, helping farmers adjust the breeding strategy in time, and ensuring the healthy growth of the fish population.

[0093] The fish school target tracking and swimming speed calculation method based on sonar images provided by this embodiment. First, by installing a sonar sensor array in the acquisition area of the target fish pond, a sonar video set and a parameter set are obtained, which can comprehensively and real-time capture the dynamic images and relevant environmental parameters of the fish school in the complex water environment, providing a rich and accurate data basis for subsequent analysis, and solving the problems of incomplete and inaccurate data acquisition in traditional methods. Secondly, by performing equal-time interval frame decomposition processing on the sonar video set based on a preset time interval set to obtain a first video frame set, a series of discrete and representative image frames can be obtained from the continuous sonar video stream, which is convenient for subsequent detailed analysis of the fish school state. At the same time, according to the frame decomposition effect at different time intervals, the most suitable video frames for subsequent processing can be selected, improving the pertinence and effectiveness of data processing. Then, by screening the first video frame set based on the clarity index and using the median filtering method to denoise to obtain a second video frame set, the blurred and poor-quality image frames and noise interference are removed, significantly improving the clarity and quality of the images, providing a good image basis for accurately identifying the fish school target and determining its position, species and other information, and effectively avoiding misjudgment and missed judgment caused by image quality problems. After that, by associating the second video frame set with the parameter set and inputting it into a pre-trained recognition model, using the complementarity of multi-modal data, and optimizing the preliminary data output by the model in combination with parameters such as temperature and pressure, the accuracy of determining the position data and species data of the target fish school is greatly improved, and the position of different species of fish schools in the fish pond can be accurately identified, providing a reliable basis for subsequent tracking and analysis of fish school behavior. After that, by first annotating the recognition frame for the first frame of the video frame image, and then pairing and annotating according to the parameters of the adjacent frame recognition frames, the corresponding relationship of the target fish school between different video frames is established, realizing continuous and accurate tracking of the fish school, and being able to clearly distinguish the movement trajectories of each individual in the fish school, solving the problems of target loss and misjudgment that are prone to occur in multi-target tracking. Finally, by extracting target data such as the center point coordinates of the recognition frame, the recognition frame number, and the video frame image frame number, the swimming speed of each fish is accurately calculated, and by annotating and excluding abnormal data, the accuracy and reliability of the calculation results are ensured, providing accurate swimming speed data support for quantitatively analyzing the health status, behavior characteristics, and activity level of the fish school, helping the aquaculture personnel to timely master the fish school dynamics and formulate reasonable aquaculture strategies.The method for fish school target tracking and swimming speed calculation based on sonar images provided by the present invention overcomes the limitations of traditional monitoring means in complex fishpond environments, realizes comprehensive and accurate data acquisition, high-quality image processing, precise target recognition and tracking, and finally obtains reliable fish school swimming speed data, providing an efficient and intelligent monitoring solution for the aquaculture field, greatly enhancing the ability to analyze the health and behavior of fish schools, and solving the problem in related technologies that the method specifically for fish school target tracking and swimming speed calculation and applicable to complex fishpond environments is not perfect, and it is easy to lose or misjudge targets when dealing with multi-target tracking, unable to meet the requirement of accurate measurement of fish school speed.

[0094] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for fish school target tracking and swimming speed calculation based on sonar images, characterized in that, The method includes: Obtaining a sonar video set and a parameter set collected by a sonar sensor array, where the sonar sensor array is installed in the acquisition area of the target fish pond, and the parameter set includes temperature parameters and pressure parameters corresponding to different areas of the target fish pond at each acquisition moment; Performing equal-time interval frame decomposition processing on the sonar video set based on a preset time interval set to obtain a first video frame set; Screening and denoising the first video frame set to obtain a second video frame set; Based on the second video frame set, parameter set, and pre-trained recognition model, determining the position data and fish species data of the target fish school; Performing recognition frame annotation on each video frame image in the second video frame set based on the fish school position data and fish species data; Extracting the target data of the recognition frame in each video frame image, and calculating the swimming speed of the target fish school based on the target data, where the target data includes: center point coordinates, recognition frame number, and video frame image number.

2. The method according to claim 1, characterized in that, The performing equal-time interval frame decomposition processing on the sonar video set based on a preset time interval set to obtain a first video frame set includes: Obtaining a preset time interval set, where the preset time interval set includes at least two time intervals; Performing equal-time interval frame decomposition processing on each sonar video in the sonar video set based on each time interval to obtain a first video frame set.

3. The method according to claim 2, characterized in that, The screening and denoising the first video frame set to obtain a second video frame set includes: Screening the first video frame set based on a clarity index to obtain a screened first video frame set; Sorting each video frame image in the screened first video frame set in chronological order to obtain a sorted first video frame set; Performing denoising processing on each video frame image using a median filtering method to obtain a second video frame set.

4. The method according to claim 3, characterized in that, The performing denoising processing on each video frame image using a median filtering method to obtain a second video frame set includes: Using a fixed window to slide successively on the i-th video frame image in the order of rows first and then columns, obtaining the RGB values of each pixel point within the current fixed window to obtain a first RGB value sequence, where the i-th video frame image is any video frame image in the second video frame set; Performing sorting processing on the first RGB value sequence to obtain a second RGB value sequence; Replacing the RGB value of the pixel point at the center position of the current fixed window with the median value of the second RGB value sequence; Labeling the area smaller than a preset area in the i-th video frame image as a background color area; Traversing each video frame image in the second video frame set to obtain a second video frame set.

5. The method according to claim 4, characterized in that, The based on the second video frame set, parameter set, and pre-trained recognition model, determining the position data and fish species data of the target fish school includes: Associating the second video frame set and the parameter set according to time to obtain a multi-modal data set, where the parameter set includes temperature parameters and pressure parameters corresponding to different areas of the target fish pond at each acquisition moment, and the multi-modal data set contains multiple multi-modal data groups, and each multi-modal data group contains a video frame image, corresponding temperature parameters, and corresponding pressure parameters; Input the multi-modal data set into a pre-trained recognition model, and use the pre-trained recognition model to perform target extraction and classification on each video frame image to determine the preliminary position data and preliminary fish population type data of the target fish population in each video frame image; Optimize the preliminary position data and preliminary fish population type data of the target fish population by using the temperature parameter and corresponding pressure parameter corresponding to each video frame image to obtain the position data and fish population type data of the target fish population.

6. The method according to claim 5, characterized in that, The recognition box annotation for each video frame image in the second video frame set based on the fish population position data and fish population type data includes: Obtain the first video frame image in the second video frame set, where the first video frame image is the video frame image with the earliest acquisition time in the second video frame set; Perform recognition box annotation on the first video frame image based on the fish population position data and fish population type data. The recognition box annotation includes annotating the recognition box number, recognition box starting coordinates, recognition box width, and recognition box height of each recognition box. One recognition box number corresponds to one target fish in a target fish population; Obtain the j-th video frame image and the (j - 1)-th video frame image in the second video frame set, where the j-th video frame image is any video frame image in the second video frame set except the first video frame image; Determine the recognition box starting coordinates, recognition box width, and recognition box height of each recognition box in the j-th video frame image based on the fish population position data and fish population type data in the j-th video frame image; Perform recognition box pairing based on the recognition box starting coordinates, recognition box width, and recognition box height of each recognition box in the j-th video frame image and the recognition box starting coordinates, recognition box width, and recognition box height of each recognition box in the (j - 1)-th video frame image, and perform recognition box annotation on the j-th video frame image based on the pairing result.

7. The method according to claim 6, characterized in that, The recognition box pairing based on the recognition box starting coordinates, recognition box width, and recognition box height of each recognition box in the j-th video frame image and the recognition box starting coordinates, recognition box width, and recognition box height of each recognition box in the (j - 1)-th video frame image, and performing recognition box annotation on the j-th video frame image based on the pairing result includes: Determine a pairing sequence set based on a preset distance threshold. The pairing sequence set includes multiple pairing sequences, and each pairing sequence includes one recognition box in the j-th video frame image and at least 0 recognition boxes in the (j - 1)-th video frame image; In the k-th pairing sequence, if there are at least 2 recognition boxes in the (j - 1)-th video frame image, calculate the difference between the recognition box starting coordinates of the recognition box in the j-th video frame image and the recognition box starting coordinates of each recognition box in the (j - 1)-th video frame image. The k-th pairing sequence is any pairing sequence in the pairing sequence set; If there is a minimum difference, use the recognition box number of the recognition box in the (j - 1)-th video frame image corresponding to the minimum difference as the recognition box number of the recognition box in the j-th video frame image; If there are multiple identical minimum differences, calculate the difference between the sum of the recognition box width and the recognition box height of the recognition box in the j-th video frame image and the sum of the recognition box width and the recognition box height of each recognition box in the (j - 1)-th video frame image, and use the recognition box number of the recognition box in the (j - 1)-th video frame image corresponding to the minimum difference as the recognition box number of the recognition box in the j-th video frame image; In the k-th pairing sequence, if there is one recognition box in the (j - 1)-th video frame image, use the recognition box number of the recognition box in the (j - 1)-th video frame image as the recognition box number of the recognition box in the j-th video frame image; In the k-th pairing sequence, if there are zero recognition boxes in the (j - 1)-th video frame image, add a new recognition box number for the recognition box in the j-th video frame image.

8. The method according to claim 7, wherein, The extracting the target data of the recognition boxes in each video frame image and calculating the swimming speed of the target fish school based on the target data includes: For the target fish numbered n in the target fish school, find the recognition boxes with the recognition box number n in each video frame image, sort the recognition boxes based on the video frame image number, and determine the time interval between adjacent recognition boxes, where n is a positive integer greater than 0 and n is less than or equal to the maximum recognition box number of the target fish school; Based on the center point coordinates and time interval of each adjacent recognition box, determine the swimming speed of the target fish numbered n; Based on the swimming speeds of each target fish, calculate the swimming speed of the target fish school.

9. The method according to claim 8, wherein, The calculating the swimming speed of the target fish school based on the swimming speeds of each target fish includes: If the swimming speed of the target fish numbered n in the target fish school is 0, label the target fish numbered n as abnormal data and do not participate in calculating the swimming speed of the target fish school.

10. The method according to claim 9, wherein, The calculating the swimming speed of the target fish school based on the swimming speeds of each target fish includes: Calculate the torsion angle of the target fish numbered n in the target fish school between each adjacent recognition box. If the difference in the torsion angle in three consecutive frames is greater than or equal to 90°, label the target fish numbered n as abnormal data and do not participate in calculating the swimming speed of the target fish school.

11. The method according to claim 10, wherein, The method further includes: Generating a fish school health analysis report based on the swimming speed of the target fish school and water quality data, where the water quality data is collected by sensors installed in the target fish pond and includes: dissolved oxygen data, ammonia nitrogen content data, electrolyte data, pH data, and turbidity data.

Images