A sink environment fish individual tracking and tail swing motion parameter real-time calculation method

Abstract

The application discloses a sink environment fish individual tracking and tail swing motion parameter real-time calculation method, and belongs to the fish individual tracking field. The method comprises the following steps: acquiring fish monitoring video frames in a sink environment in real time and determining a tracking target; performing constraint on each pixel in a current preprocessed frame through parallel difference calculation channels to obtain a joint response image; updating a trajectory of the tracking target based on the joint response image; taking a position of a latest trajectory node as an observation value, and predicting a position and a speed of the tracking target based on a Kalman tracking algorithm; performing skeleton extraction and tail end determination based on the position and the speed of the tracking target; and obtaining tail swing motion parameters based on normal projection tail swing data of a current frame. The application solves the problems that in the prior art, when multiple targets move at a close distance, local observation is easily disturbed by adjacent targets, leading to trajectory drift, and tail end feature extraction is unstable, further leading to unreliable tail swing frequency and amplitude calculation results.

Description

Technical Field

[0001] This invention belongs to the field of individual fish tracking, and particularly relates to a method for real-time calculation of tail sway motion parameters of individual fish in a tank environment. Background Technology

[0002] With the development of fish behavior science, digital management of aquaculture, and refined phenotypic analysis of laboratory animals, fish motion tracking and behavioral parameter extraction based on video images have become important research methods. Researchers typically hope to obtain information such as the position, speed, trajectory, tail wagging frequency, and tail wagging amplitude of individual fish through real-time or offline video data.

[0003] Existing fish video tracking technologies are mainly divided into two categories: traditional machine vision methods and deep learning methods. However, in actual complex aquarium environments, they still have the following significant drawbacks: Detection instability caused by water surface reflection and ripples: Water tank testing environments are often accompanied by water flow fluctuations and dynamic lighting changes. Existing technologies, when performing background subtraction or motion detection, are prone to mistaking water surface reflections and slow, localized water ripples for moving objects. Traditional machine vision tracking patents often rely on pre-extracted fixed background templates. In real water flow environments, factors such as water ripples and lighting changes can significantly reduce the effectiveness of fixed background templates.

[0004] On the other hand, while deep learning-based methods can overcome some of the effects of lighting and water surface ripples, they require manually acquiring labeled images as initial training samples for extensive iterative training. Such methods have extremely high generalization costs; once the resolution, frame rate, or camera intrinsics of the experimental video change, existing techniques often require re-labeling and retraining or extensive manual parameter tuning. More importantly, existing techniques typically only use single-path inter-frame differencing or simple fixed background matting when dealing with noise. In a real flowing water tank environment, subtle water ripples and flickering reflections cause continuous changes in pixels between adjacent frames, easily breaking through rigid, fixed hard thresholds and introducing large areas of noisy connected regions into the detection image, leading to serious false detections.

[0005] Severe interference from multiple targets' nearest neighbors: When multiple fish swim close together, cross each other, or block each other, conventional global detection or simple local search can easily mistake neighboring fish for the current target, causing trajectory drift or ID jump.

[0006] Existing fish tracking technologies typically rely on post-processing feature matching and trajectory connection to address occlusion issues. However, in aquarium experiments, when multiple fish swim close together or frequently intersect, conventional global detection or simple local prediction can easily mix pixel features of neighboring fish into the current target, leading to frequent identity jumps or trajectory drifts. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides a method for real-time calculation of individual fish tracking and tail sway motion parameters in a tank environment. This method solves the problems of existing technologies, such as the susceptibility of local observations to interference from nearby targets leading to trajectory drift when multiple targets are swimming at close range, and the instability of tail end feature extraction, which in turn leads to unreliable calculation results for tail sway frequency and amplitude.

[0008] To achieve the aforementioned objectives, the technical solution adopted by this invention is as follows: a method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment, comprising: Real-time acquisition of video frames monitoring fish within the aquarium environment and identification of tracking targets; Perform distortion correction processing on the current fish monitoring video frame, and convert the distortion-corrected current fish monitoring video frame into a standard top view to obtain the current preprocessed frame; Establish a conversion scale between pixel coordinates and actual physical length using a reference object of known size; By constraining each pixel in the current preprocessed frame through parallel differential calculation channels, a joint response image is obtained; Image segmentation is performed on the joint response image to obtain a binary mask, and connected component analysis is performed on the binary mask to obtain several independent blocks; Each independent block is filtered based on the shape features of the tracked target, and the remaining independent blocks after filtering are defined as candidate targets. When a candidate target appears continuously within a set pixel radius for several frames, and the distance between it and the current trajectory node of the tracked target is greater than the set minimum distance, the candidate target is marked as the tracked target and the trajectory of the tracked target is updated as the latest trajectory node. Using the position of the latest trajectory node as the observation value, the position and velocity of the tracked target are predicted based on the Kalman tracking algorithm; Based on the position and velocity of the tracked target, skeleton extraction and three-level priority tail end determination are performed; Based on the determined tail end, obtain the normal projection tail swing data of the current frame; Based on the normal projection tailswing data of the current frame, adaptive time window and spectral feature analysis are performed to obtain the tailswing motion parameters.

[0009] Furthermore, the parallel differential calculation channel includes a background differential channel, an inter-frame motion detection channel, and a joint gating mechanism; The background difference channel is used to calculate the element-wise grayscale difference between the current preprocessed frame and the background base map, and to mark pixels whose absolute value of the grayscale difference is greater than a preset grayscale difference threshold as potential foreign object areas. The inter-frame motion detection channel is used to calculate the element-wise grayscale difference between the current preprocessed frame and the previous preprocessed frame to obtain the element-wise grayscale difference result. Based on the element-wise grayscale difference result, the median of the grayscale difference and the absolute median difference of the grayscale difference are determined. The median of the grayscale difference is accumulated with the absolute median difference of the grayscale difference by a set multiple to obtain the motion determination threshold. Based on the element-wise grayscale difference result, pixels with grayscale differences greater than the motion determination threshold are marked as regions with obvious instantaneous displacement. The joint gating mechanism is used to retain the background difference amplitude values ​​of pixels that are simultaneously marked as potential foreign object regions and regions with obvious instantaneous displacement, and set the background difference amplitude values ​​of the remaining pixels to 0 to obtain a joint response image; the background difference amplitude value is the absolute difference between the gray value of a pixel and the gray value of the corresponding pixel in the background image.

[0010] Furthermore, the background image is an image obtained by averaging several video frames (excluding fish) acquired in advance through the tank environment monitoring equipment; during real-time monitoring, the background image is dynamically updated based on the current pre-processed frame, specifically: Calculate the grayscale difference between each pixel in the current preprocessed frame and the corresponding pixel in the background image, and mark the pixels with grayscale differences less than the background update threshold as relatively stable regions; For relatively stable regions, the gray values ​​of the corresponding pixels in the current preprocessed frame and the background image are weighted and averaged to obtain the updated gray values ​​of the relatively stable regions. The background image is dynamically updated based on the grayscale values ​​updated in relatively stable regions.

[0011] Furthermore, the image segmentation of the joint response image specifically includes: Pixels in the joint response image whose background difference magnitude is not 0 are marked as valid pixels; Obtain the median of the background difference magnitude of the effective pixels in the joint response image, and calculate the absolute deviation between the background difference magnitude of each effective pixel and the median of the background difference magnitude. The median of the absolute deviations is taken as the absolute median deviation; The first threshold is obtained by adding the median of the background difference amplitude to the absolute median difference of a set multiple. The maximum value between the preset minimum segmentation threshold and the first threshold is used as the adaptive segmentation threshold; Pixels in the joint response image whose background difference magnitude is greater than or equal to the adaptive segmentation threshold are classified as foreground, and pixels in the joint response image whose background difference magnitude is less than the adaptive segmentation threshold are classified as background, thus completing image segmentation and obtaining a binary mask.

[0012] Furthermore, the state vector of the Kalman tracking algorithm includes the fish's centroid position and swimming speed within the current preprocessed frame. During the execution of the Kalman tracking algorithm, the search radius is dynamically updated based on the covariance matrix. Taking the currently predicted fish centroid position as the center, a local image patch is obtained based on the search radius, and the pixels covering the coverage area of ​​other tracked targets in the local image patch are set to 0 to obtain the search range. The search range is used as the boundary constraint of the fish centroid position.

[0013] Furthermore, the skeleton extraction and three-level priority tail-end determination are specifically as follows: Obtain the local image patch corresponding to the location of the tracked target; The fish skeleton is extracted by performing morphological closing and opening operations within local image patches. Determine whether to record the tail data of the previous frame. If yes, mark the endpoint that is closest in physical distance to the tail of the previous frame as the tail. Otherwise, determine whether the tracking target is stationary based on the speed of the tracking target. If yes, take the endpoint farthest from the geometric center of the tracking target as the tail. Otherwise, construct an endpoint vector with the geometric center of the tracking target as the starting point and the endpoint as the ending point. Based on the speed vector and the endpoint vector of the tracking target, take the endpoint that is opposite to the direction of movement of the tracking target as the tail.

[0014] Furthermore, the acquisition of the normal projection tailswing data of the current frame specifically involves: The direction of the unit vector pointing from the geometric center of the target to the tail end is taken as the longitudinal direction of the fish body; By rotating the fish's body orthogonally along its longitudinal direction, the normal direction is obtained. The projection result is obtained by projecting the tail vector, which points from the center of the tracked target set to the tail end, onto the normal vector using vector dot product. Multiply the projection result by the conversion scale to obtain the normal projection tailswing data of the current frame.

[0015] Furthermore, the tail swing motion parameters are obtained as follows: Based on a preset adaptive time window, extract the normal projection tailswing data of the most recent frames; if the extraction of the tail of the current frame fails, mark the normal projection tailswing data of the current frame as invalid data. Arrange the normal projection tailswing data of each frame in time sequence to obtain the tailswing displacement signal sequence; Calculate the effective data percentage of the tail swing displacement signal sequence. If the effective data percentage is greater than the preset effective threshold, perform window analysis; otherwise, do not process it. The window analysis specifically includes: Invalid data is filled by linear interpolation, and then centering, slow change trend removal and bandpass filtering within a set frequency range are performed in sequence to obtain the preprocessed tail swing displacement signal sequence. The instantaneous frequency of each frame in the preprocessed tailslip displacement signal sequence is obtained using Hilbert transform; Outliers in the instantaneous frequencies are removed to obtain the effective set of instantaneous frequencies; Determine whether the effective instantaneous frequency meets the quantity requirement. If so, take the median of the effective instantaneous frequencies as the tail swing frequency. Otherwise, take the reciprocal of the time interval between adjacent peaks in the preprocessed tail swing displacement signal sequence as the periodic frequency and take the average of the periodic frequencies in the preprocessed tail swing displacement signal sequence as the tail swing frequency. The difference between the a-quantile and b-quantile values ​​in the preprocessed tail swing displacement signal sequence is taken as the tail swing amplitude; where a is greater than a preset high quantile threshold; b is less than a preset low quantile threshold; and the high quantile threshold is greater than the low quantile threshold. The percentage of effective instantaneous frequencies is calculated based on the set of effective instantaneous frequencies. Based on the standard deviation of the instantaneous frequency of each frame in the preprocessed tail swing displacement signal sequence, the dimensionless data of the standard deviation is used as the degree of frequency fluctuation. The frequency confidence level is obtained by weighting and summing the effective data ratio, effective instantaneous frequency ratio, frequency fluctuation degree, dimensionless data of tail swing amplitude, and adaptive time window size of the tail swing displacement signal sequence. Determine if the frequency confidence level is greater than the preset confidence threshold. If so, directly output the tail swing frequency and tail swing amplitude. Otherwise, mark the tail swing frequency and tail swing amplitude as low confidence results.

[0016] The beneficial effects of this invention are as follows: the dual-path logic joint gating mechanism achieves pixel-level dynamic reflection and continuous water ripple suppression, and possesses zero-sample training cost and ultra-high real-time computational efficiency. In a flowing water tank environment, traditional single-path differential algorithms are easily broken down by continuous water ripples or local flickering reflections, leading to large-area false detections. This invention, through the combination of logic and gating, ensures that slight water ripples and reflections are unlikely to simultaneously possess high-intensity background differences and drastic instantaneous displacements at the same pixel point. Therefore, in the low-level gating stage, these physical noises are forcibly formatted as a pure black background, providing a near-zero noise pure data source for subsequent Kalman filter stable tracking. Furthermore, this mechanism is entirely based on pure mathematical statistics and the lowest-level digital image logic bit operations, allowing for adaptive recognition upon program startup. Since the computation process does not involve matrix multiplication or backpropagation of heavy deep learning networks, the computational overhead on the processor is extremely low, easily achieving ultra-high frame rate real-time data stream processing on ordinary CPU devices. By employing the three criteria for tail end determination and the method for constructing a normal projection displacement sequence, the calculation of tail swing displacement is made more consistent with the actual movement patterns of fish, which is beneficial to improving the calculation accuracy of tail swing frequency and amplitude. Through adaptive time windows, frequency band peak search, parabolic interpolation refinement, and confidence evaluation mechanisms, an integrated output of tail swing frequency, tail swing amplitude, and result reliability is achieved. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0019] like Figure 1 As shown, in one embodiment of the present invention, a method for real-time calculation of tail sway motion parameters of individual fish in a tank environment enhances the independence of target observation by introducing a local mutual exclusion measurement mechanism. It also addresses the problems of trajectory drift caused by interference from nearby targets during close-range swimming of multiple targets in existing technologies, as well as the instability of tail sway feature extraction, leading to unreliable calculation results for tail sway frequency and amplitude. This method achieves stable tracking of multiple individual fish and outputs dynamic parameters such as tail sway frequency and amplitude in real time. The method includes the following steps: Camera intrinsic parameter adaptive scaling and preprocessing; reading camera parameter files and video streams to obtain the current video resolution; real-time acquisition of fish monitoring video frames in the aquarium environment and determination of tracking targets; Based on the ratio of the current video width / height to the calibrated width / height, the focal length and principal point parameters in the camera intrinsics are linearly scaled to ensure that the distortion correction module is compatible with video sources of different resolutions.

[0020] Determine the video resolution, and based on the ratio of the video resolution to the calibration resolution, linearly scale the focal length and principal point parameters in the camera intrinsics:

[0021]

[0022]

[0023]

[0024] in, This is the focal length in the X direction after linear scaling; The focal length is in the X direction; The resolution in the X direction of the video; The resolution in the calibrated X direction; This is the focal length in the Y direction after linear scaling; The focal length is in the Y direction; The x-coordinate of the principal point after linear scaling; The x-coordinate of the principal point; The resolution in the Y direction of the video; The resolution in the calibrated Y direction; The ordinate of the principal point after linear scaling; The ordinate of the main point.

[0025] Based on the linearly scaled focal length and principal point parameters, distortion correction is performed on the current fish monitoring video frame. By selecting the corner point of the perspective area through user interaction, the current fish monitoring video frame after distortion removal is converted into a standard top view to obtain the current preprocessed frame. By drawing a scale of known length on the image, a conversion scale between pixel coordinates and actual physical length (millimeters / meters) is established. This conversion scale is used to subsequently convert pixel displacement and velocity into actual physical units. Simultaneously, a recognition region is defined on the image, and a region mask is generated accordingly. Subsequent background modeling, target detection, and tracking operations are all confined to this recognition region, with data outside the region set to zero.

[0026] Constraining each pixel in the current preprocessed frame through parallel differential calculation channels, performing logical AND operations at the pixel level to remove dynamic noise of water flow and obtain a joint response image; Image segmentation is performed on the joint response image to obtain a binary mask, and connected component analysis is performed on the binary mask to obtain several independent blocks; Each independent block is filtered based on the shape features of the tracked target, such as eccentricity, solidity, and aspect ratio. The remaining independent blocks after filtering are defined as candidate targets. To filter random noise, candidate targets are confirmed through multi-frame verification: when a candidate target appears consecutively within a set pixel radius for several frames, and the distance between it and the current trajectory node of the tracked target is greater than a set minimum distance, the candidate target is marked as the tracked target, and the tracked target's trajectory is updated using the candidate target as the latest trajectory node. The target trajectory includes two parts: real-time tracking status and historical time-series records. The real-time tracking status corresponds to a unique target number and includes at least the target's current lateral position, longitudinal position, lateral velocity, longitudinal velocity, state uncertainty, the time of the most recent successful detection, and the number of consecutive frames without detection. Historical time-series records are saved in video frame order. Each record includes at least the frame number, video time, target number, lateral and longitudinal coordinates converted to actual physical units, and movement speed. It can also further record parameters such as tail swing frequency and tail swing amplitude. After a candidate target is confirmed through multiple frames, it is assigned a unique number, and a new tracking state is initialized with the center position of the candidate target's region. The results of each frame are then connected in chronological order to form the historical trajectory corresponding to the target number.

[0027] Using the position of the latest trajectory node as the observation value, the position and velocity of the tracked target are predicted based on the Kalman tracking algorithm. A Kalman filter is established for each target to predict its position and velocity. When updating observations within the local search area of ​​the predicted position, a local mutual exclusion measurement strategy is adopted: within the local image patch of the current target, the pixel values ​​of the predicted positions and coverage areas of other known targets are forcibly set to zero to completely shield neighbor interference. At the same time, the search radius is dynamically expanded based on the covariance matrix trace of the Kalman prediction and the number of consecutive lost frames. If the first high-standard measurement fails, a rescue measurement with expanded range and relaxed threshold is triggered.

[0028] Based on the position and velocity of the tracked target, skeleton extraction and three-level priority tail end determination are performed; Based on the determined tail end, obtain the normal projection tail swing data of the current frame; Based on the normal projection tailswing data of the current frame, adaptive time window and spectral feature analysis are performed to obtain the tailswing motion parameters.

[0029] The parallel differential computation channel includes a background differential channel, an inter-frame motion detection channel, and a joint gating mechanism; The background difference channel is used to calculate the element-wise grayscale difference between the current preprocessed frame and the background base map, and to mark pixels whose absolute value of the grayscale difference is greater than a preset grayscale difference threshold as potential foreign object areas. The inter-frame motion detection channel is used to calculate the absolute difference in grayscale between the current frame and the previous frame in real time, thereby capturing instantaneous displacement signals. To enable this channel to adapt to water flow disturbances, a fixed judgment number is abandoned in favor of an adaptive dynamic calculation strategy from mathematical statistics. In each frame, the statistical median of all pixel changes within the recognition area is calculated in real time, and the absolute median difference, representing the degree of pixel fluctuation dispersion, is further calculated. By linearly accumulating this median with a set multiple of the absolute median difference, a sensitive dynamic threshold for motion detection is adaptively determined for each frame. A pixel at a certain location will be accepted by the motion channel only when the inter-frame pixel change exceeds the dynamic threshold. Specifically: the element-wise grayscale difference between the current preprocessed frame and the previous preprocessed frame is calculated to obtain the element-wise grayscale difference result. Based on the element-wise grayscale difference result, the median of the grayscale difference and the absolute median difference of the grayscale difference are determined. The median of the grayscale difference is added to the absolute median difference of the grayscale difference by a set multiple to obtain the motion determination threshold. Based on the element-wise grayscale difference result, pixels with grayscale differences greater than the motion determination threshold are marked as regions with significant instantaneous displacement. After the two channels have completed their parallel calculations, a logical AND operation is performed on the two differential response images at the pixel level. This joint decision gate requires that a pixel in the tank simultaneously meet two conditions: "undergoing significant instantaneous displacement (passing the dynamic inter-frame threshold)" and "significantly different from the long-term static background (passing the fixed background threshold)." Only pixels that pass both screening criteria will have their background difference amplitude values ​​retained and aggregated into the final joint response image (candidate fish feature map). Any pixel response that only meets a single condition will be forcibly erased and set to zero by this gating mechanism. Specifically, the joint gating mechanism retains the background difference amplitude values ​​of pixels simultaneously marked as potential foreign object regions and regions experiencing significant instantaneous displacement, and sets the background difference amplitude values ​​of the remaining pixels to 0, thus obtaining the joint response image. The background difference amplitude value is the absolute difference between the grayscale value of a pixel and the grayscale value of the corresponding pixel in the background image. The larger the background difference amplitude value, the more significant the change in that pixel relative to the background.

[0030] The background image is an image obtained by averaging several video frames (corresponding pixels) acquired in advance through a water tank environment monitoring device, excluding fish. To prevent the background from becoming invalid due to long-term light drift in the water environment, the system introduces a dynamic background update mechanism. During operation, stable areas without fish are slowly assimilated with very low weight to ensure the timeliness of the background image. Specifically: Calculate the grayscale difference between each pixel in the current preprocessed frame and the corresponding pixel in the background image, and mark the pixels with grayscale differences less than the background update threshold as relatively stable regions; For relatively stable regions, the gray values ​​of the corresponding pixels in the current preprocessed frame and the background image are weighted and averaged to obtain the updated gray values ​​of the relatively stable regions. The background image is dynamically updated based on the grayscale values ​​updated in relatively stable regions.

[0031] The image segmentation of the joint response image specifically involves: Pixels in the joint response image whose background difference magnitude is not 0 are marked as valid pixels; Obtain the median of the background difference magnitude of the effective pixels in the joint response image, and calculate the absolute deviation between the background difference magnitude of each effective pixel and the median of the background difference magnitude. The median of the absolute deviations is taken as the absolute median deviation; The first threshold is obtained by adding the median of the background difference amplitude to the absolute median difference of a set multiple. The maximum value between the preset minimum segmentation threshold and the first threshold is used as the adaptive segmentation threshold; Pixels in the joint response image with a background difference magnitude greater than or equal to the adaptive segmentation threshold are classified as foreground, and pixels with a background difference magnitude less than the adaptive segmentation threshold are classified as background, thus completing image segmentation and obtaining a binary mask. In one specific embodiment, the absolute median difference is set to six, and the minimum segmentation threshold is set to fifty-five gray levels.

[0032] The state vector of the Kalman tracking algorithm includes the fish's centroid position and swimming speed within the current preprocessed frame. During the execution of the Kalman tracking algorithm, the search radius is dynamically updated based on the covariance matrix. Using the currently predicted fish centroid position as the center, a local image patch is obtained based on the search radius. The pixels covering the coverage area of ​​other tracked targets within the local image patch are set to 0, resulting in the search range, which serves as the boundary constraint for the fish centroid position. In each frame, the position and speed of the tracked target in the current frame are predicted based on the position, speed, and state uncertainty (i.e., the covariance matrix) of the target in the previous frame. Subsequently, a rectangular region is extracted in real-time from the joint response image, centered on the predicted position of the current target, according to the dynamic search radius. This rectangular region is the "local image patch." The predicted positions of other known targets are the center coordinates obtained from the state prediction of each of the other numbered targets in the same frame. Their coverage area can be determined based on the preset target box size, which in this embodiment is approximately a circular area with the prediction center as the center and a radius of 80% of half the target box size. When measuring the current target, the pixel responses of pixels within the local image patch that belong to the coverage area of ​​other targets are set to zero to reduce interference caused by neighboring fish.

[0033] The search radius is determined by the base search radius, the uncertainty of the current position prediction, and the number of consecutive undetected frames. The more uncertain the position prediction and the longer the consecutive loss time, the larger the search radius; however, a maximum range is set to avoid the search area expanding indefinitely. In this embodiment, the maximum search radius does not exceed four times the base search radius. This search radius is directly used to determine the cropping range of local image patches. If the first measurement fails, the search range is further expanded and the measurement conditions are appropriately relaxed for a second search. After a successful measurement, the position and velocity of the current target are corrected using the measurement center; if the measurement fails, the prediction result is retained and the number of consecutively lost frames is accumulated.

[0034] The process of skeleton extraction and three-level priority tail-end determination is as follows: Obtain the local image patch corresponding to the location of the tracked target; The fish skeleton is extracted by performing morphological closing and opening operations within local image patches. Determine whether to record the tail data of the previous frame. If yes, mark the endpoint that is closest in physical distance to the tail of the previous frame as the tail. Otherwise, determine whether the tracking target is stationary based on the speed of the tracking target. If yes, take the endpoint farthest from the geometric center of the tracking target as the tail. Otherwise, construct an endpoint vector with the geometric center of the tracking target as the starting point and the endpoint as the ending point. Based on the speed vector and the endpoint vector of the tracking target, take the endpoint that is opposite to the direction of movement of the tracking target as the tail.

[0035] The acquisition of the normal projection tail swing data of the current frame excludes the influence of the overall translation and rotation of the fish body. Specifically: The direction of the unit vector pointing from the geometric center of the target to the tail end is taken as the longitudinal direction of the fish body; By rotating the fish's body orthogonally along its longitudinal direction, the normal direction is obtained. The projection result is obtained by projecting the tail vector, which points from the center of the tracked target set to the tail end, onto the normal vector using vector dot product. Multiply the projection result by the conversion scale to obtain the normal projection tailswing data of the current frame.

[0036] The obtained tail swing motion parameters are specifically as follows: Based on a preset adaptive time window, extract the normal projection tailswing data of the most recent frames; if the extraction of the tail of the current frame fails, mark the normal projection tailswing data of the current frame as invalid data. Arrange the normal projection tailswing data of each frame in time sequence to obtain the tailswing displacement signal sequence; Calculate the effective data percentage of the tail swing displacement signal sequence. If the effective data percentage is greater than the preset effective threshold, perform window analysis; otherwise, do not process it. The window analysis specifically includes: Invalid data is filled by linear interpolation, and then centering, slow change trend removal and bandpass filtering within a set frequency range are performed in sequence to obtain the preprocessed tail swing displacement signal sequence. The instantaneous frequency of each frame in the preprocessed tailslip displacement signal sequence is obtained using Hilbert transform; Outliers in the instantaneous frequencies are removed to obtain the effective set of instantaneous frequencies; Determine whether the effective instantaneous frequency meets the quantity requirement. If so, take the median of the effective instantaneous frequencies as the tail swing frequency. Otherwise, take the reciprocal of the time interval between adjacent peaks in the preprocessed tail swing displacement signal sequence as the periodic frequency and take the average of the periodic frequencies in the preprocessed tail swing displacement signal sequence as the tail swing frequency. The difference between the a quantile and the b quantile in the preprocessed tail swing displacement signal sequence is used as the tail swing amplitude; where a is greater than a preset high quantile threshold; b is less than a preset low quantile threshold; the high quantile threshold is greater than the low quantile threshold; in this embodiment, the difference between the 95th quantile and the 5th quantile is used as the tail swing amplitude.

[0037] The percentage of effective instantaneous frequencies is calculated based on the set of effective instantaneous frequencies. Based on the standard deviation of the instantaneous frequency of each frame in the preprocessed tail swing displacement signal sequence, the dimensionless data of the standard deviation is used as the degree of frequency fluctuation. The frequency confidence level is obtained by weighting and summing the effective data ratio, effective instantaneous frequency ratio, frequency fluctuation degree, dimensionless data of tail swing amplitude, and adaptive time window size of the tail swing displacement signal sequence. Determine if the frequency confidence level is greater than the preset confidence threshold. If so, directly output the tail swing frequency and tail swing amplitude. Otherwise, mark the tail swing frequency and tail swing amplitude as low confidence results.

[0038] In this embodiment, all length-related parameters (such as search radius and filter kernel size), area-related parameters (such as minimum fish body area), and variance-related parameters (such as Kalman measurement noise) are automatically calculated during initialization according to the video resolution scaling ratio. Specifically, the scaling ratios of the current video width and height relative to the reference resolution are calculated and adjusted according to the physical meaning of the parameters. Since the camera focal length and principal point coordinates have clear horizontal and vertical directions, the horizontal focal length and principal point x-coordinate are scaled according to the width ratio, and the vertical focal length and principal point y-coordinate are scaled according to the height ratio. Length-related parameters such as search radius, target box size, tail search range, Gaussian smoothing scale, and morphological processing radius are uniformly scaled using the smaller value between the width and height ratios in this embodiment to avoid excessive enlargement of the processing range due to non-uniform resolution changes. Area-related parameters such as minimum fish body area and minimum tail area are scaled according to the product of the width and height ratios. Parameters with square dimensions, such as process noise, measurement noise, and initial uncertainty in state prediction, are adjusted according to the square of the length scaling ratio. Background parameters such as frame rate, detection interval, and target loss tolerance time are first expressed in physical time and then converted into frame rate based on the current video frame rate.

[0039] Although the area is ultimately represented by a numerical value, this value indicates the size of a two-dimensional pixel region. Therefore, it should be adjusted according to the product of the horizontal and vertical scaling ratios, rather than using only the ratio in one direction. In one specific embodiment, the reference resolution is 1920 x 1080, and the reference frame rate is 30 frames per second. For videos with significant non-uniform scaling, it is preferable to use a video source with the same aspect ratio as the calibration video, or to adjust the directional length parameters separately in the horizontal and vertical directions.

[0040] In addition, a delay hiding mechanism is set up to keep the prediction box displayed for a few frames during which the target is briefly lost, to prevent UI flickering. After the maximum tolerance time is exceeded, the trajectory is completely deleted and the time series table is exported.

Claims

1. A method for real-time calculation of individual fish tracking and tail sway motion parameters in a fish tank environment, characterized in that, include: Real-time acquisition of video frames monitoring fish within the aquarium environment and identification of tracking targets; Perform distortion correction processing on the current fish monitoring video frame, and convert the distortion-corrected current fish monitoring video frame into a standard top view to obtain the current preprocessed frame; Establish a conversion scale between pixel coordinates and actual physical length using a reference object of known size; By constraining each pixel in the current preprocessed frame through parallel differential calculation channels, a joint response image is obtained; Image segmentation is performed on the joint response image to obtain a binary mask, and connected component analysis is performed on the binary mask to obtain several independent blocks; Each independent block is filtered based on the shape features of the tracked target, and the remaining independent blocks after filtering are defined as candidate targets. When a candidate target appears continuously within a set pixel radius for several frames, and the distance between it and the current trajectory node of the tracked target is greater than the set minimum distance, the candidate target is marked as the tracked target and the trajectory of the tracked target is updated as the latest trajectory node. Using the position of the latest trajectory node as the observation value, the position and velocity of the tracked target are predicted based on the Kalman tracking algorithm; Based on the position and velocity of the tracked target, skeleton extraction and three-level priority tail end determination are performed; Based on the determined tail end, obtain the normal projection tail swing data of the current frame; Based on the normal projection tailswing data of the current frame, adaptive time window and spectral feature analysis are performed to obtain the tailswing motion parameters.

2. The method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment according to claim 1, characterized in that, The parallel differential computation channel includes a background differential channel, an inter-frame motion detection channel, and a joint gating mechanism; The background difference channel is used to calculate the element-wise grayscale difference between the current preprocessed frame and the background base map, and to mark pixels whose absolute value of the grayscale difference is greater than a preset grayscale difference threshold as potential foreign object areas. The inter-frame motion detection channel is used to calculate the element-by-element grayscale difference between the current preprocessed frame and the previous preprocessed frame to obtain the element-by-element grayscale difference result. Based on the element-by-element grayscale difference result, the median of the grayscale difference and the absolute median difference of the grayscale difference are determined. The median of the grayscale difference and the absolute median difference of the grayscale difference by a set multiple are accumulated to obtain the motion judgment threshold. Based on the element-by-element grayscale difference results, pixels with grayscale differences greater than the motion determination threshold are marked as regions where significant instantaneous displacement has occurred; The joint gating mechanism is used to retain the background difference amplitude values ​​of pixels that are simultaneously marked as potential foreign object regions and regions with obvious instantaneous displacement, and set the background difference amplitude values ​​of the remaining pixels to 0 to obtain a joint response image; the background difference amplitude value is the absolute difference between the gray value of a pixel and the gray value of the corresponding pixel in the background image.

3. The method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment according to claim 2, characterized in that, The background image is an image obtained by averaging several video frames (excluding fish) acquired in advance through a tank environment monitoring device. During real-time monitoring, the background image is dynamically updated based on the current pre-processed frame. Specifically: Calculate the grayscale difference between each pixel in the current preprocessed frame and the corresponding pixel in the background image, and mark the pixels with grayscale differences less than the background update threshold as relatively stable regions; For relatively stable regions, the gray values ​​of the corresponding pixels in the current preprocessed frame and the background image are weighted and averaged to obtain the updated gray values ​​of the relatively stable regions. The background image is dynamically updated based on the grayscale values ​​updated in relatively stable regions.

4. The method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment according to claim 1, characterized in that, The image segmentation of the joint response image specifically involves: Pixels in the joint response image whose background difference magnitude is not 0 are marked as valid pixels; Obtain the median of the background difference magnitude of the effective pixels in the joint response image, and calculate the absolute deviation between the background difference magnitude of each effective pixel and the median of the background difference magnitude. The median of the absolute deviations is taken as the absolute median deviation; The first threshold is obtained by adding the median of the background difference amplitude to the absolute median difference of a set multiple. The maximum value between the preset minimum segmentation threshold and the first threshold is used as the adaptive segmentation threshold; Pixels in the joint response image whose background difference magnitude is greater than or equal to the adaptive segmentation threshold are classified as foreground, and pixels in the joint response image whose background difference magnitude is less than the adaptive segmentation threshold are classified as background, thus completing image segmentation and obtaining a binary mask.

5. The method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment according to claim 1, characterized in that, The state vector of the Kalman tracking algorithm includes the fish's centroid position and swimming speed in the current preprocessed frame. During the execution of the Kalman tracking algorithm, the search radius is dynamically updated according to the covariance matrix. Based on the currently predicted fish centroid position, a local image patch is obtained, and the pixels covering the coverage area of ​​other tracked targets in the local image patch are set to 0 to obtain the search range. The search range is used as the boundary constraint of the fish centroid position.

6. The method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment according to claim 1, characterized in that, The process of skeleton extraction and three-level priority tail-end determination is as follows: Obtain the local image patch corresponding to the location of the tracked target; The fish skeleton is extracted by performing morphological closing and opening operations within local image patches. Determine whether to record the tail data of the previous frame. If yes, mark the endpoint that is closest in physical distance to the tail of the previous frame as the tail. Otherwise, determine whether the tracking target is stationary based on the speed of the tracking target. If yes, take the endpoint farthest from the geometric center of the tracking target as the tail. Otherwise, construct an endpoint vector with the geometric center of the tracking target as the starting point and the endpoint as the ending point. Based on the speed vector and the endpoint vector of the tracking target, take the endpoint that is opposite to the direction of movement of the tracking target as the tail.

7. The method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment according to claim 1, characterized in that, The acquisition of the normal projection tailswing data of the current frame specifically involves: The direction of the unit vector pointing from the geometric center of the target to the tail end is taken as the longitudinal direction of the fish body; By rotating the fish's body orthogonally along its longitudinal direction, the normal direction is obtained. The projection result is obtained by projecting the tail vector, which points from the center of the tracked target set to the tail end, onto the normal vector using vector dot product. Multiply the projection result by the conversion scale to obtain the normal projection tailswing data of the current frame.

8. The method for real-time calculation of individual fish tracking and tail sway motion parameters in a water tank environment according to claim 1, characterized in that, The obtained tail swing motion parameters are specifically as follows: Based on a preset adaptive time window, extract the normal projection tailswing data of the most recent frames; if the extraction of the tail of the current frame fails, mark the normal projection tailswing data of the current frame as invalid data. Arrange the normal projection tailswing data of each frame in time sequence to obtain the tailswing displacement signal sequence; Calculate the effective data percentage of the tail swing displacement signal sequence. If the effective data percentage is greater than the preset effective threshold, perform window analysis; otherwise, do not process it. The window analysis specifically includes: Invalid data is filled by linear interpolation, and then centering, slow change trend removal and bandpass filtering within a set frequency range are performed in sequence to obtain the preprocessed tail swing displacement signal sequence. The instantaneous frequency of each frame in the preprocessed tailslip displacement signal sequence is obtained using Hilbert transform; Outliers in the instantaneous frequencies are removed to obtain the effective set of instantaneous frequencies; Determine whether the effective instantaneous frequency meets the quantity requirement. If so, take the median of the effective instantaneous frequencies as the tail swing frequency. Otherwise, take the reciprocal of the time interval between adjacent peaks in the preprocessed tail swing displacement signal sequence as the periodic frequency and take the average of the periodic frequencies in the preprocessed tail swing displacement signal sequence as the tail swing frequency. The difference between the a-quantile and b-quantile values ​​in the preprocessed tail swing displacement signal sequence is taken as the tail swing amplitude; where a is greater than a preset high quantile threshold; b is less than a preset low quantile threshold; and the high quantile threshold is greater than the low quantile threshold. The percentage of effective instantaneous frequencies is calculated based on the set of effective instantaneous frequencies. Based on the standard deviation of the instantaneous frequency of each frame in the preprocessed tail swing displacement signal sequence, the dimensionless data of the standard deviation is used as the degree of frequency fluctuation. The frequency confidence level is obtained by weighting and summing the effective data ratio, effective instantaneous frequency ratio, frequency fluctuation degree, dimensionless data of tail swing amplitude, and adaptive time window size of the tail swing displacement signal sequence. Determine if the frequency confidence level is greater than the preset confidence threshold. If so, directly output the tail swing frequency and tail swing amplitude. Otherwise, mark the tail swing frequency and tail swing amplitude as low confidence results.

Images