Fish respiratory rate detection method based on YOLO segmentation and optical flow tracking

Abstract

The invention relates to a fish respiratory rate detection method based on YOLO segmentation and optical flow tracking, and belongs to the technical field of aquatic organism physiological status monitoring. The method comprises the steps of video data input and initialization, video preprocessing and frame rate dynamic calibration, gill segmentation by a YOLO model and target screening, gill region-of-interest extraction, optical flow feature point tracking and dynamic maintenance, gill motion displacement calculation and signal optimization, respiratory rate calculation and smooth optimization, and result output and visualization. Under complex scenes such as water wave interference and fish swimming, the respiratory rate calculation error is smaller than or equal to + / -2 BPM, the effective data rate is larger than or equal to 95%, and the method is superior to the prior art (for example, the patent CN120599516A) and can be applied to scenes such as aquaculture health early warning and aquatic organism physiological experiments.

Description

Technical Field

[0001] This invention relates to the field of aquatic organism physiological state monitoring technology, specifically to a method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking. Background Technology

[0002] Fish respiratory rate is a key indicator reflecting their physiological health and environmental adaptability. In aquaculture, a sudden increase in respiratory rate often indicates water quality deterioration (such as decreased dissolved oxygen or excessive ammonia nitrogen) or disease infection. In aquatic biological experiments, changes in respiratory rate are a core basis for assessing the physiological effects of factors such as drug toxicity and temperature stress on fish. However, existing fish respiratory rate detection technologies have many insurmountable shortcomings: 1. Manual counting method: The traditional mainstream method relies on staff to observe the number of times the fish gill covers open and close with their naked eyes. It is extremely inefficient (a single person can only monitor 5 to 10 fish per day). Moreover, long-term observation can easily cause visual fatigue. The counting error is > ±8 BPM. It is highly subjective and cannot meet the needs of large-scale, continuous monitoring. 2. Contact sensor method: By attaching accelerometers or pressure sensors to the surface of the fish to collect respiratory motion data, although automated monitoring can be achieved, the sensor attachment process can easily cause stress in the fish and interfere with normal breathing. It is not suitable for small fish such as fry (where the sensor cannot be fixed to the body surface) or fish with abundant mucus on their body surface (where the sensor is easy to fall off), and has poor versatility. 3. Traditional Computer Vision Methods: Existing technologies include improved solutions based on computer vision, such as patent CN120599516A (publication date 2025.09.05, title "Fish Respiration Monitoring Method and System"). This solution outputs rectangular bounding boxes of the fish's mouth and gills through a hybrid cascaded network model, combines optical flow algorithms to extract time-series motion signals, and then calculates the respiratory rate and determines the fish's body pressure level. However, this solution still has the following technical shortcomings, which limit its applicability in practical scenarios: Insufficient target positioning accuracy: Using a rectangular bounding box to locate the fish mouth and gills can only define the general area and cannot distinguish between the gills and surrounding non-target tissues such as fins and scales. The proportion of non-target pixels within the bounding box exceeds 30%, which causes subsequent optical flow tracking points to easily drift to non-breathing areas, introducing invalid motion interference. The tracking point drift rate is >30%. Poor stability of optical flow tracking: Only median filtering is used to remove abnormal components of optical flow, and no dynamic maintenance mechanism for the number of feature points is designed. When the fish swims quickly or the gill movement signal is weak (displacement amplitude < 0.02%), the number of effective tracking points is likely to be less than 20, which leads to the interruption of displacement calculation and the reliability of continuous monitoring is < 80%. Weak signal processing capability: It only relies on Symlet wavelet denoising to eliminate baseline drift, and does not design a signal enhancement strategy for gill micro-movements. Weak signals are easily drowned out by noise such as water ripples and bubbles. The frequency calculation error exceeds ±5 BPM, and the error is further amplified in turbid water and water flow disturbance scenarios. Poor engineering adaptability: The matching between video reading speed and hardware performance was not considered, and no dynamic frame rate calibration mechanism was designed. When the deviation between the nominal frame rate and the actual frame rate exceeds 1 second, the inaccurate timestamp leads to deviation in the breathing cycle calculation. At the same time, the GPU memory usage was not optimized, and long-term monitoring (>30 minutes) is prone to interruption due to insufficient memory, which cannot meet the actual needs of "24-hour continuous monitoring" in aquaculture.

[0003] To address the shortcomings of existing technologies, there is an urgent need for a fish respiratory frequency detection method with characteristics of "high-precision positioning, stable tracking, strong anti-interference, and engineering adaptability" to solve the monitoring pain points in complex underwater environments and meet the accuracy and stability requirements of practical application scenarios. Summary of the Invention

[0004] This invention addresses the aforementioned technical problems by providing a method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking.

[0005] The present invention achieves the above objectives through the following methods: A method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking, the method includes video data input and initialization, video preprocessing and dynamic frame rate calibration, YOLO model segmentation of gills and target selection, extraction of gill region of interest (ROI), optical flow feature point tracking and dynamic maintenance, calculation of gill motion displacement and signal optimization, calculation of respiratory frequency and smoothing optimization, and result output and visualization. The optical flow feature point tracking and dynamic maintenance described above involves converting the current video frame into a grayscale image, cropping a grayscale sub-image within the view ROI, generating a mask sub-image for the ROI region based on the target mask, performing Canny edge detection on the grayscale sub-image, preserving the edges within the mask, and obtaining an edge-enhanced image. A three-level dynamic feature point maintenance mechanism of "initialization-filtering-supplementation" is employed for the edge-enhanced image to ensure that the number of effective tracking points remains stable between 30 and 150. The initialization process involves using Shi-Tomasi corner detection to extract feature points from the edge enhancement image. If fewer than 30 feature points are detected, random sampling is performed within the mask sub-image and Gaussian noise is added. The noise-processed supplementary points are then merged with the remaining valid feature points to ensure that the total number of feature points remains between 30 and 150. The filtering process involves tracking the position of feature points between adjacent frames. The input consists of the previous frame's grayscale image, the current frame's grayscale image, and the feature points from the previous frame. The output is the current frame's feature points and tracking status, with 1 indicating successful tracking. Valid points that are "successfully tracked + within the image range + within the mask" are retained, with dual spatial constraints ensuring point validity. When the number of valid points is 30-80, the number of supplementary points is ≤20. When the number of valid points is <30, the points are re-initialized. The minimum distance between the supplementary points and existing points is ≥1 pixel to avoid clustering.

[0006] Furthermore, the video data input and initialization are as follows: the YOLO instance segmentation model is used to output a gill pixel-level mask with a segmentation accuracy of ≥90%; during inference, masks with a confidence level of ≥0.5 are selected, and the mask with the largest area is selected as the target mask. The core parameters are initialized as follows: sliding data window size of 180 frames, optical flow feature point number threshold of 30~150, and abnormal displacement judgment range of <0.02% or >20%.

[0007] Furthermore, the video preprocessing and frame rate dynamic calibration involve performing size normalization on each frame of video to control the timestamp error to ≤0.1 seconds.

[0008] Furthermore, the YOLO model segmentation of gills and target selection involves performing YOLO model inference on the normalized video frames and selecting the mask with the largest effective pixel area as the target mask.

[0009] Further, the gill region of interest (ROI) is extracted: contour detection is performed on the target mask to obtain all the outer contours of the mask; based on the contour area sorting, the contour with the largest area is selected as the core contour of the gill, and the minimum bounding rectangle of the core contour is calculated to obtain the rectangle parameters.

[0010] Furthermore, the calculation of gill movement displacement and signal optimization are as follows: Displacement vector calculation: For successfully tracked feature points in adjacent frames, calculate the Euclidean distance of each feature point to obtain a set of displacement vectors; Abnormal displacement filtering: Filter outliers in displacement vectors. If the number of valid displacement vectors after filtering is less than 5, relax the filtering conditions to a range of 5σ, where σ is the standard deviation. If the number of valid displacement vectors is still insufficient, sort the displacement values ​​from smallest to largest and take the top 10 displacement values. Relative displacement normalization: Calculate the diagonal length of the ROI, divide the mean of the effective displacement vector by the diagonal length and multiply by 100 to obtain the gill relative displacement expressed as a percentage. Weak signal enhancement: When there are ≥10 stored relative displacement historical data, the average of the most recent 20 historical data is calculated as the signal baseline; if the fluctuation amplitude of the current relative displacement and the baseline is <0.02%, a sinusoidal target signal is generated based on the target respiratory rate to correct the current displacement; if the fluctuation amplitude is ≥0.02%, the fluctuation amplitude is amplified by 20 times and superimposed on the baseline to obtain the enhanced displacement signal, i.e. enhanced displacement. Baseline drift correction: When there are ≥20 historical relative displacement data, perform linear fitting (`np.polyfit`) on the most recent 20 historical data. If the absolute value of the fitting slope is >0.1, it is determined to be baseline drift. The baseline is recalibrated to the mean of the most recent 10 historical data, which is the historical displacement mean. The enhanced displacement signal is adjusted to "new baseline + (enhanced displacement - historical displacement mean)". The respiratory rate calculation and smoothing optimization are as follows: Sliding window data management: It adopts a first-in-first-out (FIFO) sliding window mechanism to store the 180 most recent enhanced displacement signals and their corresponding timestamps. When the data volume exceeds the window length, the oldest data is deleted. FFT frequency analysis: When the number of data points in the window is ≥30, the linear trend of the displacement signal is eliminated, and then FFT transformation is performed to calculate the frequency spectrum. Only the positive frequency components are retained. The frequency with the highest power within the target frequency ±0.2Hz range is selected as the dominant frequency. The dominant frequency is multiplied by 60 to convert it into respiratory rate per minute (BPM). Peak detection backup algorithm: If the BPM calculated by FFT is not in the range of 20~80, it is determined to be invalid and the backup algorithm is enabled: perform second-order Butterworth bandpass filtering and Savitzky-Golay smoothing on the displacement signal, identify the respiratory peak, and calculate BPM = 60 / average peak interval based on the interval between adjacent peaks. Frequency smoothing optimization: ① Jump limit: If the historical breathing frequency log is not empty, the current BPM is limited to the range of "previous frame BPM ± 3". If it exceeds the range, it is truncated to the boundary value; ② Moving average: The arithmetic average of the 5 most recent valid BPMs in the video frame sequence is performed. If the number of valid BPMs is less than 5, the target breathing frequency (30 BPM) or the historical average is used to fill the gap.

[0011] The beneficial effects of this invention compared to the prior art are as follows: 1. Improve target positioning accuracy: This invention replaces the traditional (CN120599516A) rectangular bounding box with a pixel-level segmentation mask. The mask precisely defines the gill area and completely eliminates interference from non-target areas such as fins and scales, reducing the optical flow tracking point drift rate from >30% to <5%. This solves the tracking point drift problem at its root. The specific effects are compared in Table 1. 2. Enhanced Optical Flow Tracking Stability: This invention overcomes the limitations of CN120599516A's "only removing anomalies without maintaining points" by combining quantity control and spatial filtering to ensure the rationality of tracking point density and distribution, avoiding computational interruptions due to insufficient point quantity. A three-level dynamic feature point maintenance mechanism of "initialization-screening-supplementation" is designed. In complex scenarios (rapid fish swimming), the number of effective tracking points is kept stable at 30~150, avoiding displacement calculation interruptions. The reliability of continuous monitoring is improved to ≥98%, and the continuous monitoring time is increased from <30 minutes to ≥60 minutes. Specific effect comparisons are shown in Table 1.

[0012] 3. Optimize robustness in weak signals: CN120599516A only passively denoises. This invention improves the signal-to-noise ratio from the signal generation stage through "active enhancement + dynamic correction" and "target signal guidance + baseline dynamic correction" strategies. It enhances the micro-movement signal of the gills, suppresses water ripple noise interference, and solves the problem of weak signals being submerged by noise. In weak signal scenarios (fish at rest), the frequency calculation error is reduced from ±5 BPM to within ±2 BPM, and the signal-to-noise ratio is improved by 60%. See Table 1 for a comparison of specific effects. 4. Ensuring the reliability of frequency calculation: CN120599516A relies on single peak detection. This invention, by "integrating FFT frequency analysis (main algorithm) and peak detection (backup algorithm)," combined with frequency jump limitation and moving average, reduces the probability of frequency calculation failure in extreme scenarios (turbid water, water flow disturbance scenarios) from >20% to <5%, and the frequency output standard deviation is ≤±1.8BPM, covering different signal scenarios and improving reliability in extreme environments. See Table 1 for a comparison of specific effects. Achieving Engineering Deployment: CN120599516A did not consider engineering deployment issues. This invention ensures stable operation over long periods through hardware and algorithm co-optimization. By dynamically calibrating the frame rate (timestamp error ≤ 0.1 seconds) and optimizing GPU memory (releasing the cache every 3 frames), the continuous monitoring time is increased from <30 minutes to ≥60 minutes, meeting the long-term monitoring requirements in practical applications. GPU memory usage is reduced from approximately 800MB to approximately 480MB, adapting to common hardware environments. Specific performance comparisons are shown in Table 1.

[0013] Table 1 Comparison between the present invention and the prior art . Attached Figure Description

[0014] Figure 1 Overall method flowchart; Figure 2 The results of gill and head segmentation masks for fish in different directions and postures are shown. The numbers represent the confidence level of the results. Images A and C (head facing left) and images B and D (head facing right) are shown. Figure 3The optical flow tracking results are shown in the comparison chart. In the chart, A is the optical flow tracking result under Seg Mask Mode, where the yellow feature points are always constrained within the gill mask without drifting, and the tracking stability is not affected by the pose change. In the chart, B is the optical flow tracking result under YOLO Box Mode, where the purple-red feature points are prone to drifting to the non-gill area within the bounding box, and the tracking stability decreases significantly with the pose change. Figure 4 A comparison of displacement signal and respiratory rate curves, where A is the respiratory motion curve based on segmentation mask and B is the respiratory motion curve based on bounding box. Detailed Implementation

[0015] The technical solution of the present invention will be further explained below through embodiments, but the scope of protection of the present invention is not limited in any way by the embodiments.

[0016] Hardware configuration: CPU is Intel Core i7-12700K, GPU is NVIDIA RTX 3060 (6GB VRAM), RAM is 16GB, and storage is 1TB SSD; underwater imaging device is Hikvision DS-2XS3T2676G0-IW, frame rate is set to 30fps, resolution is 1920×1080, equipped with 12V 10W LED cold light source (to avoid strong light reflection interference); Software environment: Operating system is Windows 10 64-bit, programming language is Python 3.9, dependent library versions: PyTorch 1.12.0, OpenCV-Python 4.6.0.66, Ultralytics 8.0.100, SciPy 1.9.1, NumPy 1.23.5, Matplotlib 3.6.2.

[0017] like Figure 1 As shown, a method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking is presented. The method includes video data input and initialization, video preprocessing and dynamic frame rate calibration, YOLO model segmentation of gills and target selection, extraction of gill region of interest (ROI) extraction, optical flow feature point tracking and dynamic maintenance, gill motion displacement calculation and signal optimization, respiratory frequency calculation and smoothing optimization, and result output and visualization. The specific steps are as follows: S10, Video Data Input and Initialization To acquire dynamic videos of fish, use an imaging device with a frame rate of ≥25fps (supporting pre-recording or real-time acquisition) to ensure that the gill area is not continuously obstructed in the video. Load the pre-trained YOLO instance segmentation model (trained based on the Ultralytics YOLOv11 framework), configure the GPU environment (supporting NVIDIA CUDA), and initialize the core parameters: sliding data window size 180 frames, optical flow feature point threshold 30~150, and abnormal displacement judgment range <0.02% or >20%; Record the program startup time for subsequent timestamp calibration and frame rate calculation.

[0018] S20, Video Preprocessing and Dynamic Frame Rate Calibration Perform size normalization on each frame of video to unify the resolution to 640×480 pixels, reducing the computational complexity of subsequent algorithms; Frame rate calibration is performed every 30 frames: the difference between "actual time" (current time - program startup time) and "theoretical time" (cumulative frame count / nominal video frame rate) is calculated. If the difference is greater than 1 second, the actual frame rate (`actual_fps=cumulative frame count / actual time`) is recalculated, and the frame interval (`frame_interval=1 / actual_fps`) is updated synchronously to ensure that the timestamp error is ≤0.1 seconds. S30 and YOLO models for gill segmentation and target selection YOLO model inference is performed on the normalized video frames. During the inference process, `torch.no_grad()` is enabled to disable gradient calculation and reduce GPU memory usage. Extract the segmentation masks with "category label = gill" (preset category ID is 1) and confidence ≥ 0.5 from the inference results, and calculate the effective pixel area of ​​each mask by `np.sum(mask_np>0.5)`; Select the mask with the largest effective pixel area as the target mask, and use bilinear interpolation algorithm to scale it to 640×480 pixels to ensure that it is perfectly matched with the video frame size; Specifically, the training process of the YOLO instance segmentation model includes: Training dataset construction: The dataset contains 2000 sample images, covering target fish in various poses. The image scenes cover strong light, weak light, clear water, and murky water (turbidity ≤ 50 NTU). Each image is labeled with a pixel-level segmentation mask of the gill region and a "gill" category label. Data augmentation: During the training phase, random horizontal flipping (probability 0.5), random rotation (±15°), brightness adjustment (±20%), and contrast adjustment (±15%) strategies are used to improve the model's generalization ability; Model training: Based on YOLOv11-seg, the Adam optimizer was used for 120 rounds of iterative training. The loss function used was DICE loss + cross-entropy loss. The model weights with the highest accuracy on the validation set were saved for inference.

[0019] S40, Extraction of Region of Interest (ROI) for the gills Perform contour detection on the target mask (`cv2.findContours`, search mode `cv2.RETR_EXTERNAL`, approximation method `cv2.CHAIN_APPROX_SIMPLE`) to obtain all the outer contours of the mask; As a specific implementation method: Based on the contour area sorting, the contour with the largest area is selected as the core contour of the gill. The minimum bounding rectangle of the core contour is calculated by `cv2.boundingRect` to obtain the rectangle parameters (x, y, w, h) (x and y are the coordinates of the upper left corner, and w and h are the width and height). As a specific implementation method, the rectangle parameters are calculated as follows: Based on the minimum bounding rectangle, a padding expansion area is added (padding length = max(w, h) × 20%). When clipping the ROI, the boundary constraints are satisfied: `x_roi = max(0, x-padding)`, `y_roi = max(0, y-padding)`, `x_roi_end = min(640, x+w+padding)`, `y_roi_end = min(480, y+h+padding)`. The final ROI parameters are (x_roi, y_roi, x_roi_end - x_roi, y_roi_end - y_roi). S50, Flow Feature Point Tracking and Dynamic Maintenance 5.1 Grayscale Conversion and Edge Processing: The current video frame is converted to grayscale using `cv2.cvtColor(frame,cv2.COLOR_BGR2GRAY)`, and a grayscale sub-image is extracted within the ROI. A mask sub-image of the ROI region is generated based on the target mask. Canny edge detection is performed on the grayscale sub-image, and the edges within the mask are preserved using `cv2.bitwise_and(edges, edges,mask=roi_mask)` to obtain an edge-enhanced image. 5.2 Feature Point Initialization: The Shi-Tomasi corner detection algorithm (`cv2.goodFeaturesToTrack`) is used to extract feature points from the edge enhancement image. If the number of detected feature points is less than 30, random sampling is performed in the mask sub-image (1 point is taken for every 50 effective pixels of the mask) and Gaussian noise (mean 0, standard deviation 0.3) is added to avoid over-idealization of the supplementary points. The noise-processed supplementary points are merged with the remaining effective feature points to ensure that the total number of feature points is maintained within a reasonable range of 30 to 150, which ensures the stability of the tracking calculation and avoids redundant calculations caused by excessive point density. 5.3 Optical Flow Tracking Execution: The Lucas-Carnard optical flow algorithm (`cv2.calcOpticalFlowPyrLK`) is used to track the position of feature points between adjacent frames; the inputs are the grayscale image of the previous frame (`prev_gray`), the grayscale image of the current frame (`curr_gray`), and the feature points of the previous frame (`prev_pts`), and the outputs the feature points of the current frame (`curr_pts`) and the tracking status (`status`, 1 indicates successful tracking); 5.4 Valid Feature Point Filtering and Maintenance: Retain feature points that are "tracked successfully + coordinates within the image range + coordinates located within the target mask"; if the number of valid feature points is less than 30, return to step 5.2 to reinitialize feature points; if the number is greater than or equal to 30 and less than 80, supplement ≤20 feature points (supplementation logic is the same as in step 5.2). Specifically, the supplemented feature points need to be filtered by Euclidean distance calculation, and the minimum distance between the supplemented feature points and existing points is greater than or equal to 1 pixel to avoid displacement calculation deviation caused by aggregation; S60, Gill Movement Displacement Calculation and Signal Optimization 6.1 Displacement Vector Calculation: For successfully tracked feature points in adjacent frames, calculate the Euclidean distance of each feature point using `np.linalg.norm(good_new good_old, axis=1)` to obtain a set of displacement vectors; 6.2 Abnormal Displacement Filtering: Based on the 3σ principle, outliers in displacement vectors are filtered. If the number of valid displacement vectors after filtering is less than 5, the filtering conditions are relaxed to the 5σ range. If this is still insufficient, the displacement vectors are sorted from smallest to largest, and the top 10 displacement vectors are selected. 6.3 Relative displacement normalization: Calculate the diagonal length of the ROI based on the parameters of the ROI (`np.sqrt(w_roi²+h_roi²)+1e-6`, add 1e-6 to avoid division by zero error), divide the mean of the effective displacement vector by the diagonal length and multiply by 100 to obtain the gill relative displacement expressed as a percentage. 6.4 Weak Signal Enhancement: When there are ≥10 stored historical relative displacement data, the average of the most recent 20 historical data is calculated as the signal baseline; if the fluctuation amplitude of the current relative displacement relative to the baseline is <0.02%, a sinusoidal target signal (`2.0×np.sin(2×np.pi×0.5×current_time)`) is generated based on the target respiratory rate to correct the current displacement; if the fluctuation amplitude is ≥0.02%, the fluctuation amplitude is amplified by 20 times and superimposed on the baseline to obtain the enhanced displacement signal, which is the enhanced displacement. 6.5 Baseline Drift Correction: When there are ≥20 historical data points of respiratory rate relative displacement, perform linear fitting (`np.polyfit`) on the 20 most recent historical data points in the video frame sequence. If the absolute value of the fitting slope is >0.1 (determined as baseline drift), recalibrate the baseline to the mean of the 10 most recent data points, which is the historical mean of displacement. Adjust the enhanced displacement signal to "new baseline + (enhanced displacement - historical mean of displacement)". S70, Respiratory Rate Calculation and Smoothing Optimization 7.1 Sliding window data management: The system adopts a first-in-first-out (FIFO) sliding window mechanism to store the 180 most recent enhanced displacement signals and their corresponding timestamps in the video frame sequence. When the data volume exceeds the window length, the oldest data is deleted. 7.2 FFT Frequency Analysis: When the number of data points in the window is ≥30, the displacement signal is processed using `scipy.signal.detrend` to eliminate linear trends, and then an FFT transformation (`np.fft.fft`) is performed. The frequency spectrum is calculated using `np.fft.fftfreq` (only positive frequency components are retained). The frequency with the highest power within the target frequency ±0.2Hz range is selected as the dominant frequency. The dominant frequency is multiplied by 60 to convert it into respiratory rates per minute (BPM). As a preferred method, positive frequency components are selected using `positive_mask=fft_freqs>0` to eliminate the interference of negative frequency components on the selection of the dominant frequency, ensuring the accuracy of frequency calculation.

[0020] 7.3 Peak Detection Backup Algorithm: If the BPM calculated by FFT is not in the range of 20~80 (determined to be invalid), enable the backup algorithm: perform second-order Butterworth bandpass filtering (passband 0.3~1.5Hz) and Savitzky-Golay smoothing on the displacement signal (window length dynamically adjusted: set to 3 when signal length ≤12, and set to "signal length / / 4×2-1" when >12), use `scipy.signal.find_peaks` to identify respiratory peaks, and calculate BPM = 60 / average peak interval based on the interval between adjacent peaks; 7.4 Frequency Smoothing Optimization: ① Jump Limitation: If the historical respiratory rate log is not empty, the current BPM must be limited to the range of "previous frame BPM ± 3". If it exceeds the range, it will be truncated to the boundary value; ② Moving Average: Perform an arithmetic average on the 5 most recent valid BPMs. If the number of valid BPMs is less than 5, the target respiratory rate (30 BPM) or the historical average will be used to fill the gap. S80. Results Output and Visualization: 8.1 Video Frame Annotation: Overlay visualization elements onto the original video frames: ① Target mask (red semi-transparent); ② ROI rectangle (green solid line); ③ Optical flow feature points (yellow solid circles); 8.2 Dynamic Curve Plotting: Dynamic curves are plotted every 5 frames using Matplotlib, with the blue solid line representing the enhanced relative displacement signal.

[0021] 8.3 Final Result Statistics: After video processing, effective frequency data within the range of 20~80 BPM are selected, and the following statistical results are output: ① Number of effective data (total number of effective frequency data); ② Average respiratory rate (retain one decimal place); ③ Frequency standard deviation (retain one decimal place).

[0022] As a preferred implementation, `torch.cuda.empty_cache()` is executed every 3 frames to release GPU memory, reducing memory usage by 40% and supporting continuous monitoring for more than 60 minutes.

[0023] Example 2 The technical solution of Example 1 is applied to "monitoring healthy spotted sea bass with a body length of 15-20cm" as an example. The specific implementation steps are as follows: Step 1: Video Acquisition and Preprocessing (S10-S20 in Example 1) Collection environment: water temperature 25℃, dissolved oxygen 6mg / L, ensuring that the spotted bass swims naturally with their gills unobstructed; Video preprocessing: Each frame is normalized using `cv2.resize(frame, (640, 480))`, and the frame rate is calibrated every 30 frames—the initial nominal 30fps, the actual calculated `real_elapsed=2.01 seconds` (20 frames), `calculated_elapsed=2.00 seconds`, the difference of 0.01 seconds < 1 second, no adjustment is needed, and the frame interval is 0.0333 seconds.

[0024] Step 2: Gill segmentation and ROI extraction (using S30~S40 in Example 1) YOLO inference: Load pre-trained weights (`best.pt`), and obtain the gill mask after inference, with an effective pixel area of ​​1200 pixels; ROI extraction: The minimum bounding rectangle is obtained by mask contour detection (x=150, y=200, w=180, h=120). After adding 20% ​​padding, the ROI parameters are (132, 176, 216, 144), ensuring complete coverage of the gills.

[0025] Step 3: Optical flow tracing and displacement calculation (applying S50~S60 in Example 1) Feature point initialization: After performing grayscale conversion on the ROI extracted in step 2, 85 feature points are obtained through Canny edge detection, which meets the requirement of 30~150. Optical flow tracking: After tracking with the Lucas-Cannard algorithm, 72 valid points are selected, and 8 to 80 points are added, with the minimum distance between the added points and the existing points being ≥1 pixel; Displacement calculation: The mean displacement vector is 0.78 pixels, the diagonal length of the ROI is 252.7 pixels, and the relative displacement is 0.78 / 252.7×100≈0.31%, which is 0.46% after signal enhancement; the baseline fitting slope is 0.08<0.1, and no correction is required.

[0026] Step 4: Frequency calculation and result output (using S70~S80 in Example 1) FFT frequency analysis: 50 data points within a sliding window, detrending followed by FFT, dominant frequency 0.49Hz, converted to 29.4 BPM; Frequency smoothing: The most recent 5 effective frequencies are 28.9, 29.2, 29.4, 29.5, and 29.3 BPM, and the moving average is 29.3 BPM; Results: 8920 valid data points were collected during the 60-minute monitoring period, with an average respiratory rate of 29.3 ± 1.8 BPM and a valid data rate of 96%.

[0027] 3. Experimental verification and effect comparison Figure 2 The frame with a blue semi-transparent mask is overlaid to the mask recognition result. The mask accurately covers the gill area, completely eliminating interference from fins and background impurities, and the segmentation accuracy is ≥90%.

[0028] In Seg Mask Mode, the effective optical flow feature points represented by the solid yellow circles are evenly distributed within the gill mask, with no points drifting to the fins or background area. Figure 3 In the A section of YOLO Box Mode, the optical flow points are scattered within the box. Figure 3(B in the original text). This comparison verifies that the optical flow feature point tracking mechanism based on segmentation masks can more accurately constrain optical flow points to the target gill region compared to the bounding box method, demonstrating the effectiveness of the segmentation mechanism. Figure 3 ).

[0029] Figure 4 For segmentation mask mode ( Figure 4 A in the bounding box pattern Figure 4 Comparing the real-time monitoring results of fish respiratory movements under (B) conditions, the respiratory movement curve based on the segmentation mask exhibits regular and significantly undulating periodic fluctuations. The waveforms and intervals of adjacent respiratory cycles are relatively consistent, clearly reflecting the fish's respiratory rhythm. In contrast, the respiratory movement curve based on the bounding box shows small fluctuations and irregular patterns, lacking obvious periodicity, and respiratory movement information is easily obscured. The comparison shows that the segmentation mask mode can more accurately and clearly reflect the true rhythm and amplitude of fish respiration, verifying that using YOLO to segment the gill mask to limit the optical flow tracking range can significantly improve the accuracy of respiratory monitoring.

[0030] Experimental results show that the present invention is superior to the prior art in terms of monitoring accuracy, stability and anti-interference ability (see Table 2), makes up for the technical deficiencies of CN120599516A, and has practical application value; Table 2 Comparison of the technical effects of the present invention and the prior art .

[0031] This invention, through the deep fusion of YOLO instance segmentation and optical flow tracking, addresses the shortcomings of existing technologies (such as CN120599516A) such as "coarse positioning, tracking drift, weak signal, and poor engineering feasibility," and proposes a systematic improvement scheme. Experimental verification shows that this invention can maintain a breathing frequency calculation error of ≤±2 BPM, an effective data rate of ≥95%, and a continuous monitoring capability of ≥60 minutes even in complex underwater environments, significantly outperforming existing technologies. Its technical solution not only possesses high theoretical innovation but also, through engineering designs such as frame rate calibration and memory optimization, is adaptable to common hardware environments, making it widely applicable in fields such as aquaculture and aquatic biological research, demonstrating good practical value and industrialization prospects.

Claims

1. A method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking, characterized in that, The method includes video data input and initialization, video preprocessing and dynamic frame rate calibration, YOLO model segmentation of gills and target selection, extraction of gill region of interest, optical flow feature point tracking and dynamic maintenance, calculation of gill motion displacement and signal optimization, calculation of respiratory rate and smoothing optimization, and result output and visualization. The optical flow feature point tracking and dynamic maintenance described above involves converting the current video frame into a grayscale image, cropping a grayscale sub-image within the region of interest of the gill region, generating a mask sub-image of the region of interest of the gill region based on the target mask, performing Canny edge detection on the grayscale sub-image, preserving the edges within the mask, and obtaining an edge-enhanced image. A three-level dynamic feature point maintenance mechanism of "initialization-filtering-supplementation" is adopted for edge enhancement images to ensure that the number of effective tracking points remains stable at 30~150; The initialization process involves using Shi-Tomasi corner detection to extract feature points from the edge enhancement image. If fewer than 30 feature points are detected, random sampling is performed within the mask sub-image and Gaussian noise is added. The noise-processed supplementary points are then merged with the remaining valid feature points to ensure that the total number of feature points remains between 30 and 150. The filtering process involves tracking the position of feature points between adjacent frames. Input the grayscale image of the previous frame, the grayscale image of the current frame, and the feature points of the previous frame. Output the feature points of the current frame and the tracking status. 1 indicates successful tracking. Valid points that are "successfully tracked + within the image range + within the mask" are retained. The dual spatial constraints ensure the validity of the points. When the number of valid points is 30-80, the number of points added is ≤20. When the number of valid points is <30, the points are re-initialized, and the minimum distance between the added points and the existing points is ≥1 pixel.

2. The method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking according to claim 1, characterized in that, The video data input and initialization are as follows: The YOLO instance segmentation model is used to output the gill pixel-level mask with a segmentation accuracy of ≥90%; during inference, the mask with a confidence of ≥0.5 is selected and the mask with the largest area is selected as the target mask. The core parameters are initialized as follows: sliding data window size of 180 frames, optical flow feature point number threshold of 30~150, and abnormal displacement judgment range of <0.02% or >20%.

3. The method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking according to claim 1, characterized in that, The video preprocessing and frame rate dynamic calibration described above involve performing size normalization on each frame of video to control the timestamp error to ≤0.1 seconds.

4. The method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking according to claim 1, characterized in that, The YOLO model segmentation and target selection involves performing YOLO model inference on the normalized video frames and selecting the mask with the largest effective pixel area as the target mask.

5. The method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking according to claim 1, characterized in that, Region of interest extraction for gills: Perform contour detection on the target mask to obtain all the outer contours of the mask; sort the contours by area, select the contour with the largest area as the core contour of the gills, calculate the minimum bounding rectangle of the core contour, and obtain the rectangle parameters.

6. The method for detecting fish respiratory frequency based on YOLO segmentation and optical flow tracking according to claim 1, characterized in that, The calculation of gill movement displacement and signal optimization: Displacement vector calculation: For successfully tracked feature points in adjacent frames, calculate the Euclidean distance of each feature point to obtain a set of displacement vectors; Abnormal displacement filtering: Filter outliers in displacement vectors. If the number of valid displacement vectors after filtering is less than 5, relax the filtering conditions to a range of 5σ, where σ is the standard deviation. If the number of valid displacement vectors is still insufficient, sort the displacement values ​​from smallest to largest and take the top 10 displacement values. Relative displacement normalization: Calculate the diagonal length of the region of interest in the gills, divide the mean of the effective displacement vector by the diagonal length and multiply by 100 to obtain the relative displacement of the gills as a percentage. Weak signal enhancement: When there are ≥10 stored relative displacement historical data, the average of the most recent 20 historical data is calculated as the signal baseline; if the fluctuation amplitude of the current relative displacement from the baseline is <0.02%, a sinusoidal target signal is generated based on the target respiratory rate to correct the current displacement. If the fluctuation amplitude is ≥0.02%, the fluctuation amplitude is magnified by 20 times and superimposed on the baseline to obtain the enhanced displacement signal, i.e., enhanced displacement; Baseline drift correction: When there are ≥20 historical relative displacement data, linear fitting is performed on the most recent 20 historical data. If the absolute value of the fitting slope is >0.1, it is determined to be baseline drift. The baseline is recalibrated to the mean of the most recent 10 historical data, which is the historical displacement mean. The enhanced displacement signal is adjusted to "new baseline + (enhanced displacement - historical displacement mean)". The respiratory rate calculation and smoothing optimization are as follows: Sliding window data management: It adopts a first-in-first-out sliding window mechanism to store the 180 most recent enhanced displacement signals and their corresponding timestamps. When the data volume exceeds the window length, the oldest data is deleted. FFT frequency analysis: When the number of data points in the window is ≥30, the linear trend of the displacement signal is eliminated, and then FFT transformation is performed to calculate the frequency spectrum. Only the positive frequency components are retained. The frequency with the highest power within the target frequency ±0.2Hz range is selected as the dominant frequency. The dominant frequency is multiplied by 60 to convert it into respiratory rate per minute, abbreviated as BPM. Peak detection backup algorithm: If the BPM calculated by FFT is not in the range of 20~80, it is determined to be invalid and the backup algorithm is enabled: perform second-order Butterworth bandpass filtering and Savitzky-Golay smoothing on the displacement signal, identify the respiratory peak, and calculate BPM = 60 / average peak interval based on the interval between adjacent peaks. Frequency smoothing optimization: ① Jump limit: If the historical breathing frequency log is not empty, the current BPM is limited to the range of "previous frame BPM ± 3". If it exceeds the range, it is truncated to the boundary value; ② Moving average: The arithmetic average of the 5 most recent valid BPMs in the video frame sequence is performed. If the number of valid BPMs is less than 5, the target breathing frequency of 30 BPM or the historical average is used to fill the gap.

Images