Multi-target fish identification method and system based on deep learning

Abstract

The invention discloses a multi-target fish identification method and system based on deep learning. The method comprises the following steps: firstly, detecting fish targets in a video frame through a YOLO network and constructing a candidate set; then extracting bounding boxes, color features and area information of each target, and constructing a fusion feature vector; furthermore, in the inter-frame matching stage, dynamic weighted association is carried out by combining the position distance and the color similarity, and robust identity matching and ID management are realized; meanwhile, calculating and smoothing a motion track based on a continuous frame center point sequence, and optimizing speed and direction angle data; and finally, outputting a visualization result with the target identifier and the motion trail, and generating global and individual statistical reports. By fusing color features, position information and kinematics parameters, high-precision detection, stable tracking and multi-dimensional behavior statistics of ornamental fishes such as fancy carp in a complex underwater scene are realized, and the problems of frequent ID switching, high missing detection and false alarm rate, poor track quality and the like in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and multi-target tracking technology, and in particular to a method and system for multi-target fish identification based on deep learning. Background Technology

[0002] Currently, mainstream technologies for the automatic identification and tracking of ornamental fish (such as koi) generally adopt a two-stage architecture of "general object detection + multi-object tracking". Specifically, in the detection stage, a general object detection network such as YOLOv8 is typically used, and fine-tuning is performed on video data collected in a fishpond scene to obtain the position detection box of the fish. In the tracking stage, multi-object tracking frameworks such as Deep SORT and ByteTrack are introduced, and target matching and identity association between adjacent frames are achieved by calculating the intersection-union ratio between detection boxes or extracting Re-ID appearance features. This process can be summarized as follows: the camera acquires the video stream, performs object detection frame by frame, extracts the coordinates of the center point of the detection box, performs cross-frame association based on Euclidean distance or Re-ID features, and outputs the target motion trajectory with ID identification.

[0003] However, the aforementioned general technical approach has several key shortcomings in the complex environment of actual fishponds, which seriously affect the accuracy and robustness of identification and tracking: (1) Insufficient utilization of color and morphological information leads to frequent identity switching. Due to the high similarity of textures among individual koi, and the presence of light refraction, reflection, and water disturbance in the underwater environment, the apparent features of the same fish in different frames differ significantly, even exceeding the differences between different individuals, resulting in a sharp decline in the performance of Re-ID-based association methods. When relying solely on the intersection-union ratio of detection boxes for matching, identity switching problems are very likely to occur when fish swim densely, cross each other, or briefly overlap.

[0004] (2) The special morphological characteristics of fish bodies are not fully modeled, resulting in frequent false alarms and missed detections. General detectors are usually designed with rigid objects such as pedestrians or vehicles as a priori, and their preset anchor point frame ratio is difficult to adapt to the slender body shape of koi (the length-to-width ratio is usually between 1.5 and 4.0), resulting in a false detection rate of more than 12% for small targets or individuals with extreme length-to-width ratios. At the same time, the post-detection processing stage only uses non-maximum suppression and does not design a geometric filtering mechanism for the "slender and deformable" morphological characteristics of fish bodies, which cannot effectively suppress false alarm interference caused by water surface reflection, suspended matter, etc.

[0005] (3) The mechanism for handling target disappearance and reappearance is too simple, which leads to the problem of "the same fish being assigned multiple IDs". Existing trackers usually set a fixed threshold of "maximum number of disappearance frames" (e.g., 30 frames). However, in actual fish pond environments, fish may disappear for as long as 50 to 60 frames due to reasons such as diving into depth or being obscured by aquatic plants. At this time, the system will regard it as a new target and assign a brand new ID, which seriously interferes with the accuracy of long-term behavior statistics.

[0006] (4) Lack of modeling ability for dynamic color changes leads to failure of cross-time association. The body color of koi carp will drift significantly with changes in water depth and light angle, but existing methods do not take "consistency of color distribution time sequence" as a key matching criterion, and cannot effectively associate its historical trajectory when the target is reproduced.

[0007] (5) The lack of a trajectory optimization mechanism for fish movement characteristics results in poor trajectory quality. Due to the absence of kinematic constraints such as "limited swimming speed and gentle turning angle of fish", it is difficult to eliminate cross-frame position jumps caused by noise in the detector; at the same time, the lack of a trajectory smoothing mechanism makes the final output trajectory jitter obvious, which is not conducive to the subsequent quantitative analysis of swimming behavior and habits.

[0008] Ultimately, the aforementioned shortcomings stem from the fact that existing general-purpose multi-target tracking frameworks are mainly designed for rigid targets and fail to adequately adapt to the high instability of target appearance and complex occlusion characteristics under underwater imaging conditions. In the model training and post-processing stages, they fail to effectively integrate the strong prior information unique to fish—color distribution, morphological constraints, and motion models—to construct a unified correlation metric. Furthermore, tracking parameters (such as the maximum number of vanishing frames and matching thresholds) are usually set statically based on experience and cannot be adaptively adjusted according to scene factors such as fish density and lighting changes.

[0009] Therefore, there is an urgent need to propose a fish identification and tracking method that can deeply integrate color, shape and motion information and has an adaptive life cycle management mechanism, so as to significantly reduce the frequency of identity switching, improve trajectory continuity and long-term statistical accuracy, thereby overcoming the key bottlenecks of existing technologies in underwater multi-target fish identification. Summary of the Invention

[0010] The purpose of this invention is to provide a multi-target fish identification method and system based on deep learning. By fusing color features, location information and kinematic parameters, it can achieve high-precision detection, stable tracking and multi-dimensional behavior statistics of ornamental fish such as koi in complex underwater scenes, and overcome the problems of frequent ID switching, high false alarm rate and poor trajectory quality in the prior art.

[0011] To achieve the above objectives, the present invention provides the following solution: A deep learning-based method for multi-target fish identification includes the following steps: S1, Video Frame Acquisition and Preprocessing: Acquire BGR format video frame images of underwater fish videos while maintaining their original resolution; S2, Deep Learning Object Detection and Selection: A pre-trained YOLO series deep learning network is used to detect fish targets in the pre-processed video frame images, outputting the bounding box, confidence score and category label of the target, and a candidate fish target detection set is obtained based on preset conditions. S3, Multi-feature extraction and vector construction: For each fish target, extract its bounding box information and calculate the center point coordinates; crop its region of interest (ROI), convert it to the HSV color space and generate a color feature set; integrate the bounding box, color features and target area to construct the detection feature vector of the fish target; S4, Inter-frame target matching and ID management: Based on the detection feature vector of fish targets, calculate the positional distance and color similarity between historically tracked targets and the current fish target, use color similarity to dynamically weight the positional distance to generate a weighted distance matrix; perform optimal assignment matching based on this matrix, assign new IDs to unmatched candidate targets, count the disappearance of unmatched historical targets, and cancel their IDs when the count exceeds the limit; S5, Motion Trajectory Optimization and Recording: Calculate the velocity and motion direction angle of the fish target based on the center point coordinates in consecutive frames, and use a moving average algorithm for smoothing optimization; cache the center point sequence for each ID to form and update the motion trajectory in real time; S6, Visualization Output and Statistical Report Generation: Overlays the bounding box, ID, motion trajectory, and direction marker of the fish target onto the video frame; when the program terminates, outputs a global statistical report containing the total number of frames, the number of detections, and the number of tracks, as well as individual statistical reports on the occurrence frame number, duration, and motion parameters of each ID.

[0012] Furthermore, in S2, the preset condition is: Confidence level > 0.2, target aspect ratio ∈ [0.8, 5.0], target pixel area > 500 pixels 2 The candidate fish target detection set is represented as D={d1,d2,…,d…} N}, where N is the number of candidate detection targets, d i Let d be the i-th candidate detection target, and d i Includes the bounding box, confidence level, and category label of the target.

[0013] Furthermore, S3, multi-feature extraction and vector construction, specifically includes: S301, ROI Extraction and Color Feature Construction: Based on the bounding box output in step S2, the region of interest (ROI) of the fish target is cropped from the original image; the ROI is converted from the BGR color space to the HSV color space, and a binary mask corresponding to each color is generated based on the predefined HSV range of common fish colors; the pixel proportion of each color mask in the ROI is counted, and colors whose pixel proportion meets the preset threshold are selected to construct a color feature set for each fish target; S302, Dominant Color Extraction and ID Binding: The pixel data of the ROI is reshaped into a two-dimensional matrix, and the K-means clustering algorithm is used to cluster the pixels. The cluster center with the most pixels in the clustering results is extracted as the dominant color of the fish target. This dominant color is used for the visual fixed color binding of the target ID in the future. S303, Detection feature vector construction: For each fish target, integrate the bounding box from step S2, the color feature set from step S301, and the target area calculated based on the bounding box to generate the detection feature vector of the fish target; S304, Target center point calculation: Based on the coordinates of the upper left and lower right corners of the bounding box, the center point coordinates of each fish target are calculated and used for target association in subsequent frames.

[0014] Furthermore, in S301, the preset threshold is 5%, that is, when the pixel proportion P of a certain color is 5%. i,k When the percentage is greater than 5%, the color k is included in the color feature set F of the i-th candidate detection target. i,colors And P i,k =count k / total pixels ×100%, where count k Total represents the number of pixels within the color k mask. pixels This represents the total number of pixels in the ROI. In S304, the formula for calculating the coordinates of the center point is: ; Among them, c x c is the x-coordinate of the center point. y x1 and y1 are the ordinates of the center point, respectively; x2 and y2 are the ordinates of the upper left corner of the bounding box, respectively; and x2 and y2 are the ordinates of the lower right corner of the bounding box, respectively.

[0015] Furthermore, S4, inter-frame target matching and ID management, specifically includes: S401, Weighted distance matrix construction: Obtain the historical tracking target set and the candidate fish target detection set from step S3, and calculate the positional distance and color similarity between the two types of targets respectively; dynamically weight the positional distance based on color similarity to obtain the matching weighted distance between the two types of targets; set a maximum distance threshold, and set the weighted distance exceeding the threshold to infinity to complete the construction of the weighted distance matrix; S402, Inter-frame target matching and ID management: The weighted distance matrix constructed in step S401 is numerically sorted by row, and the optimal assignment strategy is used to complete the one-to-one matching of historical tracking targets and candidate detection targets; the disappearance count of unmatched historical tracking targets is accumulated, and new tracking IDs are assigned to unmatched candidate detection targets and their tracking parameters are initialized; when the disappearance count of a historical tracking target exceeds the preset maximum number of disappearance frames, the tracking ID of the target is cancelled.

[0016] Further, in step S401, the position distance is calculated as follows: the distance between the center points of the historically tracked target and the candidate detected target is calculated using the Euclidean distance formula, which is: ; Among them, centroid old,i Let be the coordinates of the center point of the i-th historical tracked target, centroid new,j Let c be the coordinates of the center point of the j-th candidate target. x,old,i c y,old,i These are the x and y coordinates of the historical tracking target center point, respectively, c x,new,j c y,new,j These are the x and y coordinates of the center point of the candidate detection target, respectively; Color similarity calculation: The intersection-union ratio (Jaccard coefficient) is used to calculate the color feature similarity between two types of targets. The formula is as follows: ; Among them, F old,i,colors Let F be the set of color features of the i-th historical tracked target. new,j,colors Let be the set of color features of the j-th candidate target, where |·| represents the number of elements in the set; Weighted distance calculation: The positional distance is dynamically weighted based on color similarity to obtain the matching weighted distance. The formula is as follows: ; Among them, D match (i,j) represents the weighted distance between the i-th historical tracked target and the j-th candidate detected target; the smaller the value, the higher the matching degree between the two types of targets; D pos (i,j) represents the Euclidean distance between the center points of the i-th historical tracked target and the j-th candidate detected target; S col(i,j) represents the color feature similarity between the i-th historical tracked target and the j-th candidate detected target; Threshold truncation: Set the maximum distance threshold (max) distan If D match (i,j)>max distance Then D match Assigning the value of (i,j) to infinity prevents the target from being matched.

[0017] Furthermore, S5, motion trajectory optimization and recording, specifically includes: S501, Motion Feature Calculation and Optimization: Calculate the displacement of the target based on the center point coordinates of the target with the same tracking ID in two consecutive frames; calculate the instantaneous velocity of the target by combining the video frame rate; calculate the motion direction angle of the target by using the arctangent function; store the instantaneous velocity and motion direction angle into double-ended queues of preset length respectively; process the data in the queues using a moving average algorithm to obtain the optimized target motion features. S502, Trajectory Recording and Update: A fixed-length cache is used to store the center point coordinate sequence of each tracking ID target, forming and updating the target motion trajectory; when the length of the coordinate sequence in the cache exceeds the preset cache length, the earliest stored center point coordinates are deleted to keep the cache length constant.

[0018] Furthermore, in S501, the displacement is calculated as follows: based on the center point coordinates of two consecutive frames, the target's displacement in the horizontal and vertical directions is calculated using the following formula: ; in, , The x and y coordinates of the target center point in the current frame. , These are the x and y coordinates of the target's center point in the previous frame; Instantaneous velocity calculation: The instantaneous velocity of the target is calculated by combining the video frame rate. The formula is as follows: ; in, The Euclidean distance represents the displacement. fps =30 frames per second is the video frame rate. v The unit is pixels per second; Calculation of motion direction angle: The motion direction angle of the target is calculated using the arctangent function. The formula is as follows: ; in, θ The direction angle of motion is expressed in degrees, and its value ranges from -180° to 180°. Moving average optimization: Set the maximum length of the double-ended queue to 20, and measure the instantaneous speed. vwith the direction angle of motion θ Each data point is stored in its corresponding deque; when the queue length exceeds 20, the oldest data point is deleted; the average value of all data in the queue is calculated as the optimized target motion feature, using the following formula: ; ; in, speeds A double-ended queue for storing instantaneous speeds, directions For storing the direction angle of motion, a double-ended queue, len (·) represents the length of the queue, v avg For the optimized average speed, θ avg This represents the optimized average motion direction angle.

[0019] Furthermore, S6, the visualization output and statistical report generation, specifically includes: S601, Visualization Output: Draws the bounding box, center point, motion trajectory, and motion direction arrow of each tracked target on the original image (when the optimized speed exceeds the preset speed threshold); adds information tags containing target ID, optimized speed, and current frame number, as well as global statistics including total number of frames, real-time detection count, current tracking count, and processing frame rate (FPS); supports writing the processed image to a video file or triggering a snapshot for saving; S602, Statistical Report Generation: When the identification program terminates, output global statistical data and individual statistical data for each tracking ID; the global statistical data includes the total number of processed frames, the cumulative number of detected targets, the total number of tracking IDs, the maximum number of simultaneous tracking, and the average processing frame rate; the individual statistical data includes the first occurrence frame number, the last occurrence frame number, the total number of occurrence frames, the duration, the average speed, the maximum speed, and the maximum target area for each ID.

[0020] This invention also provides a deep learning-based multi-target fish identification system, applied to the above-described deep learning-based multi-target fish identification method, comprising: The video frame acquisition and preprocessing module is used to acquire BGR format video frame images of underwater fish videos and maintain their original resolution; The deep learning target detection and screening module is used to perform fish target detection on pre-processed video frame images using a pre-trained YOLO series deep learning network, output the target's bounding box, confidence score and category label, and filter the candidate fish target detection set based on preset conditions. The multi-feature extraction and vector construction module is used to extract the bounding box information and calculate the center point coordinates for each fish target; crop its region of interest (ROI), convert it to the HSV color space and generate a color feature set; and integrate the bounding box, color features and target area to construct the detection feature vector of the fish target. The inter-frame target matching and ID management module is used to calculate the positional distance and color similarity between historically tracked targets and the current fish target based on the detection feature vector of the fish target. The positional distance is dynamically weighted using color similarity to generate a weighted distance matrix. Based on this matrix, optimal assignment matching is performed, new IDs are assigned to unmatched candidate targets, and the disappearance count of unmatched historical targets is performed. When the count exceeds the limit, the ID is cancelled. The motion trajectory optimization and recording module is used to calculate the velocity and motion direction angle of the fish target based on the center point coordinates in consecutive frames, and to perform smoothing optimization using a moving average algorithm; it caches the center point sequence for each ID to form and update the motion trajectory in real time. The visualization output and statistical report generation module is used to overlay the bounding box, ID, motion trajectory and direction identifier of fish targets on video frames; when the program terminates, it outputs a global statistical report containing the total number of frames, the number of detections and the number of tracking, as well as individual statistical reports on the occurrence frame number, duration and motion parameters of each ID.

[0021] As can be seen from the above technical solutions, compared with the prior art, the deep learning-based multi-target fish identification method and system provided by the present invention have the following beneficial effects: First, the accuracy of underwater fish detection has been significantly improved, with a substantial reduction in false alarm and missed detection rates. The detection phase utilizes a pre-trained YOLO deep learning network as a foundation, incorporating preset conditions such as confidence level, aspect ratio, and area to perform a secondary screening of the detection results. This accurately eliminates non-fish interference targets such as surface reflections and fallen leaves, while also preventing the missed detection of small-sized fish with specific aspect ratios, thus improving the accuracy of candidate targets. Furthermore, by cropping the Region of Interest (ROI) and converting it to the HSV color space, a color feature set is constructed. The unique color attributes of fish are used to enhance target differentiation. Even in underwater lighting conditions and color shift scenarios, the robustness of recognition is still improved compared to solutions that rely on grayscale or general appearance features. Secondly, the stability of multi-target tracking is enhanced, overcoming the challenges of frequent ID switching and loss. An innovative matching logic of "dynamically weighted positional distance based on color similarity" is adopted to generate a weighted distance matrix and complete optimal assignment matching. This avoids ID jumps caused by traditional pure position association during fish intermingling or brief occlusion, and matching failures caused by pure appearance association during color drift, thus reducing the ID switching rate. Simultaneously, through end-to-end management of "new ID allocation - unmatched target disappearance count - exceeding threshold cancellation," the problem of "multiple IDs for the same fish" or "premature ID loss" caused by traditional fixed disappearance frame settings is solved. Even after a target reappears after occlusion, it can still be accurately associated with the original ID, improving long-term tracking accuracy. Third, the practical value of motion trajectory and data statistics is enhanced, supporting applications in multiple scenarios. By calculating speed and direction angle through the center point of continuous frames, and using a moving average algorithm to eliminate trajectory jitter, combined with a fixed-length cache to update the trajectory in real time, high-quality data is provided for fish behavior research (swimming speed analysis, path preference). When the program terminates, it outputs a global statistical report (total number of frames, number of detections, number of tracks) and an individual statistical report (ID, frame number, duration, motion parameters), which can be directly used in scenarios such as aquarium management and ornamental fish breeding without additional processing. Fourth, it boasts strong scene adaptability, balancing real-time performance with cost-effectiveness. It maintains the original resolution of video frames throughout the process, and combined with the anti-interference capabilities of the HSV color space and fish-specific feature extraction logic, it improves adaptability in underwater reflective, murky, and densely populated fish scenarios. Simultaneously, it employs a lightweight YOLO model, fixed-length caching (memory control), and moving average (low complexity) design, enabling real-time 4K underwater video processing (FPS≥25) on ordinary hardware (i5 + mid-range graphics card). This reduces deployment costs compared to solutions requiring dedicated high-performance hardware, balancing practicality and cost-effectiveness. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the multi-target fish identification method based on deep learning of the present invention; Figure 2 This is a flowchart of the core detection and tracking scheme of an embodiment of the present invention; Figure 3 This is a visualization and statistical flowchart of an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The purpose of this invention is to provide a multi-target fish identification method and system based on deep learning, which realizes a three-in-one correlation measurement of "color-shape-movement" and adaptive life cycle management, significantly reducing ID switching rate and improving long-term statistical accuracy.

[0026] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0027] Example 1

[0028] like Figures 1-3 As shown, the present invention provides a multi-target fish identification method based on deep learning, comprising the following steps: S1, Video Frame Acquisition and Preprocessing: Acquire BGR format video frame images of underwater fish videos and maintain their original resolution; do not scale, and retain maximum color information.

[0029] S2, Deep Learning Object Detection and Selection: A pre-trained YOLO series (e.g., YOLOv8) deep learning network is used to detect fish targets in the pre-processed video frame images, outputting the bounding boxes b of the targets. box =(x1,y1,x2,y2), confidence level (conf), and category label (cls), and a candidate fish target detection set is obtained by filtering based on preset conditions; wherein, the preset conditions are: Confidence level (conf) > 0.2, target aspect ratio ∈ [0.8, 5.0], target pixel area > 500 pixels. 2 The candidate fish target detection set is represented as D={d1,d2,…,d…} N}, where N is the number of candidate detection targets, d i Let d be the i-th candidate detection target, and d i Includes the bounding box, confidence level, and category label of the target.

[0030] S3, multi-feature extraction and vector construction, specifically includes: S301, ROI Extraction and Color Feature Construction: Based on the bounding box output in step S2, the region of interest (ROI) of the fish target is cropped from the original image; the ROI is converted from the BGR color space to the HSV color space, and a binary mask corresponding to each color is generated based on the predefined HSV range of common fish colors; the pixel proportion of each color mask in the ROI is counted, and colors whose pixel proportion meets the preset threshold are selected to construct a color feature set for each fish target; In S301, for example, the predefined range of common fish colors HSV includes: Red: H∈[0,10]∪[160,180],S∈[40,255],V∈[40,255]; White: H∈[0,180], S∈[0,50], V∈[200,255]; Black: H∈[0,180], S∈[0,255], V∈[0,50]; Yellow: H∈[20,40], S∈[40,255], V∈[40,255]; The preset threshold is 5%, that is, when the percentage of pixels of a certain color P... i,k When the percentage is greater than 5%, the color k is included in the color feature set F of the i-th candidate detection target. i,colors And P i,k =count k / total pixels ×100%, where count k Total represents the number of pixels within the color k mask. pixels This represents the total number of pixels in the ROI. S302, Dominant Color Extraction and ID Binding: The pixel data of the ROI is reshaped into a two-dimensional matrix, and the K-means clustering algorithm is used to cluster the pixels. The cluster center with the most pixels in the clustering results is extracted as the dominant color of the fish target. This dominant color is used for the visual fixed color binding of the target ID in the future. The parameters of the K-means clustering algorithm are set as follows: number of clusters k=3, clustering termination condition: criteria=(EPS+MAX_ITER,20,1.0); EPS indicates that clustering is terminated when the clustering error is less than 1.0, and MAX_ITER indicates that clustering is terminated when the number of clustering iterations reaches 20. The dominant color and the tracking ID have a one-to-one correspondence and remain bound unchanged throughout the tracking lifecycle of the ID.

[0031] The cluster center with the most pixels is selected as the dominant color, and a fixed BGR value is assigned to this ID during subsequent drawing to avoid color jumps between frames.

[0032] S303, Detection Feature Vector Construction: For each fish target, integrate the bounding box from step S2, the color feature set from step S301, and the target area calculated based on the bounding box to generate the detection feature vector for that fish target; for each target d i Generate feature tuples: V i =(b box , Fi_colors, S i ), where S i =Area=(x2-x1)×(y2-y 1) , as a size feature; S304, Target Center Point Calculation: Based on the coordinates of the top-left and bottom-right corners of the bounding box, the center point coordinates of each fish target are calculated for subsequent inter-frame target association. The formula for calculating the center point coordinates is: ; Among them, c x c is the x-coordinate of the center point. y x1 and y1 are the ordinates of the center point, respectively; x2 and y2 are the ordinates of the upper left corner of the bounding box, respectively; and x2 and y2 are the ordinates of the lower right corner of the bounding box, respectively.

[0033] S4, Inter-frame Target Matching and ID Management, specifically includes: S401, Weighted distance matrix construction: Obtain the historical tracking target set and the candidate fish target detection set from step S3, and calculate the positional distance and color similarity between the two types of targets respectively; dynamically weight the positional distance based on color similarity to obtain the matching weighted distance between the two types of targets; set a maximum distance threshold, and set the weighted distance exceeding the threshold to infinity to complete the construction of the weighted distance matrix; In step S401, the distance calculation is performed by using the Euclidean distance formula to calculate the distance between the center points of historically tracked targets and candidate detected targets. The formula is as follows: ; Among them, centroid old,i Let be the center point coordinates (in pixels) of the i-th historical tracked target, centroid new,j Let c be the coordinates of the center point of the j-th candidate target. x,old,i c y,old,i These are the x and y coordinates of the historical tracking target center point, respectively, c x,new,j c y,new,j These are the x and y coordinates of the center point of the candidate detection target, respectively; Color similarity calculation: The intersection-union ratio (Jaccard coefficient) is used to calculate the color feature similarity between two types of targets (numerical range 0-1), and the formula is as follows: ; Among them, F old,i,colors Let F be the set of color features of the i-th historical tracked target. new,j,colors Let S be the set of color features of the j-th candidate target, where |·| represents the number of elements in the set; where S col (i,j)=1 indicates that the colors are completely identical, and the weight is 1; S col (i,j)=0 indicates complete dissimilarity, weight=2, distance is amplified; Weighted distance calculation: The positional distance is dynamically weighted based on color similarity to obtain the matching weighted distance. The formula is as follows: ; Among them, D match (i,j) represents the weighted distance between the i-th historical tracked target and the j-th candidate detected target; the smaller the value, the higher the matching degree between the two types of targets; D pos (i,j) represents the Euclidean distance between the center points of the i-th historical tracked target and the j-th candidate detected target; S col (i,j) represents the color feature similarity between the i-th historical tracked target and the j-th candidate detected target; Threshold truncation: Set the maximum distance threshold (max) distance =150 pixels, if D match (i,j)>max distance Then D match Assigning the value of (i,j) to infinity prevents the target from being matched.

[0034] S402, Inter-frame target matching and ID management: The weighted distance matrix constructed in step S401 is numerically sorted by row, and the optimal assignment strategy is used to complete the one-to-one matching of historical tracking targets and candidate detection targets; the disappearance count of unmatched historical tracking targets is accumulated, and new tracking IDs are assigned to unmatched candidate detection targets and their tracking parameters are initialized; when the disappearance count of a historical tracking target exceeds the preset maximum number of disappearance frames, the tracking ID of the target is cancelled. In S402, the optimal assignment strategy includes: Sort each row of the weighted distance matrix in ascending order of value, and extract the column index corresponding to the minimum value of each row as the candidate matching object for the historical tracking target; If the candidate matching object is not occupied by other historical tracking targets, then establish the matching relationship between the historical tracking target and the candidate detection target, update the detection feature vector of the historical tracking target, and reset its disappearance count to 0; If a candidate matching object is already occupied, skip the current minimum value, take the column index corresponding to the second smallest value in the row and re-evaluate, until an unoccupied candidate object is found or all values ​​in the row have been traversed; For historically tracked targets that did not match candidate detection targets, their disappearance count is reset to "disappeared". i Increment by 1; For candidate targets that do not match historical tracking targets, assign a new tracking ID, initialize the detection feature vector, motion feature dequeue, and trajectory cache for that ID, and count its disappearance. new Set to 0; Set the maximum number of frames to disappear. sappeared (Default 30 frames) If the disappearance count of a historical tracked target (disappeared) > max disappeared If so, then remove the target's ID and all associated parameters from the tracking list.

[0035] S5, motion trajectory optimization and recording, specifically includes: S501, Motion Feature Calculation and Optimization: Calculate the displacement of the target based on the center point coordinates of the target with the same tracking ID in two consecutive frames; calculate the instantaneous velocity of the target by combining the video frame rate; calculate the motion direction angle of the target by using the arctangent function; store the instantaneous velocity and motion direction angle into double-ended queues of preset length respectively; process the data in the queues using a moving average algorithm to obtain the optimized target motion features. S502, Trajectory Recording and Update: A fixed-length cache is used to store the center point coordinate sequence of each tracking ID target, forming and updating the target motion trajectory; when drawing, it is converted into a NumPy array, and the trajectory line is drawn using cv2.polylines; when the length of the coordinate sequence in the cache exceeds the preset cache length (e.g., 100), the earliest stored center point coordinates are deleted to keep the cache length constant.

[0036] Furthermore, in S501, the displacement is calculated as follows: based on the center point coordinates of two consecutive frames, the target's displacement in the horizontal and vertical directions is calculated using the following formula: ; in, , The x and y coordinates of the target center point in the current frame. , These are the x and y coordinates of the target's center point in the previous frame; Instantaneous velocity calculation: The instantaneous velocity of the target is calculated by combining the video frame rate. The formula is as follows: ; in, The Euclidean distance represents the displacement. fps =30 frames per second is the video frame rate. v The unit is pixels per second; Calculation of motion direction angle: The motion direction angle of the target is calculated using the arctangent function. The formula is as follows: ; in, θ The direction angle of motion is expressed in degrees, and its value ranges from -180° to 180°. Moving average optimization: Set the maximum length of the double-ended queue to 20, and measure the instantaneous speed. v with the direction angle of motion θ Each data point is stored in its corresponding deque; when the queue length exceeds 20, the oldest data point is deleted; the average value of all data in the queue is calculated as the optimized target motion feature, using the following formula: ; ; in, speeds A double-ended queue for storing instantaneous speeds, directions For storing the direction angle of motion, a double-ended queue, len (·) represents the length of the queue, v avg For the optimized average speed, θ avg This represents the optimized average motion direction angle.

[0037] S6, visualization output and statistical report generation, specifically includes: S601, Visualization Output: Draws the bounding box, center point, motion trajectory, and motion direction arrow for each tracked target on the original image (if the optimized speed exceeds the preset speed threshold, if...). v >5 px / s); Add information tags containing target ID, optimized speed, current frame number, and global statistics including total number of frames, real-time detection count, current tracking count, and processing frame rate (FPS); Supports writing the processed image to a video file or triggering a snapshot for saving; S602, Statistical Report Generation: When the identification program terminates, output global statistical data and individual statistical data for each tracking ID; the global statistical data includes the total number of processed frames, the cumulative number of detected targets, the total number of tracking IDs, the maximum number of simultaneous tracking, and the average processing frame rate; the individual statistical data includes the first occurrence frame number, the last occurrence frame number, the total number of occurrence frames, the duration, the average speed, the maximum speed, and the maximum target area for each ID.

[0038] This invention significantly improves the ability of koi carp to maintain their identity in dense, occluded, and fast-moving scenarios by introducing the core technology of "color-position dual-core weighted matching": It utilizes color similarity (Scol) to dynamically weight Euclidean distance, ensuring that the matching distance (Dmatch) has the lowest weight when colors are identical and a higher weight when colors differ significantly, thus achieving zero-switching tracking even when fish are crossing or briefly occluded. Simultaneously, it persistently records a color list and dominant color for each frame, forming an individual color profile, enabling the system to trace color changes and directly use it for growth monitoring and health assessment. This is further enhanced by simultaneous extraction of speed and direction, and trajectory dequation. With its fixed-length cache, the system outputs multi-dimensional behavioral parameters such as continuous trajectory, instantaneous speed, and swimming direction while ensuring real-time performance (average FPS ≥ 27). It achieves low-cost, high-precision individual behavior analysis without additional hardware. Furthermore, the visualization strategy of permanently binding the dominant color to the ID allows observers to intuitively distinguish different koi without having to remember their numbers, significantly reducing the labor intensity of manual verification. The entire solution is based on a general-purpose camera and a lightweight YOLO model, and can run in real-time on a single 4K video stream on an i5+RTX3060 desktop computer without additional power consumption or equipment modification, offering significant advantages such as low deployment cost and environmental friendliness.

[0039] Example 2 This invention also provides a deep learning-based multi-target fish identification system, applied to the above-described deep learning-based multi-target fish identification method, comprising: The video frame acquisition and preprocessing module is used to acquire BGR format video frame images of underwater fish videos and maintain their original resolution; The deep learning target detection and screening module is used to perform fish target detection on pre-processed video frame images using a pre-trained YOLO series deep learning network, output the target's bounding box, confidence score and category label, and filter the candidate fish target detection set based on preset conditions. The multi-feature extraction and vector construction module is used to extract the bounding box information and calculate the center point coordinates for each fish target; crop its region of interest (ROI), convert it to the HSV color space and generate a color feature set; and integrate the bounding box, color features and target area to construct the detection feature vector of the fish target. The inter-frame target matching and ID management module is used to calculate the positional distance and color similarity between historically tracked targets and the current fish target based on the detection feature vector of the fish target. The positional distance is dynamically weighted using color similarity to generate a weighted distance matrix. Based on this matrix, optimal assignment matching is performed, new IDs are assigned to unmatched candidate targets, and the disappearance count of unmatched historical targets is performed. When the count exceeds the limit, the ID is cancelled. The motion trajectory optimization and recording module is used to calculate the velocity and motion direction angle of the fish target based on the center point coordinates in consecutive frames, and to perform smoothing optimization using a moving average algorithm; it caches the center point sequence for each ID to form and update the motion trajectory in real time. The visualization output and statistical report generation module is used to overlay the bounding box, ID, motion trajectory and direction identifier of fish targets on video frames; when the program terminates, it outputs a global statistical report containing the total number of frames, the number of detections and the number of tracking, as well as individual statistical reports on the occurrence frame number, duration and motion parameters of each ID.

[0040] Furthermore, embodiments of the present invention provide a computer-readable storage medium storing executable instructions, which, when executed, cause a processor to perform the deep learning-based multi-target fish identification method described in Embodiment 1.

[0041] Matters not covered in this invention are common knowledge.

[0042] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0043] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A multi-target fish identification method based on deep learning, characterized in that, Includes the following steps: S1, Video Frame Acquisition and Preprocessing: Acquire BGR format video frame images of underwater fish videos while maintaining their original resolution; S2, Deep Learning Object Detection and Selection: A pre-trained YOLO series deep learning network is used to detect fish targets in the pre-processed video frame images, outputting the bounding box, confidence score and category label of the target, and a candidate fish target detection set is obtained based on preset conditions. S3, Multi-feature extraction and vector construction: For each fish target, extract its bounding box information and calculate the center point coordinates; crop its region of interest (ROI), convert it to the HSV color space and generate a color feature set; integrate the bounding box, color features and target area to construct the detection feature vector of the fish target; S4, Inter-frame target matching and ID management: Based on the detection feature vector of fish targets, calculate the positional distance and color similarity between historically tracked targets and the current fish target, use color similarity to dynamically weight the positional distance to generate a weighted distance matrix; perform optimal assignment matching based on this matrix, assign new IDs to unmatched candidate targets, count the disappearance of unmatched historical targets, and cancel their IDs when the count exceeds the limit; S5, Motion Trajectory Optimization and Recording: Calculate the velocity and motion direction angle of the fish target based on the center point coordinates in consecutive frames, and use a moving average algorithm for smoothing optimization; cache the center point sequence for each ID to form and update the motion trajectory in real time; S6, Visualization Output and Statistical Report Generation: Overlays the bounding box, ID, motion trajectory, and direction marker of the fish target onto the video frame; when the program terminates, outputs a global statistical report containing the total number of frames, the number of detections, and the number of tracks, as well as an individual statistical report containing the occurrence frame number, duration, and motion parameters of each ID.

2. The multi-target fish identification method based on deep learning according to claim 1, characterized in that, In S2, the preset condition is: Confidence level > 0.2, target aspect ratio ∈ [0.8, 5.0], target pixel area > 500 pixels 2 The candidate fish target detection set is represented as D={d1,d2,…,d…} N }, where N is the number of candidate detection targets, d i Let d be the i-th candidate detection target, and d i Includes the bounding box, confidence level, and category label of the target.

3. The multi-target fish identification method based on deep learning according to claim 1, characterized in that, S3, multi-feature extraction and vector construction, specifically includes: S301, ROI Extraction and Color Feature Construction: Based on the bounding box output in step S2, the Region of Interest (ROI) of the fish target is cropped from the original image; the ROI is converted from the BGR color space to the HSV color space, and a binary mask corresponding to each color is generated based on the predefined HSV range of common fish colors; the pixel ratio of each color mask in the ROI is counted, and colors whose pixel ratio meets the preset threshold are selected to construct a color feature set for each fish target; S302, Dominant Color Extraction and ID Binding: The pixel data of the ROI is reshaped into a two-dimensional matrix, and the K-means clustering algorithm is used to cluster the pixels. The cluster center with the most pixels in the clustering results is extracted as the dominant color of the fish target. This dominant color is used for the subsequent visualization and fixed color binding of the target ID. S303, Detection feature vector construction: For each fish target, integrate the bounding box from step S2, the color feature set from step S301, and the target area calculated based on the bounding box to generate the detection feature vector of the fish target; S304, Target center point calculation: Based on the coordinates of the upper left and lower right corners of the bounding box, the center point coordinates of each fish target are calculated and used for target association in subsequent frames.

4. The multi-target fish identification method based on deep learning according to claim 3, characterized in that, In step S301, the preset threshold is 5%, that is, when the pixel proportion P of a certain color is 5%. i,k When the percentage is greater than 5%, the color k is included in the color feature set F of the i-th candidate detection target. i,colors And P i,k =count k / total pixels ×100%, where count k Total represents the number of pixels within the color k mask. pixels This represents the total number of pixels in the ROI. In S304, the formula for calculating the coordinates of the center point is: ； Among them, c x c is the x-coordinate of the center point. y x1 and y1 are the ordinates of the center point, respectively; x2 and y2 are the ordinates of the upper left corner of the bounding box, respectively; and x2 and y2 are the ordinates of the lower right corner of the bounding box, respectively.

5. The multi-target fish identification method based on deep learning according to claim 1, characterized in that, S4, inter-frame target matching and ID management, specifically includes: S401, Weighted distance matrix construction: Obtain the historical tracking target set and the candidate fish target detection set from step S3, and calculate the positional distance and color similarity between the two types of targets respectively; dynamically weight the positional distance based on color similarity to obtain the matching weighted distance between the two types of targets; set a maximum distance threshold, and set the weighted distance exceeding the threshold to infinity to complete the construction of the weighted distance matrix; S402, Inter-frame target matching and ID management: The weighted distance matrix constructed in step S401 is numerically sorted by row, and the optimal assignment strategy is used to complete the one-to-one matching of historical tracking targets and candidate detection targets; the disappearance count of unmatched historical tracking targets is accumulated, and new tracking IDs are assigned to unmatched candidate detection targets and their tracking parameters are initialized; when the disappearance count of a historical tracking target exceeds the preset maximum number of disappearance frames, the tracking ID of the target is cancelled.

6. The multi-target fish identification method based on deep learning according to claim 5, characterized in that, In step S401, the distance calculation is performed by using the Euclidean distance formula to calculate the distance between the center points of historically tracked targets and candidate detected targets. The formula is as follows: Centroid old,i Let be the coordinates of the center point of the i-th historical tracked target, centroid new,j Let c be the coordinates of the center point of the j-th candidate target. x,old,i c y,old,i These are the x and y coordinates of the historical tracking target center point, respectively, c x,new,j c y,new,j These are the x and y coordinates of the center point of the candidate detection target, respectively; Color similarity calculation: The intersection-union ratio (IUU) of sets is used to calculate the color feature similarity between two types of targets. The formula is: ； Among them, F old,i,colors Let F be the set of color features of the i-th historical tracked target. new,j,colors Let be the set of color features of the j-th candidate target, where |·| represents the number of elements in the set; Weighted distance calculation: The positional distance is dynamically weighted based on color similarity to obtain the matching weighted distance. The formula is as follows: ； Among them, D match (i,j) represents the weighted distance between the i-th historical tracked target and the j-th candidate detected target; the smaller the value, the higher the matching degree between the two types of targets; D pos (i,j) represents the Euclidean distance between the center points of the i-th historical tracked target and the j-th candidate detected target; S col (i,j) represents the color feature similarity between the i-th historical tracked target and the j-th candidate detected target; Threshold truncation: Set the maximum distance threshold (max) distance If D match (i,j)>max distance Then D match Assigning the value of (i,j) to infinity prevents the target from being matched.

7. The multi-target fish identification method based on deep learning according to claim 1, characterized in that, S5, motion trajectory optimization and recording, specifically includes: S501, Motion Feature Calculation and Optimization: Calculate the displacement of the target based on the center point coordinates of the target with the same tracking ID in two consecutive frames; calculate the instantaneous velocity of the target by combining the video frame rate; calculate the motion direction angle of the target by using the arctangent function; store the instantaneous velocity and motion direction angle into double-ended queues of preset length respectively; process the data in the queues using a moving average algorithm to obtain the optimized target motion features. S502, Trajectory Recording and Update: A fixed-length cache is used to store the center point coordinate sequence of each tracking ID target, forming and updating the target motion trajectory; when the length of the coordinate sequence in the cache exceeds the preset cache length, the earliest stored center point coordinates are deleted to keep the cache length constant.

8. The multi-target fish identification method based on deep learning according to claim 7, characterized in that, In step S501, the displacement is calculated as follows: based on the center point coordinates of two consecutive frames, the target's displacement in the horizontal and vertical directions is calculated using the following formula: ； in, , The x and y coordinates of the target center point in the current frame. , These are the x and y coordinates of the target's center point in the previous frame; Instantaneous velocity calculation: The instantaneous velocity of the target is calculated by combining the video frame rate. The formula is as follows: ； in, The Euclidean distance represents the displacement. fps =30 frames per second is the video frame rate. v The unit is pixels per second; Calculation of motion direction angle: The motion direction angle of the target is calculated using the arctangent function. The formula is as follows: ； in, θ The direction angle of motion is expressed in degrees, and its value ranges from -180° to 180°. Moving average optimization: Set the maximum length of the double-ended queue to 20, and measure the instantaneous speed. v with the direction angle of motion θ Each data point is stored in its corresponding deque; when the queue length exceeds 20, the oldest data point is deleted; the average value of all data in the queue is calculated as the optimized target motion feature, using the following formula: ； ； in, speeds A double-ended queue for storing instantaneous speeds, directions For storing the direction angle of motion, a double-ended queue, len (·) represents the length of the queue, v avg For the optimized average speed, θ avg This represents the optimized average motion direction angle.

9. The multi-target fish identification method based on deep learning according to claim 1, characterized in that, S6, visualization output and statistical report generation, specifically includes: S601, Visualization Output: Draws the bounding box, center point, motion trajectory, and motion direction arrow for each tracked target on the original image; adds information tags containing target ID, optimized speed, and current frame number, as well as global statistics including total number of frames, real-time detection count, current tracking count, and processing frame rate; supports writing the processed image to a video file or triggering a snapshot for saving; S602, Statistical Report Generation: When the identification program terminates, output global statistical data and individual statistical data for each tracking ID; the global statistical data includes the total number of processed frames, the cumulative number of detected targets, the total number of tracking IDs, the maximum number of simultaneous tracking, and the average processing frame rate; the individual statistical data includes the first occurrence frame number, the last occurrence frame number, the total number of occurrence frames, the duration, the average speed, the maximum speed, and the maximum target area for each ID.

10. A deep learning-based multi-target fish identification system, applied to the execution of the deep learning-based multi-target fish identification method according to any one of claims 1-9, characterized in that, include: The video frame acquisition and preprocessing module is used to acquire BGR format video frame images of underwater fish videos and maintain their original resolution; The deep learning target detection and screening module is used to perform fish target detection on pre-processed video frame images using a pre-trained YOLO series deep learning network, output the target's bounding box, confidence score and category label, and filter the candidate fish target detection set based on preset conditions. The multi-feature extraction and vector construction module is used to extract the bounding box information and calculate the center point coordinates for each fish target; crop its region of interest (ROI), convert it to the HSV color space and generate a color feature set; and integrate the bounding box, color features and target area to construct the detection feature vector of the fish target. The inter-frame target matching and ID management module is used to calculate the positional distance and color similarity between historically tracked targets and the current fish target based on the detection feature vector of the fish target. The positional distance is dynamically weighted using color similarity to generate a weighted distance matrix. Based on this matrix, optimal assignment matching is performed, new IDs are assigned to unmatched candidate targets, and the disappearance count of unmatched historical targets is performed. When the count exceeds the limit, the ID is cancelled. The motion trajectory optimization and recording module is used to calculate the velocity and motion direction angle of the fish target based on the center point coordinates in consecutive frames, and to perform smoothing optimization using a moving average algorithm; it caches the center point sequence for each ID to form and update the motion trajectory in real time. The visualization output and statistical report generation module is used to overlay the bounding box, ID, motion trajectory and direction identifier of fish targets on video frames; when the program terminates, it outputs a global statistical report containing the total number of frames, the number of detections and the number of tracking, as well as individual statistical reports on the occurrence frame number, duration and motion parameters of each ID.

Images