An underwater fish matching method

Abstract

The application relates to an underwater fish matching method. The method comprises the following steps: acquiring a to-be-matched picture and at least one matched fish picture; extracting the to-be-matched picture by using a fish frame selection model to obtain one or more extracted fish pictures; generating one or more first appearance embedding vectors according to the one or more extracted fish pictures; generating one or more second appearance embedding vectors according to the one or more matched fish pictures; matching the first appearance embedding vector with the second appearance embedding vector by using a Hungarian algorithm to obtain the second appearance embedding vector corresponding to each first appearance embedding vector; and obtaining the matched fish picture corresponding to each fish in the to-be-matched picture according to the second appearance embedding vector corresponding to each first appearance embedding vector. The method has the characteristics of high matching accuracy.

Description

Technical Field

[0001] This application relates to the field of image matching, and more specifically, to a method for matching underwater fish. Background Technology

[0002] In aquaculture, underwater monitoring, and ecological research, management and research focus on specific individuals rather than fragmented test results. If the same fish is identified as different individuals during tracking, it can lead to an overestimation of the number of individuals and fragmented movement trajectories, thereby affecting the accuracy of growth assessment, health status analysis, and behavioral statistics.

[0003] Existing multi-object matching methods typically extract features based on bounding box regions when constructing target appearance features, assuming that all pixels within the bounding box belong to the target object. However, in practical applications, especially in underwater fish farming environments, the bounding box often contains a large number of background interference areas, such as water reflections, aquatic plants, and tank edges. This background information significantly affects the stability of the target appearance features, leading to identity switching (IDSW) and trajectory breakage problems when the target is occluded, changes pose, or turns. Secondly, existing technologies rely solely on appearance feature vectors for matching and tracking, which is not robust enough for tracking fish with similar appearances.

[0004] In underwater multi-target matching scenarios involving fish, ambiguous states often arise where "multiple fish have similar appearances" after targets occlude each other, interact closely, or experience short-term appearance degradation. Existing methods often rely on geometric quantities such as appearance feature similarity or IoU / distance as the primary basis for association. When the appearance differences among multiple candidate matching pairs are insufficient to form a clear ranking, the association results are easily affected by noise, background texture interference, or detection box jitter. Introducing orientation constraints indiscriminately across all frames can easily lead to errors when orientation estimation is unstable, targets suddenly turn, or detection is intermittent, exacerbating false associations.

[0005] In summary, existing technologies suffer from inaccurate fish tracking.

[0006] Existing technology discloses a real-time fish target detection and tracking method based on computer vision, including the following steps: 1) real-time monitoring of crucian carp as the biological detection object; 2) manually acquiring the initial outline of the target fish and the field of view for tracking the fish; 3) extraction and matching tracking of the target fish outline; 3.1) calculating a new attention region based on the target fish outline of the previous moment; 3.2) searching for the outline of the target fish based on the target fish attention region; 3.2.1) clustering within the target fish attention region and finding the target fish category; 3.2.2) further matching and tracking the fish targets detected by clustering. This method does not consider background interference, nor does it consider the situation where multiple fish targets are mixed, therefore its accuracy is low under these conditions. Summary of the Invention

[0007] This invention addresses the low accuracy of existing underwater fish matching technologies by providing a method for underwater fish matching that boasts high accuracy.

[0008] The primary objective of this invention is to solve the aforementioned technical problems. The technical solution of this invention is as follows: A method for matching underwater fish includes: S1: Obtain the image to be matched, and at least one matched fish image; The image to be matched contains at least one fish, the matched fish images contain one fish, and the total number of matched fish images is greater than or equal to the total number of fish in the image to be matched. S2: Use a fish selection model to extract the images to be matched, and obtain one or more extracted fish images; S3: Generate one or more first appearance embedding vectors based on one or more of the extracted fish images; generate one or more second appearance embedding vectors based on one or more of the matched fish images; S4: Using the Hungarian algorithm, match each first appearance embedding vector with a second appearance embedding vector to obtain the second appearance embedding vector corresponding to each first appearance embedding vector. S5: Based on the second appearance embedding vector corresponding to each first appearance embedding vector, obtain the matched fish image corresponding to each fish in the image to be matched.

[0009] Further, in step S3, based on one or more of the extracted fish images, one or more first appearance embedding vectors are generated, including: S301: Select one fish image from one or more extracted fish images as the first fish image; S302: Use an external instance segmentation model to generate a mask for the first fish image to obtain a fish body mask; S303: Process the first fish image using the ROI-align algorithm to obtain the ROI features corresponding to the first fish image; S304: Weight the ROI features corresponding to the first fish image and the fish body mask to obtain a feature map for background interference suppression; S305: The feature map of the background interference suppression is processed by the tracking head to obtain the first appearance embedding vector; S306: Select another fish image from one or more of the extracted fish images as the new first fish image, and repeat steps S302 to S305 until all of the extracted fish images have been selected.

[0010] Further, in step S304, the ROI features corresponding to the first fish image and the fish body mask are weighted, including: S30401: Obtain preset background retention parameters; S30402: Process the fish mask according to the preset background retention parameters to obtain the fish weight; S30403: Combine the fish body weights with the ROI features to obtain a feature map for background interference suppression.

[0011] Furthermore, using the Hungarian algorithm, matching is performed on one or more of the first appearance embedding vectors and the second appearance embedding vectors, including: S401: Calculate the normalized exponential similarity and cosine similarity between each of the first appearance embedding vectors and each of the second appearance embedding vectors. S402: Combine the normalized exponential similarity and cosine similarity to form the first appearance cost matrix; S403: For the first appearance cost matrix, use the Hungarian algorithm to match and obtain the second appearance embedding vector corresponding to each first appearance embedding vector.

[0012] Furthermore, the formula for calculating the normalized index similarity is as follows:

[0013] Let r represent the r-th first appearance embedding vector, where r, This indicates the first appearance embedding vector index. This represents the second appearance embedding vector. , This indicates the second appearance embedding vector index. This represents the total number of first appearance embedding vectors. This represents the total number of second appearance embedding vectors.

[0014] A method for matching underwater fish in videos includes: S01: Obtain the true fish center coordinates of each fish in the current video frame's image and fish image, and the first appearance embedding vector of each fish; Obtain one or more historical video frame fish images, and the fish center coordinates, fish velocity, and second appearance embedding vector of each fish image. Each historical video frame's fish image contains one fish; the current video frame's image contains at least one fish. S02: Using the Kalman filter algorithm, based on the time difference between the current video frame and the historical video frames, the fish center coordinates and the corresponding fish speed of each historical video frame are calculated to obtain the predicted fish center coordinates of multiple current video frames. S03: Match the predicted fish center coordinates of multiple current video frames with the actual fish center coordinates of multiple current video frames to obtain candidate pairs of each fish in the fish image of the current video frame and the fish image of the historical video frame. S04: Based on the candidate pairs of fish images in the current video frame and fish images in historical video frames, the first appearance embedding vector corresponding to each fish in the current video frame, and the second appearance embedding vector corresponding to each fish image in historical video frames, calculate the best appearance similarity of each candidate pair. S05: Determine whether the appearance similarity of the multiple candidate pairs meets the first preset condition; if the preset condition is met, proceed to step S06; if the preset condition is not met, proceed to step S07. S06: Based on the fish image of the current video frame and fish images of multiple historical video frames, execute the underwater fish matching method to obtain the fish image of the historical video frame corresponding to each fish in the current video frame; S07: Based on the first appearance embedding vector and the center coordinates of the real fish corresponding to each fish in the current video frame, and the second appearance embedding vector and the center coordinates of the fish corresponding to each fish image in the historical video frame, the Hungarian algorithm is used to match each fish in the current video frame and each fish image in the historical video frame to obtain the fish image in the historical video frame corresponding to each fish in the current video frame.

[0015] Further, in step S03, matching the predicted fish center coordinates of multiple current video frames with the actual fish center coordinates of multiple current video frames includes: S0301: Combine the predicted fish center coordinates of each current video frame with the actual fish center coordinates of each current video frame in pairs to form multiple first combinations; S0302: Calculate the Mahalanobis distance between the predicted fish center coordinates and the actual fish center coordinates of the current video frame in the first combination; S0303: Select the first combination whose Mahalanobis distance meets the second preset condition as a candidate pair for each fish in the fish image of the current video frame and the fish image of the historical video frame.

[0016] Further, in step S07, the Hungarian algorithm is used to match each fish image in the current video frame and each fish image in a historical video frame, including: S0701: Calculate the normalized exponential similarity and cosine similarity between each of the first appearance embedding vectors and each of the second appearance embedding vectors, and combine them to form the first appearance cost matrix; S0702: Calculate the direction vector between the real fish center coordinates corresponding to each of the first appearance embedding vectors and the fish center coordinates corresponding to each of the second appearance embedding vectors to obtain the first direction vector; S0703: Obtain the direction vector corresponding to the second appearance embedding vector, and combine it with the first direction vector to form the second appearance cost matrix; S0703: Combine the first appearance cost matrix and the second appearance cost matrix to form a third appearance cost matrix; S0704: For the third appearance cost matrix, use the Hungarian algorithm to match and obtain the second appearance embedding vector corresponding to each first appearance embedding vector; S0705: Based on the second appearance embedding vector corresponding to each of the first appearance embedding vectors, obtain the historical video frame fish image corresponding to each fish in the current video frame.

[0017] Furthermore, in step S0702, the formula for calculating the first direction vector is as follows:

[0018] This represents the coordinates of the real fish center corresponding to the i-th first appearance embedding vector. This represents the coordinates of the fish center corresponding to the j-th second appearance embedding vector. Represents a constant. Indicates to Find the second norm.

[0019] Furthermore, in step S0703, the formula for calculating the second appearance cost matrix is ​​as follows:

[0020] Indicates matrix transpose. Let represent the direction vector corresponding to the j-th second appearance embedding vector, where j represents the index of the second appearance embedding vector. Let represent the first direction vector between the i-th first appearance embedding vector and the j-th second appearance embedding vector, where i represents the index of the first appearance embedding vector. Represents the inverse cosine function. Indicates will It is limited to the range where the minimum value is -1 and the maximum value is 1.

[0021] Compared with the prior art, the beneficial effects of the present invention are: This invention generates appearance embedding vectors from fish images, effectively suppressing the interference of background regions on the target appearance features. It improves the stability of fish appearance features even when fish undergo pose changes, rapid turns, or partial occlusion. Particularly under conditions of low confidence detection thresholds and long-term video sequences, this method significantly reduces the number of redundant target identities generated during the matching process without increasing model training complexity, thereby improving the accuracy of fish matching.

[0022] By using the Kalman filter algorithm to predict the coordinates of fish in a video and judging whether ambiguity occurs based on the prediction results, the accuracy of fish matching can be significantly improved in scenarios with complex backgrounds, appearance disturbances, and detection of fluctuations. Attached Figure Description

[0023] Figure 1 This is a flowchart of an underwater fish matching method provided in Example 1.

[0024] Figure 2 This is a schematic diagram of the process for generating appearance embedding vectors provided in Example 1.

[0025] Figure 3 A schematic diagram of the appearance cost provided for Example 1.

[0026] Figure 4 This is a flowchart illustrating an underwater fish matching method for video, as provided in Example 1.

[0027] Figure 5 This is a schematic diagram of the process of matching video frames using the Hungarian algorithm provided in Example 1. Detailed Implementation

[0028] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions; It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0029] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] Example 1 like Figure 1 As shown, an underwater fish matching method includes: S1: Obtain the image to be matched, and at least one matched fish image; The image to be matched contains at least one fish, the matched fish images contain one fish, and the total number of matched fish images is greater than or equal to the total number of fish in the image to be matched. S2: Use a fish selection model to extract the images to be matched, and obtain one or more extracted fish images; S3: Generate one or more first appearance embedding vectors based on one or more of the extracted fish images; generate one or more second appearance embedding vectors based on one or more of the matched fish images; S4: Using the Hungarian algorithm, match each first appearance embedding vector with a second appearance embedding vector to obtain the second appearance embedding vector corresponding to each first appearance embedding vector. S5: Based on the second appearance embedding vector corresponding to each first appearance embedding vector, obtain the matched fish image corresponding to each fish in the image to be matched.

[0031] Furthermore, such as Figure 2 As shown, in step S3, one or more first appearance embedding vectors are generated based on one or more of the extracted fish images, including: S301: Select one fish image from one or more extracted fish images as the first fish image; S302: Use an external instance segmentation model to generate a mask for the first fish image to obtain a fish body mask; S303: Process the first fish image using the ROI-align algorithm to obtain the ROI features corresponding to the first fish image; S304: Weight the ROI features corresponding to the first fish image and the fish body mask to obtain a feature map for background interference suppression; S305: The feature map of the background interference suppression is processed by the tracking head to obtain the first appearance embedding vector; S306: Select another fish image from one or more of the extracted fish images as the new first fish image, and repeat steps S302 to S305 until all of the extracted fish images have been selected.

[0032] Figure 3 In this context, the ID is used to identify whether the observations of the same fish in different time frames belong to the same real individual, and is usually a unique number. The trajectory refers to the continuous features formed by the swimming of the same fish in different video frames.

[0033] In one specific embodiment, the background preservation coefficient is first defined. , A mask based on the i-th fish, scaled to the size of the ROI feature dimension. Constructing a soft weight graph The formula is as follows:

[0034] u, v are the coordinates of the matrix positions. The value is 0 or 1; when it is 1, A value of 1 indicates that the original features of the region are preserved; a value of 0 indicates that... for This indicates that the background area is not directly cleared, but is retained. Basic information about proportions We take 0.3.

[0035] For each channel c of the feature map of the i-th fish, a soft weight map is used. The weighting is performed using the following formula:

[0036] Finally, the weighted feature map replaces the original feature map, and the final appearance embedding vector is generated.

[0037] It should be noted that during the generation of the embedding vector of the target instance, instance-level segmentation mask information is introduced to constrain the feature response within the detection box, thereby reducing the interference of the background region on the appearance features of the target.

[0038] After fine-tuning the Masa YOLOx detector using a fish dataset, compared to Masa's existing target tracking methods based on detection box features, this method effectively suppresses the interference of background regions on target appearance features by introducing instance segmentation mask constraints. It improves the stability of target appearance features, reduces the number of identity switching, and enhances trajectory continuity when the target undergoes pose changes, rapid turns, or partial occlusion. Particularly under low-confidence detection thresholds and long-term video conditions, this method significantly reduces the number of redundant target identities generated during tracking without increasing model training complexity. In a test video of 101 frames (eight fish), the maximum ID decreased from 26 to 22; in a test video of 690 frames, the maximum ID decreased from 347 to 139.

[0039] Masking constraints are mainly used to suppress background interference and improve the purity of target appearance features. However, when the target undergoes rapid attitude changes or severe occlusion, relying solely on appearance features may still lead to identity switching.

[0040] Further, in step S304, the ROI features corresponding to the first fish image and the fish body mask are weighted, including: S30401: Obtain preset background retention parameters; S30402: Process the fish mask according to the preset background retention parameters to obtain the fish weight; S30403: Combine the fish body weights with the ROI features to obtain a feature map for background interference suppression.

[0041] Furthermore, such as Figure 3 As shown, the Hungarian algorithm is used to match one or more of the first appearance embedding vectors and the second appearance embedding vectors, including: S401: Calculate the normalized exponential similarity and cosine similarity between each of the first appearance embedding vectors and each of the second appearance embedding vectors. S402: Combine the normalized exponential similarity and cosine similarity to form the first appearance cost matrix; S403: For the first appearance cost matrix, use the Hungarian algorithm to match and obtain the second appearance embedding vector corresponding to each first appearance embedding vector.

[0042] Furthermore, the formula for calculating the normalized index similarity is as follows:

[0043] Let r represent the r-th first appearance embedding vector, where r, This indicates the first appearance embedding vector index. This represents the second appearance embedding vector. , This indicates the second appearance embedding vector index. This represents the total number of first appearance embedding vectors. This represents the total number of second appearance embedding vectors.

[0044] s2 is the cosine similarity calculation formula, which calculates the cosine similarity between the trajectory and the appearance embedding vector of the current fish detection.

[0045]

[0046] s represents the final appearance similarity score, calculated as follows:

[0047] In one specific embodiment, the appearance cost matrix is ​​as follows:

[0048] like Figure 4 As shown, an underwater fish matching method for videos includes: S01: Obtain the true fish center coordinates of each fish in the current video frame's image and fish image, and the first appearance embedding vector of each fish; Obtain one or more historical video frame fish images, and the fish center coordinates, fish velocity, and second appearance embedding vector of each fish image. Each historical video frame's fish image contains one fish; the current video frame's image contains at least one fish. S02: Using the Kalman filter algorithm, based on the time difference between the current video frame and the historical video frames, the fish center coordinates and the corresponding fish speed of each historical video frame are calculated to obtain the predicted fish center coordinates of multiple current video frames. S03: Match the predicted fish center coordinates of multiple current video frames with the actual fish center coordinates of multiple current video frames to obtain candidate pairs of each fish in the fish image of the current video frame and the fish image of the historical video frame. S04: Based on the candidate pairs of fish images in the current video frame and fish images in historical video frames, the first appearance embedding vector corresponding to each fish in the current video frame, and the second appearance embedding vector corresponding to each fish image in historical video frames, calculate the best appearance similarity of each candidate pair. S05: Determine whether the appearance similarity of the multiple candidate pairs meets the first preset condition; if the preset condition is met, proceed to step S06; if the preset condition is not met, proceed to step S07. S06: Based on the fish image of the current video frame and fish images of multiple historical video frames, execute the underwater fish matching method to obtain the fish image of the historical video frame corresponding to each fish in the current video frame; S07: Based on the first appearance embedding vector and the center coordinates of the real fish corresponding to each fish in the current video frame, and the second appearance embedding vector and the center coordinates of the fish corresponding to each fish image in the historical video frame, the Hungarian algorithm is used to match each fish in the current video frame and each fish image in the historical video frame to obtain the fish image in the historical video frame corresponding to each fish in the current video frame.

[0049] In one specific embodiment, during the matching phase of fish tracking, the first frame only requires assigning a unique identifier (ID) to each fish and constructing a trajectory (the trajectory is the ID, along with some encapsulation of the attributes detected in this fish). Subsequent frames repeat the previous detection steps, and in the matching phase, they need to be matched with the trajectory. If the same fish is detected in the second frame, the fish ID from the previous frame needs to be inherited. At this time, the set of detection boxes for all fish currently detected, along with the motion state vectors and state covariance matrices corresponding to the trajectories, are sent to the Kalman filter motion constraint module. The Kalman filter uses the motion state vectors and state covariance matrices to output the predicted motion state vectors. The motion state vectors contain the predicted center coordinates, and the Mahalanobis distance between the predicted center coordinates and the detection center of each fish is calculated. A gating matrix is ​​constructed based on whether the Mahalanobis distance between the current detection center and the predicted trajectory center exceeds a threshold. Gating matrix Each value actually indicates whether (trajectory, detection) forms a candidate group (this is the role of the Kalman filter motion model, which changes the original fully connected calculation of cost for trajectory and detection to forming a candidate group, and only calculating cost within the candidate group, and also introduces position information to avoid matching the current detection with a trajectory with too large a position difference). Matching with historical trajectories only requires a global similarity judgment of appearance embedding vectors within the candidate group through the Hungarian algorithm, and the optimal trajectory is selected to inherit its ID.

[0050] For each trajectory, construct a state vector containing the center position and velocity.

[0051] Where and represent the center position of the trajectory, and and represent the velocities of the trajectory in the horizontal and vertical directions, respectively.

[0052] When initializing a new trajectory, the center of the current detection box is used as the initial position, and the velocity is initialized to 0, that is:

[0053] Simultaneously, the initial state covariance matrix is ​​set. .

[0054] Using the constant velocity model, it is assumed that the fish's movement is uniform within a very short time frame in each frame. Therefore, in step S02, the formula for calculating the fish's center coordinates and corresponding fish velocity in each historical video frame based on the time difference between the current video frame and historical video frames is as follows:

[0055]

[0056] This represents the time difference between the current video frame and historical video frames. , This indicates the center coordinates of the fish in the historical video frame. , This indicates the predicted fish center coordinates for the current video frame.

[0057] The frame number corresponding to the Kalman state saved on the trajectory. With the current frame number The difference is expressed by the following formula:

[0058] Therefore, the prediction center of the j-th historical trajectory in the current frame using Kalman filtering is:

[0059] It should be noted that this invention constructs an integrated association framework of "trajectory prediction hard gating - global optimal association - adaptive new trajectory (fish that cannot be matched in historical frames) suppression": First, the motion state of each trajectory is established and the reachable area of ​​the current frame is obtained based on Kalman prediction. The "trajectory-detection" pair with inconsistent predictions is directly judged as an unselectable association pair (hard gating), thereby converging the original matching process of comparing all detections with all trajectories into a finite set of candidate associations retained only under motion consistency constraints. Then, the Hungarian matching algorithm is introduced on this candidate set for global optimal allocation to complete multi-target association, avoiding the erroneous matching caused by the original local greedy selection when the targets are dense or the appearance is disturbed. Furthermore, a new trajectory (fish that cannot be matched in historical frames) criterion that integrates gating results and appearance consistency is proposed. The new trajectory (fish that cannot be matched in historical frames) is only triggered when the detected target is "far away from all existing trajectories" or "within the gating range but inconsistent with the appearance of all trajectories", so as to suppress false link breaks and false ID creation caused by background interference or short-term appearance degradation.

[0060] This invention considers a "trajectory-detection" pair to have physical consistency and includes it in the candidate matching set only when the detected target falls within the predicted reachable range of the corresponding trajectory. Other pairs with inconsistent predictions are directly excluded, thus introducing a hard constraint based on motion consistency at the association stage. Based on the candidate matching set generated by the above constraints, this invention further constructs an association cost matrix that integrates appearance similarity and historical consistency. Within this constrained candidate space, a globally optimal allocation strategy is used to complete multi-target association, avoiding erroneous associations caused by local optima or greedy matching in situations of dense targets, similar appearances, or short-term disturbances. Simultaneously, this invention reconstructs the triggering conditions for new trajectories: only when the detected target is outside the prediction gate range of all existing trajectories, or although within the gate range, it exhibits significant appearance inconsistency with all candidate trajectories, is it determined to be a true new target and a new trajectory is triggered. For cases where the association confidence decreases only due to short-term appearance degradation, detection fluctuations, or background interference, the original trajectory state is maintained first, and a new trajectory is not immediately created. Through the aforementioned collaborative mechanism of "predictive gating constraints - global optimal association - new trigger suppression", this invention significantly enhances the continuous matching of trajectories while maintaining real-time performance, and effectively reduces the phenomenon of chain breakage and ID switching in unobstructed motion scenes and complex background conditions.

[0061] Further, in step S03, matching the predicted fish center coordinates of multiple current video frames with the actual fish center coordinates of multiple current video frames includes: S0301: Combine the predicted fish center coordinates of each current video frame with the actual fish center coordinates of each current video frame in pairs to form multiple first combinations; S0302: Calculate the Mahalanobis distance between the predicted fish center coordinates and the actual fish center coordinates of the current video frame in the first combination; S0303: Select the first combination whose Mahalanobis distance meets the second preset condition as a candidate pair for each fish in the fish image of the current video frame and the fish image of the historical video frame.

[0062] For the j-th trajectory, the uncertainty matrix of its current position prediction The formula is:

[0063] Let R represent the two-dimensional submatrix of the corresponding position component in the state covariance matrix of the j-th trajectory prediction. The '-' sign indicates that the state covariance matrix is ​​obtained after trajectory prediction but has not yet been updated with real detection. R represents the observation noise matrix.

[0064] The center of the i-th detection box in the current frame is: Calculate the Mahalanobis distance between the detection center of detection i and the predicted center of trajectory j. The formula is:

[0065]

[0066] express The transpose of .

[0067] This indicates whether the detected position of the i-th fish is within the predicted range of the j-th trajectory, when the following conditions are met. hour, A value of 1 indicates that the i-th fish detection and the j-th trajectory form a candidate pair; otherwise, a value of 0 indicates that no candidate pair is formed. All detections and trajectories... Values ​​constitute the gating matrix As the output of the Kalman gating module, according to the gating matrix The detection and trajectory will be used to form candidate pairs. The value is set to 70.

[0068] Furthermore, such as Figure 5 As shown, in step S07, the Hungarian algorithm is used to match each fish image in the current video frame and each fish image in a historical video frame, including: S0701: Calculate the normalized exponential similarity and cosine similarity between each of the first appearance embedding vectors and each of the second appearance embedding vectors, and combine them to form the first appearance cost matrix; S0702: Calculate the direction vector between the real fish center coordinates corresponding to each of the first appearance embedding vectors and the fish center coordinates corresponding to each of the second appearance embedding vectors to obtain the first direction vector; S0703: Obtain the direction vector corresponding to the second appearance embedding vector, and combine it with the first direction vector to form the second appearance cost matrix; S0703: Combine the first appearance cost matrix and the second appearance cost matrix to form a third appearance cost matrix; S0704: For the third appearance cost matrix, use the Hungarian algorithm to match and obtain the second appearance embedding vector corresponding to each first appearance embedding vector; S0705: Based on the second appearance embedding vector corresponding to each of the first appearance embedding vectors, obtain the historical video frame fish image corresponding to each fish in the current video frame.

[0069] In one specific embodiment, during the data association stage, directional constraints are not forcibly imposed on all detected targets. Instead, it is first determined whether there is appearance matching ambiguity in the current frame (e.g., there are multiple feasible candidate trajectories for the detected target and the appearance advantage is insufficient). Only for such ambiguous detected targets, the direction correction term is triggered. The consistency between the "motion trend predicted by the trajectory direction momentum" and the "instantaneous direction of the detected target relative to the historical observation of the trajectory" is measured. In the case of ambiguity, matching pairs with consistent motion trends are selected first to achieve identity regression and stable recovery after occlusion. In the case of no ambiguity, the appearance dominated by the motion is maintained to avoid the direction estimation noise from interfering with normal association.

[0070] When performing detection and trajectory association, candidate pairs that can be matched are first generated through Kalman prediction gating or distance gating: for each detection and each trajectory, it is determined whether it meets the motion consistency constraint (the detection center falls within the prediction threshold). Those that do not meet the constraint are directly marked as unselectable edges and given a high cost in subsequent global allocation to ensure that they will not be selected; a basic cost based on appearance similarity is constructed on the candidate pairs. Then, an "ambiguity trigger condition" is calculated for each detection. The directional momentum correction is triggered only when the detection still corresponds to multiple candidate trajectories after gating (e.g., the number of candidates is ≥2) and the best appearance match is not strong enough (e.g., the maximum similarity is below the threshold): for the detection and each candidate trajectory, the current displacement direction is formed by using the detection center and the trajectory's most recent observation center obs_c1, and compared with the direction pointed to by the trajectory mom_xy to obtain the degree of directional inconsistency as the directional cost; finally, the directional cost is superimposed on the basic cost by the coefficient lambda_dir, but it only applies to the detection targets that are judged as ambiguous, thereby avoiding the directional term from "taking over" in unambiguous scenarios; where lambda_dir is set to 0.5. The system performs a global optimal allocation (Hungarian algorithm) on the above constraints and the modified cost matrix to complete the association. In scenarios where it is difficult to allocate due to occlusion recurrence or similar appearance, the system prioritizes the reconnection of trajectories that are consistent with the historical movement trend, thereby improving the ID recovery rate after occlusion and reducing ID switching and trajectory fragmentation.

[0071] Furthermore, in step S0702, the formula for calculating the first direction vector is as follows:

[0072] This represents the coordinates of the real fish center corresponding to the i-th first appearance embedding vector. This represents the coordinates of the fish center corresponding to the j-th second appearance embedding vector. Represents a constant. Indicates to Find the second norm.

[0073] Furthermore, in step S0703, the formula for calculating the second appearance cost matrix is ​​as follows:

[0074] Indicates matrix transpose. Let represent the direction vector corresponding to the j-th second appearance embedding vector, where j represents the index of the second appearance embedding vector. Let represent the first direction vector between the i-th first appearance embedding vector and the j-th second appearance embedding vector, where i represents the index of the first appearance embedding vector. Represents the inverse cosine function. Indicates will It is limited to the range where the minimum value is -1 and the maximum value is 1.

[0075] Ambiguity is defined as follows: when the number of candidate pairs is greater than or equal to 2 and the best appearance similarity s among the candidate pairs is less than the threshold of 0.55, ambiguity exists. If the current detection of fish 2 is ambiguous, its directional cost is calculated. If the directional costs for each fish in the trajectory are [0.2, 0.0, 0.1], and λ is set to 0.5, the total cost is [0.7, 0.2, 0.6] + λ[0.2, 0.0, 0.1] = [0.8, 0.2, 0.65]. Let be the unit direction momentum vector of trajectory j, as shown in the following formula:

[0076] In one specific embodiment, the third appearance cost matrix is ​​as follows:

[0077] Ultimately, the Hungarian algorithm selects a pair of one-to-one matches in this matrix to minimize the total cost.

[0078] Finally, the trajectory and detection were successfully matched one by one, and the motion state vector within the Kalman filter was updated. During the state vector update process, a correction is made based on the deviation between the current detection box center and the predicted center, using the following formula:

[0079] in This represents the Kalman gain matrix.

[0080] After using direction vector detection and trajectory matching, update The formula is as follows:

[0081]

[0082]

[0083] It is the detection center that matches the trajectory. This is the frame number detected this time. The frame number when the trajectory was last detected and the direction was updated. difference, The value is 0.6.

[0084] For detections that do not match a trajectory, a new trajectory is triggered only if the detection confidence of the target is higher than 0.65, it is outside the prediction gate range of all existing trajectories, or it is within the gate range but its appearance similarity with each candidate trajectory is lower than the threshold of 0.4. Otherwise, the detection is considered as a case of short-term appearance degradation, detection fluctuation, or background interference causing a decrease in association confidence, and the original trajectory status is maintained instead of being created immediately.

[0085] The following is experimental data from a 680-frame video of 8 fish. IDSW: The number of times the ID changes when the same real fish appears consecutively over time. Frag: If the trajectory of the same real target is interrupted once and then reappears, it is considered a frag. num_ids: The total number of different track_ids created throughout the entire video. Matches: The total number of times the trajectory was successfully matched with the ground truth bounding box across all frames. unmatched_prev: The total number of tracks that existed in the previous frame but did not match any detection in the current frame across all frames. unmatched_cur: The total number of detections in the current frame that did not match any historical trajectory.

[0086] The same or similar labels correspond to the same or similar parts; The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for matching underwater fish, characterized in that, include: S1: Obtain the image to be matched, and at least one matched fish image; The image to be matched contains at least one fish, the matched fish images contain one fish, and the total number of matched fish images is greater than or equal to the total number of fish in the image to be matched. S2: Use a fish selection model to extract the images to be matched, and obtain one or more extracted fish images; S3: Generate one or more first appearance embedding vectors based on one or more of the extracted fish images; Generate one or more second appearance embedding vectors based on one or more of the matched fish images; S4: Using the Hungarian algorithm, match each first appearance embedding vector with a second appearance embedding vector to obtain the second appearance embedding vector corresponding to each first appearance embedding vector. S5: Based on the second appearance embedding vector corresponding to each first appearance embedding vector, obtain the matched fish image corresponding to each fish in the image to be matched.

2. The underwater fish matching method according to claim 1, characterized in that, In step S3, based on one or more of the extracted fish images, one or more first appearance embedding vectors are generated, including: S301: Select one fish image from one or more extracted fish images as the first fish image; S302: Use an external instance segmentation model to generate a mask for the first fish image to obtain a fish body mask; S303: Process the first fish image using the ROI-align algorithm to obtain the ROI features corresponding to the first fish image; S304: Weight the ROI features corresponding to the first fish image and the fish body mask to obtain a feature map for background interference suppression; S305: The feature map of the background interference suppression is processed by the tracking head to obtain the first appearance embedding vector; S306: Select another fish image from one or more of the extracted fish images as the new first fish image, and repeat steps S302 to S305 until all of the extracted fish images have been selected.

3. The underwater fish matching method according to claim 2, characterized in that, In step S304, the ROI features corresponding to the first fish image and the fish body mask are weighted, including: S30401: Obtain preset background retention parameters; S30402: Process the fish mask according to the preset background retention parameters to obtain the fish weight; S30403: Combine the fish body weights with the ROI features to obtain a feature map for background interference suppression.

4. The underwater fish matching method according to claim 1, characterized in that, Using the Hungarian algorithm, matching is performed on one or more of the first appearance embedding vectors and the second appearance embedding vectors, including: S401: Calculate the normalized exponential similarity and cosine similarity between each of the first appearance embedding vectors and each of the second appearance embedding vectors. S402: Combine the normalized exponential similarity and cosine similarity to form the first appearance cost matrix; S403: For the first appearance cost matrix, use the Hungarian algorithm to match and obtain the second appearance embedding vector corresponding to each first appearance embedding vector.

5. The underwater fish matching method according to claim 4, characterized in that, The formula for calculating normalized index similarity is as follows: Let r represent the r-th first appearance embedding vector, where r, This indicates the first appearance embedding vector index. This represents the second appearance embedding vector. , This indicates the second appearance embedding vector index. This represents the total number of first appearance embedding vectors. This represents the total number of second appearance embedding vectors.

6. A method for matching underwater fish in videos, characterized in that, include: S01: Obtain the true fish center coordinates of each fish in the current video frame's image and fish image, and the first appearance embedding vector of each fish; Obtain one or more historical video frame fish images, and the fish center coordinates, fish velocity, and second appearance embedding vector of each fish image. Each historical video frame's fish image contains one fish; the current video frame's image contains at least one fish. S02: Using the Kalman filter algorithm, based on the time difference between the current video frame and the historical video frames, the fish center coordinates and the corresponding fish speed of each historical video frame are calculated to obtain the predicted fish center coordinates of multiple current video frames. S03: Match the predicted fish center coordinates of multiple current video frames with the actual fish center coordinates of multiple current video frames to obtain candidate pairs of each fish in the fish image of the current video frame and the fish image of the historical video frame. S04: Based on the candidate pairs of fish images in the current video frame and fish images in historical video frames, the first appearance embedding vector corresponding to each fish in the current video frame, and the second appearance embedding vector corresponding to each fish image in historical video frames, calculate the best appearance similarity of each candidate pair. S05: Determine whether the appearance similarity of the multiple candidate pairs meets the first preset condition; if the preset condition is met, proceed to step S06; if the preset condition is not met, proceed to step S07. S06: Based on the fish image of the current video frame and fish images of multiple historical video frames, execute the underwater fish matching method according to any one of claims 1 to 5 to obtain the fish image of the historical video frame corresponding to each fish in the current video frame; S07: Based on the first appearance embedding vector and the center coordinates of the real fish corresponding to each fish in the current video frame, and the second appearance embedding vector and the center coordinates of the fish corresponding to each fish image in the historical video frame, the Hungarian algorithm is used to match each fish in the current video frame and each fish image in the historical video frame to obtain the fish image in the historical video frame corresponding to each fish in the current video frame.

7. The underwater fish matching method for video according to claim 6, characterized in that, In step S03, the predicted fish center coordinates of multiple current video frames are matched with the actual fish center coordinates of multiple current video frames, including: S0301: Combine the predicted fish center coordinates of each current video frame with the actual fish center coordinates of each current video frame in pairs to form multiple first combinations; S0302: Calculate the Mahalanobis distance between the predicted fish center coordinates and the actual fish center coordinates of the current video frame in the first combination; S0303: Select the first combination whose Mahalanobis distance meets the second preset condition as a candidate pair for each fish in the fish image of the current video frame and the fish image of the historical video frame.

8. The underwater fish matching method for video according to claim 6, characterized in that, In step S07, the Hungarian algorithm is used to match each fish in the current video frame and each fish image in a historical video frame, including: S0701: Calculate the normalized exponential similarity and cosine similarity between each of the first appearance embedding vectors and each of the second appearance embedding vectors, and combine them to form the first appearance cost matrix; S0702: Calculate the direction vector between the real fish center coordinates corresponding to each of the first appearance embedding vectors and the fish center coordinates corresponding to each of the second appearance embedding vectors to obtain the first direction vector; S0703: Obtain the direction vector corresponding to the second appearance embedding vector, and combine it with the first direction vector to form the second appearance cost matrix; S0703: Combine the first appearance cost matrix and the second appearance cost matrix to form a third appearance cost matrix; S0704: For the third appearance cost matrix, use the Hungarian algorithm to match and obtain the second appearance embedding vector corresponding to each first appearance embedding vector; S0705: Based on the second appearance embedding vector corresponding to each of the first appearance embedding vectors, obtain the historical video frame fish image corresponding to each fish in the current video frame.

9. The underwater fish matching method for video according to claim 8, characterized in that, In step S0702, the formula for calculating the first direction vector is as follows: This represents the coordinates of the real fish center corresponding to the i-th first appearance embedding vector. This represents the coordinates of the fish center corresponding to the j-th second appearance embedding vector. Represents a constant. Indicates to Find the second norm.

10. The underwater fish matching method for video according to claim 8, characterized in that, In step S0703, the formula for calculating the second appearance cost matrix is ​​as follows: Indicates matrix transpose. Let represent the direction vector corresponding to the j-th second appearance embedding vector, where j represents the index of the second appearance embedding vector. Let represent the first direction vector between the i-th first appearance embedding vector and the j-th second appearance embedding vector, where i represents the index of the first appearance embedding vector. Represents the inverse cosine function. Indicates will It is limited to the range where the minimum value is -1 and the maximum value is 1.

Images