Fish individual identification method based on contrast language-image pre-training model

Abstract

A fish individual identification method based on a contrast language-image pre-training model comprises the following steps: data acquisition and preprocessing: model construction: constructing a fish individual identification model based on a pre-trained CLIP model, introducing a learnable text prompt module, and aligning text descriptions of different fish individuals with image features; a lightweight fine-grained feature enhancement module is designed; in the first stage, image encoder parameters and text encoder parameters are frozen, and only learnable text prompts are optimized; in the second stage, image encoder parameters are unfrozen, text encoder parameters are fixed, a fine-grained feature enhancement module is combined, training is carried out through cross entropy loss and a triple loss function, and image feature representation is optimized; and model testing: extracting feature vectors of test images, and performing similarity calculation on the feature vectors and feature vectors in the training set to realize fish individual identification. The method can accurately capture the fine-grained features of fish individuals, can improve the recognition precision in a complex environment, and has high robustness.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision image training, and in particular relates to a fish individual fine-grained recognition method based on a contrast language-image pre-training model, which is suitable for long-term tracking, behavior analysis and precise breeding management of fish individuals in a complex underwater environment. BACKGROUND

[0002] Fish individual re-identification is derived from the uniqueness, difference, predictability and singularity of the collected data of different individuals, and is caused by image attributes such as color, gradient, edge, imaging angle, etc. The significant recognition of individual differences and how to perceive and process visual stimuli are closely related, and are being studied by multiple disciplines including neurobiology, cognitive psychology and computer vision.

[0003] Fish individual recognition is a key technology for intelligent management of aquaculture, but faces the following challenges: (1) The appearance difference between individuals is subtle, only reflected in local features such as scale texture, fin shape or body proportion, etc. (2) The posture changes dramatically, the fish body posture and shape change greatly, resulting in unstable feature extraction. (3) The underwater environment imaging is complex, due to underwater light, dynamic background, body occlusion and light refraction and scattering, etc. The underwater environment light is complex, and the dynamic background interference is serious, resulting in significant appearance difference of the same fish in different environments. (4) Lack of identity identification, fish appearance relies heavily on natural features. (5) Data labeling is difficult, due to the fast swimming speed of fish and frequent occlusion, which easily introduces noise, resulting in high ID labeling error rate, ultimately affecting model training. Overall, these challenges require the model to capture finer individual features and have stronger scene adaptation ability to effectively distinguish different fish individuals.

[0004] Early image processing methods divide fish individuals into different IDs through local feature representation (including SIFT features, ORB features and SURF features), and realize ID assignment of new images by using feature matching based on Euclidean distance. This traditional method relies on local feature representation (such as SIFT) or single modal model, and is difficult to be directly applied to fine-grained recognition, and also difficult to capture the subtle differences of fish individuals.

[0005] Existing technologies such as idTracker, FIRN based on YOLOv4, CGFI, FishFace, etc. have made some progress, but have not fully utilized cross-modal information, and the local feature extraction capability is limited, and have not fully integrated global and local features, resulting in insufficient generalization ability across species or across environments. SUMMARY

[0006] The application provides a fish individual identification method based on a contrastive language-image pre-training model, in particular, proposes a fish individual identification method combining a CLIP pre-training model, a learnable text prompt, and fine-grained feature enhancement, solves the technical problems that the existing fish individual identification method is insufficient in capturing fine-grained features, is difficult to distinguish individuals with highly similar appearances, and the existing model has weak generalization ability in complex underwater environments, and cannot comprehensively fuse global and local features, resulting in high misidentification rate.

[0007] To solve at least one of the above technical problems, the technical solution adopted by the application is:

[0008] The fish individual identification method based on the contrastive language-image pre-training model comprises the following steps:

[0009] Data acquisition and preprocessing: collect fish image data, divide the video data into single-frame images, extract fish individual instances using a target detection model, and generate a training set and a test set;

[0010] Model construction: construct a fish individual identification model CLIP-AFIR based on a pre-trained CLIP model, introduce a learnable text prompt module, align the text description of different fish individuals with image features, and design a lightweight fine-grained feature enhancement module to enhance the local detail feature extraction capability;

[0011] Model training: the model is trained in two stages, wherein in the first stage, the image encoder parameters and the text encoder parameters are frozen, and only the learnable text prompt is optimized; in the second stage, the image encoder parameters are unfrozen, the text encoder parameters are fixed, the fine-grained feature enhancement module is combined, and the image feature representation is optimized through a cross-entropy loss function and a triplet loss function;

[0012] Model testing: extract the test image feature vector, calculate the similarity with the feature vector in the training set, and realize fish individual identification.

[0013] Further, when collecting data, a high-resolution camera with a resolution of not less than 2560x1920 and a frame rate of not less than 25fps is used to collect images, and a target detection model is used to extract fish individual instances.

[0014] Further, the learnable text prompt module comprises the following specific steps:

[0015] Initialize the text template as "A photo of a [X]1 [X]2 [X]3... [X] N fish", wherein [X]1 to [X] N are learnable text labels, and the embedding dimension of each text label is consistent with the word embedding dimension of the text encoder;

[0016] The fixed text part in the initialization text template is converted into a fixed word embedding vector through byte pair encoding;

[0017] The learnable text mark [X]1 to [X] N is spliced with the fixed word embedding vector to form a complete text input sequence;

[0018] The spliced learnable text sequence is input into the frozen text encoder to generate the final text feature vector T f ∈R CT , wherein C T = 512.

[0019] Further, the fine-grained feature enhancement module has the following specific steps:

[0020] The input image is divided into multiple local regions, and a plurality of local image blocks are generated through sliding window technology, including: an input image I, a sliding window size of k×k, and a sliding step of s, to generate a series of local image blocks wherein N is the number of generated local image blocks;

[0021] The mask self-attention of each local image block is calculated.

[0022] Further, when calculating the number of local image blocks, the overlapping area of the local image blocks is calculated, wherein the size of the overlapping area is (k-s)×k; when the size of the input image is H×W, the number of local image blocks that can be generated is .

[0023] Further, in the first stage training process, only the learnable text prompt is optimized, and a contrastive learning loss function is used to optimize the alignment of the text description and the image feature;

[0024] The loss function of the first stage includes an image-text contrastive loss function L t2i and a text-image contrastive loss function L i2t , and their formulas are respectively:

[0025]

[0026]

[0027] wherein S(V f ,T f ) is the cosine similarity of the image feature V f and the text feature T f ; τ is a temperature parameter; B is the batch size; and S(y i) is an index set, representing the same index set as the i-th image (i∈[1, B]), denoted as S(y i ) = {p∈[1, B]: y p = y i}, where y i contains the positive sample image index p with the same ID.

[0028] Further, the total loss L stage1 of the first stage training is composed of the image-text contrast loss function L t2i and the text-image contrast loss function L i2t in the first stage, specifically:

[0029] L stage1 = L t2i + L i2t .

[0030] Further, the total loss function L stage2 of the second stage training is composed of the cross-entropy loss function, the triplet loss function and the image-text contrast loss function in the second stage, specifically:

[0031]

[0032] wherein, L ID is the cross-entropy loss function; L Triplet is the triplet loss function; is the image-text contrast loss function; λ and β are weight coefficients.

[0033] Further, the calculation formula of the cross-entropy loss function L ID is:

[0034]

[0035] wherein, represents the target distribution of the ID of the k-th fish; ∈ is a hyperparameter, used to encourage the model to maintain a low confidence during training; p k represents the predicted score of class k; N is the total number of classes; k represents the index of the fish class.

[0036] Further, the calculation formula of the triplet loss function L Triplet is:

[0037] L Triplet = max(d p -d n + α, 0)

[0038] wherein, and respectively represent the feature distance between the positive sample pair and the negative sample pair; represents the selected anchor sample; represents the positive sample; represents the negative sample; α is the triplet loss function L Triplet interval; p represents the index of the positive sample; n represents the index of the negative sample; f represents the layer where the feature vector is located; a represents the anchor sample; d represents the distance function; V represents the feature vector.

[0039] Further, the image-text contrast loss function in the second stage The calculation formula is:

[0040]

[0041] wherein, represents the target distribution of the ID of the kth fish; ∈ is a hyperparameter for encouraging the model to maintain a low confidence during training; N is the total number of classes; CE i2t represents the cross-entropy loss from image to text; represents the image feature vector of the ith fish sample; represents the feature vector of the text description corresponding to the ith sample; τ represents a temperature parameter for adjusting the smoothness of the similarity distribution and controlling the attention weight of the model to difficult samples; y j represents the text description of the jth fish class.

[0042] Further, the model test is calculated by the Euclidean distance, the feature vector of the test image is matched with the feature vector in the training set, and the individual recognition of the fish is realized; the specific steps are:

[0043] The test image is input into the trained model to extract its feature vector;

[0044] The feature vector is calculated with the Euclidean distance of the feature vectors of all images in the training set; by calculating the Euclidean distance between the feature vector of the test image and the feature vector of each image in the training set, a distance value can be obtained;

[0045] According to the distance value, the images in the training set are sorted, and the image with the smallest distance value is selected as the matching result.

[0046] The fish individual recognition method based on the contrastive language-image pre-training model designed in the application can enhance the cross-modal alignment capability through the PLM module, and improve the local feature distinguishability through the FFEM module, and the two-stage training strategy can optimize the feature space intra-class aggregation and inter-class separation; can accurately capture the fine-grained features of fish individuals, and can accurately distinguish fish individuals with highly similar appearance; moreover, it can also improve the recognition accuracy in complex underwater environment, and has strong robustness and adaptability, and can also comprehensively fuse global and local features.

[0047] The evaluation indexes of the method in the application on the industrial breeding data set include cumulative matching characteristics CMC and average precision mean mAP, wherein the measured Rank-1 accuracy reaches 97.5%, and the average precision mean mAP is as high as 62.2%, which is obviously better than the existing model. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The flow chart of the fish individual recognition method based on the pre-trained CLIP model in the application;

[0049] Figure 2 The overall architecture schematic diagram of the fish individual recognition model CLIP-AFIR based on the pre-trained CLIP model in the application;

[0050] Figure 3 The data acquisition flow chart in the application;

[0051] Figure 4 The pre-processing flow chart in the application;

[0052] Figure 5 The structure schematic diagram of the learnable text prompt module in the application;

[0053] Figure 6 The flow schematic diagram of the fine-grained feature enhancement module in the application;

[0054] Figure 7 The contrastive diagram of Grad-CAM visualization performed in the application;

[0055] Figure 8 The t-SNE feature dimension reduction visualization diagram of VGG16 performed in the application;

[0056] Figure 9 The t-SNE feature dimension reduction visualization diagram of ResNet50 performed in the application;

[0057] Figure 10 The t-SNE feature dimension reduction visualization diagram of ArcFish performed in the application;

[0058] Figure 11Figure 1 shows the t-SNE feature dimension reduction visualization chart of CLIP-AFIR after stage 1 in this application.

[0059] Figure 12 Figure 2 shows the t-SNE feature dimension reduction visualization chart of CLIP-AFIR after stage 2 in this application. DETAILED DESCRIPTION

[0060] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0061] The present embodiment proposes a fish individual recognition method based on a contrastive language-image pre-trained model, wherein the flowchart of the recognition method is as shown in Figure 1 Figure 1, and correspondingly, the overall architecture of the fish individual recognition model CLIP-AFIR based on the pre-trained CLIP model is as shown in Figure 2 The steps are as follows:

[0062] S1, data acquisition and preprocessing: collect fish image data, divide video data into single-frame images, extract fish individual instances using a target detection model, and generate training and test sets.

[0063] As shown in Figure 3 Figure 1, it is a collection flowchart of fish image data collected from an industrial circulating water aquaculture environment. When collecting data, a high-resolution camera is used to collect fish image data in the actual aquaculture environment, and a target detection model (such as YOLOv8) is used to extract fish individual instances. The resolution of the high-resolution camera is not less than 2560x1920, and the frame rate is not less than 25fps, to ensure that the collected images are clear and have sufficient detail information. The target detection model accurately detects and extracts fish individual instances from the collected images, and generates an image data set containing fish individuals.

[0064] As shown in Figure 4 Figure 2, after data collection is completed, video processing software is used to divide video data into single-frame images, and a target detection model is used to automatically extract fish individual instances, to generate training and test sets.

[0065] S2, model construction: based on the pre-trained CLIP model, a fish individual recognition model CLIP-AFIR is constructed, a learnable text prompt module (Prompt Learner Module, PLM) is introduced, and the text descriptions of different fish individuals are aligned with the image features. A lightweight fine-grained feature enhancement module (Fine-grained Feature Enhancement Module, FFEM) is designed to enhance the extraction ability of the fish individual recognition model for local detail features.

[0066] Specifically, as shown in Figure 2 , the CLIP-AFIR model is based on a pre-trained CLIP model, and by introducing a learnable text prompt module PLM and a fine-grained feature enhancement module FFEM, the extraction ability of the CLIP-AFIR model for local detailed features is enhanced. Among them, the CLIP model is a pre-trained contrastive language-image model, which has strong cross-modal feature learning ability and can map images and texts into a shared feature space. The learnable text prompt module is to align the text description of different fish individuals with the image features by learning adjustable text prompts, so that the text description and the image features are closer in the feature space. The fine-grained feature enhancement module FFEM is to enhance the extraction ability of the CLIP-AFIR model for local detailed features of images through sliding window technology and mask self-attention mechanism.

[0067] S21, the learnable text prompt module PLM, as shown in Figure 5 , includes the following steps, specifically:

[0068] Initialize the text template as "A photo of a [X]1[X]2[X]3...[X] N fish", wherein [X]1 to [X] N are learnable text labels, and the embedding dimension of each text label is consistent with the word embedding dimension of the text encoder.

[0069] Then, the fixed text part (such as "A photo of a" and "fish") in the initialized text template is converted into a fixed word embedding vector through byte pair encoding BPE.

[0070] The learnable text labels [X]1 to [X] N are spliced with the fixed word embedding vector to form a complete text input sequence.

[0071] The spliced learnable text sequence is input into the frozen text encoder to generate the final text feature vector T f ∈R CT , wherein C T = 512.

[0072] S22, the fine-grained feature enhancement module FFEM is to enhance the extraction ability of the CLIP-AFIR model for local detailed features through the following steps, as shown in Figure 6 , the specific steps are:

[0073] The input image is divided into multiple local regions, and multiple local image blocks are generated through sliding window technology. Specifically, for the input image I, a sliding window size of k x k is used to slide on the image with a step size of s, generating a series of local image blocks where N is the number of generated local image blocks.

[0074] When calculating the number of local image blocks, the overlapping region of the local image blocks is calculated first, where the size of the overlapping region is (k-s) x k. Therefore, when the size of the input image is H x W, the number of local image blocks that can be generated is .

[0075] Then, the mask self-attention of each local image block is calculated to enhance the extraction of local detail features.

[0076] S3, model training: a two-stage fine-tuning strategy is used to train the model, where the first stage: freeze the image encoder parameters and the text encoder parameters, only optimize the learnable text prompt; the second stage: unfreeze the image encoder parameters, fix the text encoder parameters, combine the fine-grained feature enhancement module, and train through the cross-entropy loss function and the triplet loss function, optimize the image feature representation.

[0077] S31, in the first stage of the training process, the image encoder parameters and the text encoder parameters are frozen, only the learnable text prompts of different fish individuals are trained, the contrast learning loss function is used, the alignment of the text and image features is optimized, and the image features and the corresponding text description are closer in the feature space.

[0078] where the loss function of the first stage includes the image-text contrast loss function L t2i and the text-image contrast loss function L i2t , whose formulas are respectively:

[0079]

[0080] In the formula, S(V f ,T f ) is the cosine similarity of the image feature V f and the text feature T f ; τ is the temperature parameter; B is the batch size; S(y i ) is the index set, which represents the same index set as the i-th image (i∈[1,B]), denoted as S(y i )={p∈[1,B]:y p =y i}, where y iThe positive sample image index p of the same ID is included; t2i (Text-to-Image) and i2t (Image-to-Text) represent the contrast loss in two different directions, which are used to align the semantic relationship of text to image and image to text respectively.

[0081] Further, the total loss function L of the first stage training stage1 The image-text contrast loss function L t2i and the text-image contrast loss function L i2t are composed, specifically:

[0082] L stage1 = L t2i + L i2t .

[0083] S32, in the second stage training process, the image encoder parameters are unfrozen, the text encoder parameters are fixed, the fine-grained feature enhancement module is combined, the cross entropy loss and the function triplet loss function are used to optimize the image feature representation, and the purpose is to improve the recognition accuracy and robustness of the model.

[0084] The total loss function L of the second stage training stage2 is composed of the cross entropy loss function, the triplet loss function and the image-text contrast loss function in the second stage, specifically:

[0085]

[0086] Wherein, L ID is the cross entropy loss function; L Triplet is the triplet loss function; is the image-text contrast loss function; λ and β are weight coefficients.

[0087] Further, the calculation formula of the cross entropy loss L ID is:

[0088]

[0089] Wherein, represents the target distribution of the ID of the kth fish; ∈ is a hyperparameter, which is used to encourage the model to maintain a low confidence during training; p k represents the predicted score of class k; N is the total number of classes; k represents the index of the fish individual class.

[0090] Further, the calculation formula of the triplet loss function L Triplet is:

[0091] L Triplet = max(d p -d n+ a, 0)

[0092] wherein, and represent the feature distance between positive sample pairs and negative sample pairs, respectively; represents the selected anchor sample; represents the positive sample; represents the negative sample; a is the margin of the triplet loss function L Triplet ; p represents the index of the positive sample; n represents the index of the negative sample; f represents the layer where the feature vector is located; a represents the anchor; d represents the distance function; V represents the feature vector.

[0093] Further, the calculation formula of the image-text contrastive loss function is as follows:

[0094]

[0095] wherein, represents the target distribution of the ID of the kth fish; ∈ is a hyperparameter for encouraging the model to maintain a low confidence during training; N is the total number of categories; CE i2t represents the cross-entropy loss from image to text; represents the image feature vector of the ith fish sample; represents the feature vector of the text description corresponding to the ith sample; τ represents a temperature parameter for adjusting the smoothness of the similarity distribution and controlling the attention weight of the model to difficult samples; y j represents the text description of the jth fish category.

[0096] The discriminative ability of the model is improved by optimizing the distance between sample features. During the training process, images of the same ID are pulled closer, while images of different IDs are pushed farther apart.

[0097] S4, model testing: extract the feature vector of the test image, and calculate the similarity with the feature vector in the training set to realize fish individual recognition.

[0098] The CLIP-AFIR model is tested by calculating the Euclidean distance. The feature vector of the test image is matched with the feature vector in the training set to realize fish individual recognition. The specific steps are as follows:

[0099] First, input the test image into the trained model to extract its feature vector;

[0100] Then, calculate the Euclidean distance between the feature vector and the feature vector of all images in the training set; by calculating the Euclidean distance between the feature vector of the test image and the feature vector of each image in the training set, a distance value can be obtained;

[0101] Finally, the images in the training set are sorted according to the distance values, and the image with the smallest distance value is selected as the matching result.

[0102] For further understanding of the method of the present application by those skilled in the art, the technical solutions of the present application will be explained in detail below in combination with specific embodiments. Obviously, the described embodiments are only part of the embodiments of the present application, not all.

[0103] A fish individual recognition method based on a contrastive language-image pre-trained model, comprising the following steps:

[0104] S1, data acquisition and preprocessing

[0105] As shown in Figure 3 , fish image data was collected from an industrial recirculating aquaculture environment using an E264MPCX10 camera (resolution of 2560x1920, frame rate of 25fps) supplemented with white light illumination to ensure clear images. A total of 14510 images were obtained from 363 groupers.

[0106] As shown in Figure 4 , high-resolution cameras were used to collect video data of fish individuals, and video processing software was used to convert the video data into single-frame images. A target detection model was used to automatically extract fish individual instances, and the Any Labeling tool was used to correct the detection results to ensure the accuracy of the labels. The 14510 images obtained were randomly divided into a training set (10,157 images), a validation set (1,451 images), and a test set (2,902 images) in a ratio of 7:1:2.

[0107] S2, model construction

[0108] S21, as shown in Figure 5 , the learnable text prompt module is implemented by the following steps:

[0109] The text template is initialized as "A photo of a [X]1 [X]2 [X]3... [X] N fish", where [X]1 to [X] N are learnable text tokens, and the embedding dimension of each token is consistent with the word embedding dimension of the text encoder.

[0110] The fixed text part in the initialized text template (such as "A photo of a" and "fish") is converted into a fixed word embedding vector by Byte Pair Encoding (BPE).

[0111] The learnable text tokens [X]1 to [X] N are concatenated with the fixed word embedding vector to form a complete text input sequence.

[0112] The spliced learnable text sequence is input into the frozen text encoder to obtain the final text feature vector T f ∈R CT where C T = 512.

[0113] S22, as shown, the fine-grained feature enhancement module is to enhance the extraction ability of CLIP-AFIR model to local detail features through the following steps, the specific steps are: Figure 6

[0114] The input image is divided into multiple local regions, and multiple local image blocks are generated through sliding window technology. Specifically, for the input image I, a sliding window with size k x k is used to slide on the image with a step size s, generating a series of local image blocks where N is the number of generated local image blocks.

[0115] The calculation method of generating local image blocks is as follows:

[0116]

[0117] where H and W are the size of the input image.

[0118] The overlapping region of the local image block is calculated, and the size of the overlapping region is (k-s) x k.

[0119] Then, for each local image block Calculate its mask self-attention to enhance the extraction of local detail features.

[0120] S3, model training

[0121] S31, in the first stage of the training process, the contrast learning loss function is used to optimize the alignment of text and image features.

[0122] The loss function of the first stage includes image-text contrast loss function L t2i and text-image contrast loss function L i2t , which are calculated as follows:

[0123]

[0124] where S(V f ,T f ) is the cosine similarity of image feature V f and text feature T f , τ is the temperature parameter, B is the batch size. S(y i ) is the index set, which represents the same index set as the i-th image (i∈[1,B]), denoted as S(y​i ) = {p e [1, B]: y p i}, where y i contains the positive sample image index p of the same ID. After multiple experiments, the parameters are adjusted, and when τ = 0.1, the algorithm performs best.

[0125] The total loss function L stage1 in the first stage training includes the image-text contrast loss function L t2i and the text-image contrast loss function L i2t , as follows:

[0126] L stage1 = L t2i + L i2t

[0127] S32, in the second stage training process, the cross-entropy loss function and the triplet loss function are used to further optimize the image feature representation.

[0128] The total loss function L stage2 in the second stage training includes the cross-entropy loss function, the triplet loss function, and the image-text contrast loss function, as follows:

[0129]

[0130] where L ID is the cross-entropy loss function; L Triplet is the triplet loss function; is the image-text contrast loss function; λ and β are weight coefficients. After multiple experiments, the parameters are adjusted, and when λ = 0.25, β = 1, the algorithm performs best.

[0131] The calculation method of the cross-entropy loss function L ID is as follows:

[0132]

[0133] where, represents the target distribution of the ID of the kth fish, ∈ is a hyperparameter used to encourage the model to maintain a low confidence during training; p k represents the predicted score of class k; N is the total number of classes.

[0134] The calculation method of the triplet loss function L Triplet is as follows:

[0135] L Triplet = max(d p -d n + α, 0)​

[0136] wherein, and represent the feature distance between positive sample pairs and negative sample pairs, respectively; denotes the selected anchor sample; denotes the positive sample; denotes the negative sample; and a is the margin of the triplet loss function L Triplet The discriminative ability of the model is improved by optimizing the distance between sample features. During the training process, images of the same ID are pulled closer, while images of different IDs are pushed further apart.

[0137] The calculation formula of the image-text contrast loss function in the second stage is

[0138]

[0139] wherein, denotes the target distribution of the ID of the kth fish; ∈ is a hyperparameter used to encourage the model to maintain a low confidence during the training process; N is the total number of classes; CE i2t denotes the cross-entropy loss from image to text; denotes the image feature vector of the ith fish sample; denotes the feature vector of the text description corresponding to the ith sample; τ denotes a temperature parameter used to adjust the smoothness of the similarity distribution and control the attention weight of the model to difficult samples; y j denotes the text description of the jth fish class.

[0140] S4, model testing

[0141] The feature vector of the test image is matched with the feature vectors in the training set by calculating the Euclidean distance to realize the recognition of fish individuals. The specific steps are as follows:

[0142] First, the test image is input into the trained model to extract its feature vector.

[0143] Then, the feature vector is calculated with the feature vectors of all images in the training set by Euclidean distance; by calculating the Euclidean distance between the feature vector of the test image and the feature vector of each image in the training set, a distance value can be obtained.

[0144] Finally, the images in the training set are sorted according to the distance value, and the image with the smallest distance value is selected as the matching result.

[0145] ​In the present embodiment, to verify the effectiveness of the CLIP-AFIR model of the present application, the CLIP-AFIR model in the present embodiment and the existing mainstream models VGG16, ResNet series, MobileNet series, SwinTransformer, FIRN, CGFI, FishFace and ArcFish were compared in terms of results on an industrial breeding data set (containing 363 groupers, 14,510 images). The same training set and test set were used for evaluation, and the evaluation indexes were cumulative matching characteristics CMC and average precision mean value mAP. The cumulative matching characteristics CMC is based on the similarity between the query image and all candidate images in the database. The higher the image ranking, the greater the similarity with the query image. The calculation method is as follows:

[0146]

[0147] wherein N is the total number of query samples; rank i represents the correct matching ranking of the i-th query sample; I(rank i ≤k) is an indicator function, which is 1 when the correct matching appears in the first k positions, and 0 otherwise.

[0148] AP is the average precision of each correct sample in all search results; mAP represents the average of all APs, i.e. the average precision mean value. The average precision AP is calculated as follows:

[0149]

[0150] wherein |R i | is the total number of all correct matches of query i; P(k) represents the precision, which is the proportion of correct matches in the total search results in the current top k positions; q i represents the number of categories, and each category has an AP value.

[0151] The recognition method based on the CLIP-AFIR model of the present application and the recognition methods based on the existing VGG16, ResNet series, MobileNet series, SwinTransformer, FIRN, CGFI, FishFace and ArcFish models were compared, and the comparison results are shown in Table 1.

[0152] Table 1 Comparison of core indexes of the CLIP-AFIR model of the present application and the VGG16, ResNet series, MobileNet series, SwinTransformer, FIRN, CGFI, FishFace and ArcFish models

[0153]

[0154] As can be seen from Table 1, the core indicators mAP, Rank-1, Rank-5 and Rank-10 obtained by the fish individual recognition method based on the CLIP-AFIR model are significantly better than those of the existing methods.

[0155] From the comprehensive evaluation index mAP, the value obtained by the present application reaches 62.2, which is much higher than all the other comparison methods. This index reflects the balance of the recognition ability of the present application on different ID labels. A higher mAP value indicates that the present application can more accurately classify and recognize fish images of different IDs, and the performance on each fish ID is more balanced, and there is no situation that some categories are recognized well while others are poor.

[0156] For the Rank-1 index, the present application also achieves the highest accuracy of 97.5%, which means that the present application can directly identify fish individuals in most cases, greatly improving the robustness and reliability of fish re-identification. In the existing recognition methods, the model based on VGG16 decreased by 5.6%, the model based on ArcFish decreased by 6.8%, and the model based on Swin Transformer decreased by 10.6%; compared with these methods, the advantage of the present application is obvious. For the Rank-5 and Rank-10 indexes, the present application achieves an accuracy of 98.8% and 99.5%, respectively, i.e. the probability that the real label appears in the top 5 and top 10 of the model prediction. The Rank-5 and Rank-10 indexes are the core evaluation parameters based on the Top-K retrieval accuracy, which are mathematically defined as follows: in the top K candidate ID list (K = 5 or 10) of the prediction result, if it contains the real target ID, it is judged as a hit. Taking the fish sample with ID = 1 as an example, when the model outputs the Top-5 prediction sequence, the real ID = 1 must appear in the first to fifth position interval of the result list. Compared with the strict standard of Rank-1 which only examines the first prediction result, Rank-5 and Rank-10 relax the judgment threshold and more comprehensively represent the model's ability to represent fish features. For example, if the first prediction is mistaken for ID = 3, but the real ID = 1 appears in the third position, Rank-5 still judges it as correct. Higher Rank-5 and Rank-10 values indicate that the model can still efficiently locate the target after expanding the candidate range, thereby maintaining high recall rate and robustness in complex scenarios.

[0157] As Figure 7To further analyze the decision-making process of the present application, Grad-CAM (Gradient-weighted Class Activation Mapping) technology was used to visualize the attention distribution of different methods. Grad-CAM generates heat maps to show the image regions that the model focuses on when making decisions. The attention area of the present application effectively captures more feature information in the image in addition to covering the salient feature area. This characteristic enhances its adaptability to complex scenes and highlights its advantage in attention allocation; while other comparative methods (such as VGG16) have their attention scattered in the background or non-salient areas, resulting in a higher false detection rate. Other models mainly focus on local areas of the image, with attention points scattered and concentrated in a few salient feature locations, such as areas with high contrast or clear texture. This indicates that these models tend to obtain limited local features and may overlook other important details. For the Swin Transformer model, it shows a relatively broad attention distribution. However, compared with the present application, its attention area is not as comprehensive. In addition to the salient feature area, the present application can effectively capture more feature information in the image. This characteristic enhances its adaptability to complex scenes and highlights its advantage in attention allocation.

[0158] As shown in Figures 8-12 To visually demonstrate the feature distribution of different fish individuals, t-SNE feature dimensionality reduction was used to reduce the dimensionality of the feature embedding, and the results showed that the feature embedding of the present application in the first stage was relatively scattered, with unclear boundaries and obvious mixing between different IDs, making it a vague description. In contrast, in the second stage, the feature points of different IDs formed compact clusters, and the separation between IDs was significantly enhanced, indicating that the two-stage training strategy of the present application can further optimize the model parameters. Compared with the present application, there is still some overlap between the IDs of other methods. Overall, t-SNE visualization clearly shows that the present application performs better in local detail feature extraction in terms of intra-class consistency and inter-class separation.

[0159] To quantitatively evaluate the performance of the present application in actual aquaculture environments, 5, 10, 15, and 20 fish individual samples were randomly selected from the laboratory dataset (containing 30 groupers, 1642 images) and matched using the Euclidean distance algorithm. In addition to CMC and mAP indicators, precision (Precision), recall (Recall), and F1 score (F1 Score) commonly used in object detection were added to evaluate performance. To ensure the reliability and repeatability of the experimental results, the randomly selected samples were repeated 10 times, and the average value of the evaluation indicators was calculated. The final experimental results are shown in Table 2.

[0160] Further, the precision refers to the proportion of samples that are actually positive classes among all samples predicted as positive classes by the model; it measures the accuracy of the model in predicting positive classes. The calculation method is as follows:

[0161]

[0162] Where TP represents the number of samples that are correctly predicted by the model and are actually correct; FP represents the number of samples that are predicted as positive classes by the model, but are actually negative classes.

[0163] The recall rate refers to the proportion of samples that are correctly predicted as positive classes by the model among all samples that are actually positive classes; it measures the coverage ability of the model for positive classes. The calculation method is as follows:

[0164]

[0165] Where FN represents the number of samples that are predicted as negative classes by the model, but are actually positive classes.

[0166] The F1 score is the harmonic mean of precision and recall, used to comprehensively evaluate the performance of the model. The calculation method is as follows:

[0167]

[0168] Table 2 Comparison of average values of evaluation indicators of the CLIP-AFIR model of the present application and existing VGG16 and Swin Transformer models

[0169]

[0170] As can be seen from Table 2, the average values of the evaluation indicators obtained by the recognition method based on the CLIP-AFIR model of the present application all show significant advantages among all samples. The Rank-1 decreases steadily from 98.6 for 5 samples to 91.1 for 20 samples, with a performance decline rate of only 7.5%, which is significantly better than the recognition methods of other models. Notably, it still maintains Rank-5 of 97.3 and Rank-10 of 98.8 in the case of 20 samples. The precision and recall are 90.1 and 89.0, respectively, which are 2.9% and 5.5% higher than the second best method VGG16, respectively. The consistency of these multi-dimensional indicators verifies the effectiveness and robustness of the present application in the non-continuous fish individual recognition task.

[0171] The fish individual recognition method based on the contrastive language-image pre-training model designed in the application can enhance the cross-modal alignment capability through the PLM module, improve the local feature distinguishability through the FFEM module, and optimize the feature space intra-class aggregation and inter-class separation through the two-stage training strategy; can accurately capture the fine-grained features of fish individuals, and can accurately distinguish fish individuals with highly similar appearances; and can also improve the recognition accuracy in complex underwater environments, and has strong robustness and adaptability, and can also comprehensively integrate global and local features.

[0172] The evaluation indexes of the method of the application on the industrial breeding data set include cumulative matching characteristics CMC and average precision mean mAP, wherein the measured Rank-1 accuracy reaches 97.5%, and the average precision mean mAP is as high as 62.2%, which is obviously better than the existing model.

[0173] The above describes the embodiments of the application in detail, and the content is only the preferred embodiments of the application and cannot be considered as limiting the scope of the application. Any equivalent changes and improvements made within the scope of the application should still belong to the patent coverage of the application.

Claims

1. A fish individual recognition method based on a contrastive language-image pre-training model, characterized in that, The steps are as follows: Data collection and preprocessing: collect fish image data, divide video data into single-frame images, extract fish individual instances using a target detection model, and generate training and test sets; Model construction: based on the pre-trained CLIP model, construct the fish individual recognition model CLIP-AFIR, introduce a learnable text prompt module, and align the text descriptions of different fish individuals with image features; And design a lightweight fine-grained feature enhancement module to enhance the ability to extract local detailed features; Model training: two-stage training model is adopted, in which the first stage: freeze the image encoder parameters and text encoder parameters, only optimize the learnable text prompt; the second stage: unfreeze the image encoder parameters, fix the text encoder parameters, combine the fine-grained feature enhancement module, and train through the cross-entropy loss function and the triplet loss function to optimize the image feature representation; Model testing: extract the feature vector of the test image, calculate the similarity with the feature vector in the training set, and realize fish individual recognition. 2.The fish individual recognition method based on the contrastive language-image pre-trained model according to claim 1, wherein, When collecting data, high-resolution cameras with resolution not less than 2560x1920 and frame rate not less than 25fps are used to collect images, and fish individual instances are extracted through a target detection model. 3.The fish individual recognition method based on a contrastive language-image pre-trained model according to claim 1 or 2, characterized in that, The specific steps of the learnable text prompt module are as follows: Initialize the text template to "A photo of a [X]1[X]2[X]3...[X] N fish", where [X]1 to [X] N are learnable text tokens, each with an embedding dimension consistent with the word embedding dimension of the text encoder; Convert the fixed text part in the initialization text template to a fixed word embedding vector through byte pair encoding; learnable text token [X]1to [X] N concatenated with the fixed word embedding vector, forming a complete text input sequence; inputting the spliced learnable text sequence into the frozen text encoder to generate a final text feature vector where C T = 512. 4.The fish individual recognition method based on the contrastive language-image pre-trained model according to claim 1, wherein, The specific steps of the fine-grained feature enhancement module are as follows: The input image is divided into a plurality of local regions, and a plurality of local image blocks are generated by a sliding window technique, including: an input image I, a sliding window size of k×k, a sliding step of s, and a series of local image blocks are generated wherein N is the number of generated local image blocks Compute mask self-attention for each local image tile Compute mask self-attention for each local image tile 5.The fish individual recognition method based on the contrastive language-image pre-trained model according to claim 4, wherein, When the number of local image blocks is calculated, an overlapping area of the local image blocks is calculated, wherein the size of the overlapping area is (k-s) x k; when the size of the input image is H x W, the local image blocks that can be generated are .

6. The fish individual recognition method based on a contrastive language-image pre-trained model according to any one of claims 1-2, 4-5, characterized in that, In the first stage training process, only the learnable text prompt is optimized, and the alignment of the text description and the image feature is optimized using a contrast learning loss function; the loss function of the first stage includes an image-text contrast loss function L t2i and a text-image contrast loss function L i2t , whose formulas are respectively: where S(V f , T f ) is the cosine similarity between image feature V f and text feature T f ; τ is the temperature parameter; B is the batch size; S(y i ) is the index set, which represents the same index set as the i-th image (i∈[1, B]) and is denoted as S(y i )={p∈[1, B]:y p =y i}, where y i contains the positive sample image index p with the same ID. 7.The fish individual recognition method based on the contrastive language-image pre-trained model according to claim 6, wherein, The total loss L of the first stage training stage1 The function is composed of an image-text contrast loss function L t2i and a text-image contrast loss function L i2t , specifically: L stage1 = L t2i + L i2t .

8. The fish individual recognition method based on a contrastive language-image pre-trained model according to any one of claims 1-2, 4-5, 7, characterized in that, The total loss function L of the second stage training stage2 The total loss function L of the second stage training is composed of the cross-entropy loss function, the triplet loss function and the image-text contrast loss function in the second stage, specifically: wherein L ID is a cross-entropy loss function; L Triplet is a triplet loss function; is an image-text contrastive loss function; λ and β are weight coefficients. 9.The fish individual recognition method based on the contrastive language-image pre-trained model according to claim 8, wherein, The cross-entropy loss function L ID The calculation formula is: where, P (k | x) denotes the target distribution of the ID of the kth fish; ∈ is a hyperparameter to encourage the model to maintain a low confidence during training; p k P (k | x) denotes the predicted score of class k; N is the total number of classes; k denotes the index of the individual class of the fish. 10.The fish individual recognition method based on the contrastive language-image pre-trained model according to claim 8, wherein, The triplet loss function L Triplet The calculation formula is: wherein, and respectively represent the feature distance between positive sample pairs and negative sample pairs; denotes a selected anchor sample; denotes a positive sample; denotes a negative sample; a is the triplet loss function L Triplet ; p represents the index of positive samples; n represents the index of negative samples; f represents the layer where the feature vector is located; a represents the anchor sample; d represents the distance function; V represents the feature vector. 11.The fish individual recognition method based on the contrastive language-image pre-trained model according to claim 8, characterized in that, The image-text contrast loss function in the second stage The calculation formula is: where, represents the target distribution of the ID of the kth fish; ∈ is a hyperparameter used to encourage the model to maintain a low confidence during training; N is the total number of categories; CE i2t represents the cross-entropy loss from image to text; represents the image feature vector of the ith fish sample; represents the feature vector of the text description corresponding to the ith sample; τ represents a temperature parameter used to adjust the smoothness of the similarity distribution and control the attention weight of the model to difficult samples; y j represents the text description of the jth fish category.

12. The fish individual recognition method based on a contrastive language-image pre-trained model according to any one of claims 1-2, 4-5, 7, 9-11, characterized in that, The model testing is realized by calculating the Euclidean distance between the feature vector of the test image and the feature vector in the training set; the specific steps are as follows: Input the test image into the trained model to extract its feature vector; Calculate the Euclidean distance between the feature vector and the feature vector of all images in the training set; by calculating the Euclidean distance between the feature vector of the test image and the feature vector of each image in the training set, a distance value can be obtained; According to the distance value, sort the images in the training set, and select the image with the smallest distance value as the matching result.

Images