A small sample fish image individual identification method, system and device

Abstract

The present application relates to a kind of small sample fish image individual identification method, system and equipment, belong to intelligent aquaculture and computer vision technical field, the method includes image acquisition, self-supervised pre-training, shape keeping preprocessing, low-rank self-adaptive fine-tuning, individual identification inference.The present application also provides a kind of small sample fish image individual identification system for running the method, and using the method can realize fish individual identification equipment.The present application is pre-trained by mask autoencoder self-supervised on large-scale unlabeled fish image, so that the model can fully learn general features such as fish contour, texture and posture distribution, still can keep higher recognition accuracy under the condition of small amount of labeled sample, significantly reduce the dependence on artificial labeling.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of intelligent aquaculture and computer vision, and particularly relates to a small sample fish image individual identification method, system and device. BACKGROUND

[0002] With the development of large-scale and intensive aquaculture, fine management and long-term tracking of individual fish become an important link to ensure production efficiency and germplasm improvement. Traditional fish individual identification relies on manual marking (such as punching, cutting fins, implanting tags, etc.) or manual measurement. Such methods not only have invasiveness to the fish body, increasing the operation risk and stress response, but also are difficult to achieve long-term, continuous and high-frequency monitoring in large-scale aquaculture scenarios.

[0003] In recent years, image recognition technology based on deep learning has been introduced into fish individual identification tasks, realizing non-contact identity discrimination by learning fish texture, morphology and spot appearance features. However, existing methods usually require a large number of labeled samples to achieve high recognition accuracy, and in real aquaculture scenarios, it is extremely costly to collect and label a large number of images for each fish one by one, and the model performance under small sample conditions is difficult to guarantee. In addition, complex aquaculture environments can introduce water ripples, reflections, background debris and other disturbances, leading to unstable capture of fish morphological features by the model and insufficient cross-species or cross-scene migration ability.

[0004] Therefore, there is an urgent need for a technical solution that can achieve high-precision fish individual identification under small sample labeling conditions, while having good cross-species adaptability and deployment feasibility, to meet the actual needs of modern smart fisheries and precision aquaculture. SUMMARY

[0005] The present application aims to overcome the defects of existing fish individual identification methods, which rely heavily on large-scale labeled samples, have poor adaptability to complex aquaculture environments, and have insufficient cross-species generalization ability. A small sample fish image individual identification method, system and device based on mask autoencoder and low-rank adaptation are proposed. The present application performs self-supervised pre-training on a large number of unlabeled fish images, combines morphological preservation preprocessing and parameter efficient fine-tuning strategy, significantly reduces the labeling cost, and realizes high-precision identification of fish individuals and cross-species migration.

[0006] To achieve the above purpose, the small sample fish image individual identification method proposed by the present application comprises the following steps:

[0007] Step 1, image acquisition: acquiring multi-view fish images of target fish in aquaculture or experimental scenarios to form an unlabeled fish image set and a corresponding small sample labeled fish individual image set;

[0008] Step 2, self-supervised pre-training: inputting the set of unlabeled fish images into a mask autoencoder framework, randomly blocking and masking the input images, and reconstructing the masked areas based only on the visible image blocks to train the visual encoder in a self-supervised manner to obtain a pre-trained encoder that learns the overall shape and local texture features of the fish body;

[0009] Step 3, shape-preserving preprocessing: performing foreground segmentation on the labeled fish individual images in step 1 to obtain a fish body mask, replacing the original background with a uniform background, and symmetrically padding the fish body area to a square canvas and uniformly scaling the size to eliminate background interference while preserving the overall shape and proportion of the fish body, obtaining preprocessed fish individual images;

[0010] Step 4, low-rank adaptive fine-tuning: introducing a low-rank adaptive structure into the multi-head self-attention module in the pre-trained encoder, performing low-rank decomposition on the weight updates of the attention query matrix and / or numerical matrix, training only the introduced low-rank parameters, and performing partial fine-tuning of part of the high-level encoder layers with a small learning rate, using the small sample set of labeled fish individual images to classify and train the preprocessed fish individual images obtained in step 3, obtaining a small sample fish individual recognition model;

[0011] Step 5, individual recognition inference: performing the same shape-preserving preprocessing on the fish images to be recognized as in step 3, inputting the preprocessed images into the small sample fish individual recognition model in step 4, and outputting the corresponding fish individual identity label.

[0012] Preferably, the reconstruction loss of the mask autoencoder in step 2 is represented as:

[0013] where N is the number of masked image blocks, is the original pixel value of the i-th masked image block, is the corresponding reconstructed pixel value.

[0014] Preferably, in step 3, the fish body is symmetrically expanded and filled in the rectangular region to make the overall image square.

[0015] Preferably, in step 4, the query projection matrix is in the form of low-rank adaptation:

[0016] ;

[0017] where is the original weight obtained by pre-training and kept frozen, To query the low-rank incremental weight of the projection matrix, A and B are trainable low-rank matrices with rank r, and the parameters of A and B are updated only in the fine-tuning stage, so as to complete the adaptation of the fish individual identification task under the condition of controlled parameter quantity.

[0018] Preferably, in step 4, the low-rank adaptive fine-tuning only acts on the first not more than 6 encoding layers of the encoder network, and the parameters of the remaining encoding layers except the first not more than 6 encoding layers remain frozen and are not updated.

[0019] The application also provides a small sample fish image individual identification system, which comprises an image acquisition module, a self-supervised pre-training module, a shape-preserving preprocessing module, a low-rank adaptive fine-tuning module and an individual identification module.

[0020] The image acquisition module is used to arrange cameras above or on the side of the breeding pond to periodically acquire fish images or video frames; obtain multi-view fish images of target fish in a breeding scene or an experimental scene to form an unlabeled fish image set and a small sample labeled fish individual image set.

[0021] The self-supervised pre-training module is deployed on a server or cloud to perform mask autoencoder pre-training on the unlabeled fish image set; and step 2 in the small sample fish image individual identification method is run.

[0022] The shape-preserving preprocessing module performs foreground segmentation, background replacement and symmetric padding on the acquired images; and step 3 in the small sample fish image individual identification method is run.

[0023] The low-rank adaptive fine-tuning module updates the model parameters using the small sample labeled fish individual image; and step 4 in the small sample fish image individual identification method is run.

[0024] The individual identification module performs forward inference on real-time images and outputs individual identity information.

[0025] The above modules can be distributedly implemented by the same server or multiple edge computing devices, wherein the self-supervised pre-training and model fine-tuning are suitable for being executed on a server or cloud with strong computing power, and the individual identification inference can be deployed on an edge computing terminal close to the breeding site to reduce data transmission overhead and response time delay.

[0026] The application also provides a fish individual identification device, which comprises at least one image acquisition unit, a processor and a memory. The image acquisition unit is used to acquire fish images or video frames in a breeding water body; the memory stores a computer program; and the processor, when executing the computer program, implements each step in the small sample fish image individual identification method and outputs a fish individual identification result.

[0027] Compared with the prior art, the present application has the beneficial effects of:

[0028] 1、The present application can make the model fully learn the general features such as fish body contour, texture and posture distribution by performing mask autoencoder self-supervised pre-training on large-scale unlabeled fish images, and can still maintain high recognition accuracy under the condition of a small number of labeled samples, thereby significantly reducing the dependence on manual labeling.

[0029] 2、The morphological preservation preprocessing strategy proposed in the present application can eliminate the interference of complex breeding background, avoid the loss of fish head or tail information caused by cropping or non-proportional scaling, and improve the robustness of the model to the overall morphology and local texture of the fish body.

[0030] 3、The present application adopts a low-rank adaptive fine-tuning structure, which only needs to train a small number of new parameters to complete the rapid adaptation to specific species or scenes, and balances the parameter efficiency and recognition performance, and is conducive to deployment on edge devices with limited computing power.

[0031] 4、Relying on self-supervised pre-training and parameter efficient fine-tuning, the present application can still achieve effective identification on new species that have not been seen before in a zero-shot or few-shot manner, has good cross-species migration ability, and is suitable for individual identification needs of various farmed fish. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart and structural schematic diagram of the small sample fish image individual identification method in the embodiments of the present application. DETAILED DESCRIPTION

[0033] The present application will be further described in detail below in conjunction with the drawings and specific embodiments, and it should be understood that these embodiments are only used to illustrate the present application, but not to limit the present application.

[0034] Embodiment 1: Small sample fish image individual identification method

[0035] The present embodiment provides a small sample fish image individual identification method, the flowchart of which is shown in Figure 1 , including image acquisition, self-supervised pre-training, morphological preservation preprocessing, low-rank adaptive fine-tuning and individual identification inference.

[0036] In the image acquisition stage, multi-view images of fish in the breeding water body are obtained by a camera to form an unlabeled fish image set and a labeled fish individual image set. The unlabeled image set covers multiple fish species and multiple environmental conditions, and is used for self-supervised pre-training; the labeled image set is mainly for specific breeding species, and each fish contains at least several images and is labeled with individual number.

[0037] In the self-supervised pre-training stage, the unlabeled fish images are divided into a plurality of image blocks of fixed size, a part of the image blocks are randomly occluded, only the visible image blocks are input into the visual encoder for feature extraction, and then the occluded image blocks are reconstructed by the decoder. In this process, by minimizing the reconstruction loss, the encoder gradually learns a general feature representation that discriminates the structure and texture of the fish body.

[0038] As a specific implementation, the reconstruction loss of the mask autoencoder is represented as:

[0039] where N is the number of occluded image blocks, is the original pixel value of the i-th occluded image block, is the corresponding reconstructed pixel value.

[0040] In the shape-preserving preprocessing stage, first, the segmentation network or the hintable segmentation model is used to perform foreground segmentation on the original image to obtain a mask of the fish body region. Then, the regions outside the fish body are replaced with a uniform background color to reduce environmental noise. Then, the fish body bounding rectangle is symmetrically expanded and filled to make the image square, preferably a square, and uniformly scaled to a preset input size, so that the fish body shape is not stretched and deformed while the input is standardized.

[0041] In the low-rank adaptive fine-tuning stage, the pre-trained visual encoder is used as a feature extraction backbone network, and a low-rank decomposition branch is introduced into the query matrix and / or the numerical matrix in some self-attention modules. Only the newly added low-rank parameters are trained, and the original pre-trained weights are mostly frozen. At the same time, the weights of some high-level encoder layers are updated with a small learning rate. As an optimization, low-rank adaptive fine-tuning only acts on the first not more than 6 encoding layers of the encoder network, and the parameters of the remaining encoding layers except the first not more than 6 encoding layers are kept frozen and not updated. The ablation experiment shows that when the low-rank adaptive fine-tuning is further extended to all encoding layers, the feature extraction and discrimination ability of the model does not increase but decreases, and the overall performance decreases significantly, with the accuracy decreasing from 96.9% to 28.0%, a decrease of 68.9%. Through classification training on labeled fish individual images, a small sample fish individual recognition model suitable for the current species and scene is obtained.

[0042] As a preferred implementation, the query projection matrix in the self-attention layer is The low-rank adaptive form is adopted:

[0043] ;

[0044] where is the original weight obtained by pre-training and kept frozen, To query the low-rank incremental weight of the projection matrix, A and B are trainable low-rank matrices with rank r, and the parameters of A and B are updated only in the fine-tuning stage, so as to complete the adaptation of the fish individual identification task under the condition of controlled parameter quantity.

[0045] In the individual identification inference stage, the morphological maintenance preprocessing consistent with the training stage is performed on the real-time collected fish images, the preprocessed images are input into the small sample fish individual identification model, the predicted individual number is output, and the individual growth, feeding and behavior information can be counted according to the identification result.

[0046] In an application embodiment, in order to verify the identification ability and accuracy of the specific individual of the method, a plurality of target fish in the culture pond are selected, and an electronic tag (such as RFID or PIT tag) with a unique code is implanted in each fish body as a physical marker of the real identity of the individual. Then, the batch of individuals are photographed multiple times, 5 pictures for each individual, the obtained fish images are processed according to the morphological maintenance preprocessing process described above in the embodiment, and are input into the small sample fish individual identification model obtained by pre-training the masked autoencoder and low-rank adaptive fine-tuning to perform inference, and the corresponding individual number is output. By comparing the individual number output by the model with the real identity decoded from the electronic tag one by one, the number of correctly identified images and the total number of images are counted, and the individual identification accuracy is calculated. The test results show that the identification accuracy of the target individual implanted with the electronic tag by the method described in the embodiment is about 98%, and the stable and reliable identification of the single fish can be realized in the actual farming scene and small sample situation.

[0047] In a cross-species migration application embodiment, the method is first used to pre-train a general visual encoder capable of representing the overall morphology and surface texture of the fish body on an unlabeled fish image set containing multiple cultured fish species (such as golden pompano, east star barb, etc.) by using a masked autoencoder for self-supervised pre-training. Then, only a small amount of labeled individual images of one target cultured species (such as golden pompano) are used to fine-tune the pre-trained encoder, and a basic model for golden pompano individual identification is trained.

[0048] On this basis, the model is applied to another new cultured species (such as large yellow croaker or other marine fish) which does not participate in the pre-training and fine-tuning stages. For the new species, without changing the main structure of the model, only a small amount of high-definition off-water photos of each individual are collected to construct an "individual template library", and the corresponding feature vectors are extracted; during inference, the image to be identified is sent to the same encoder after the shape maintaining preprocessing of the application, the features are extracted and similarity matching is performed with the features of each individual in the template library, and the template individual with the highest similarity is taken as the recognition result. In the whole process, the backbone network does not need to be retrained, and only a small amount of template samples are needed to distinguish different individuals in the new species, which belongs to cross-species recognition under zero-shot or few-shot conditions.

[0049] Embodiment 2: Small sample fish image individual recognition system

[0050] The embodiment provides a small sample fish image individual recognition system, which comprises an image acquisition module, a self-supervised pre-training module, a shape maintaining preprocessing module, a low-rank adaptive fine-tuning module and an individual recognition module.

[0051] The image acquisition module is used for arranging cameras above or on the side of the culture pond to periodically collect fish images or video frames; the self-supervised pre-training module is deployed on a server or in the cloud to perform mask autoencoder pre-training on an unannotated fish image set; the shape maintaining preprocessing module performs foreground segmentation, background replacement and symmetric padding on the collected images; the low-rank adaptive fine-tuning module updates the model parameters by using small sample annotated fish individual images; and the individual recognition module performs forward inference on real-time images and outputs individual identity information.

[0052] The above modules can be distributedly implemented by the same server or multiple edge computing devices, wherein the self-supervised pre-training and model fine-tuning are suitable for being executed on a server or in the cloud with strong computing power, and the individual recognition inference can be deployed on an edge computing terminal close to the culture site to reduce data transmission overhead and response time delay.

[0053] Embodiment 3: Fish individual recognition device

[0054] The embodiment provides a fish individual recognition device, which comprises at least one image acquisition unit, a processor and a memory. The image acquisition unit is used for collecting fish images or video frames in a culture water body; the memory stores a computer program; and the processor, when executing the computer program, implements each step of the small sample fish image individual recognition method of embodiment 1 and outputs a fish individual recognition result.

[0055] Those skilled in the art can understand that various modifications or equivalent replacements made to the above embodiments without departing from the spirit and essence of the application shall fall within the protection scope of the application.

Claims

1. A method for individual fish image recognition in a small sample, characterized in that, The method includes the following steps: Step 1, Image Acquisition: Acquire multi-view images of the target fish in aquaculture or experimental settings to form an unlabeled fish image set and a corresponding small sample of labeled individual fish images; Step 2, Self-supervised pre-training: Input the unlabeled fish image set into the mask autoencoder framework, randomly block the input image, and reconstruct the occluded area based only on the visible image blocks. Train the visual encoder in a self-supervised manner to obtain a pre-trained encoder that learns the overall morphology and local texture features of the fish. Step 3, Shape Preservation Preprocessing: Perform foreground segmentation on the labeled fish individual images described in Step 1 to obtain fish body masks, replace the original background with a uniform background, symmetrically fill the area where the fish body is located onto a square canvas and uniformly scale the size to obtain the preprocessed fish individual images. Step 4, Low-rank adaptive fine-tuning: A low-rank adaptive structure is introduced into the multi-head self-attention module of the pre-trained encoder described in Step 2. The weight updates of the attention query matrix and / or numerical matrix are decomposed into low-rank values. Only the introduced low-rank parameters are trained, and some high-level encoder layers are partially fine-tuned with a small learning rate. The preprocessed fish individual images obtained in Step 3 are then used to perform classification training using the small-sample labeled fish individual image set to obtain the small-sample fish individual recognition model; the query matrix in the self-attention layer... Adopting a low-rank adaptive form: ； in To obtain and retain the original weights during pre-training, To query the low-rank incremental weights of the matrix, A and B are trainable low-rank matrices of rank r; low-rank adaptive fine-tuning is applied only to the first 6 coding layers of the encoder network, and the parameters of the remaining coding layers other than the first 6 coding layers remain frozen and not updated. Step 5, Individual Recognition Reasoning: Perform the same morphological preservation preprocessing as in Step 3 on the image of the fish to be identified, input the preprocessed image into the small sample fish individual recognition model described in Step 4, and output the corresponding fish individual identity identifier.

2. The method for individual fish image recognition in a small sample according to claim 1, characterized in that, The reconstruction loss of the mask autoencoder in step 2 is expressed as: Where N is the number of occluded image patches. Let be the original pixel value of the i-th occluded image patch. This corresponds to the reconstructed pixel value.

3. The method for individual fish image recognition in a small sample according to claim 1, characterized in that, In step 3, the circumscribed rectangular region of the fish is symmetrically expanded and filled to make the overall image a square.

4. A small sample fish image individual recognition system, characterized in that, The system includes an image acquisition module, a self-supervised pre-training module, a morphology preservation preprocessing module, a low-rank adaptive fine-tuning module, and an individual recognition module. The image acquisition module is used to deploy cameras above or to the side of the aquaculture pond to periodically acquire images or video frames of fish; to obtain multi-view images of target fish in aquaculture or experimental settings, forming an unlabeled fish image set and a corresponding small sample labeled fish individual image set; The self-supervised pre-training module is deployed on a server or in the cloud to perform masked autoencoder pre-training on an unlabeled fish image set; and runs step 2 of the small sample fish image individual recognition method according to any one of claims 1-3. The morphology-preserving preprocessing module performs foreground segmentation, background replacement, and symmetry filling on the acquired image; and runs step 3 of the method for individual recognition of small sample fish images as described in any one of claims 1-3. The low-rank adaptive fine-tuning module updates the model parameters using small sample labeled individual fish images; and runs step 4 of the small sample fish image individual recognition method according to any one of claims 1-3. The individual identification module performs forward inference on real-time images and outputs individual identity information.

5. A fish individual identification device, characterized in that, The device includes at least one image acquisition unit, a processor, and a memory; the image acquisition unit is used to acquire images or video frames of fish in aquaculture water; the memory stores a computer program; when the processor executes the computer program, it implements each step of the method for individual identification of small sample fish images according to any one of claims 1-3, and outputs the individual fish identification result.

Images