Wild animal long-tail data enhancement method combining prior knowledge and diffusion model

Abstract

The invention relates to the technical field of image enhancement, in particular to a wild animal long-tail data enhancement method combining prior knowledge and a diffusion model. The method comprises the following steps: S1, constructing a wild animal recognition model, and introducing extra priori knowledge in combination with a large-scale language model to obtain rich and effective text description of environment characterization and action characterization; s2, using a potential diffusion model as a basic generator, and integrating the text description generated in the S1 to improve the quality and diversity of the generated image; and S3, adopting a contrast language-image pre-training model to screen negative samples so as to enhance long-tail data of the wild animals. According to the method, the influence of long-tail data of the wild animals on model training is solved, the tail category recognition performance of the model is enhanced, and the recognition capability of the wild animal model is further improved.

Description

Technical Field

[0001] The invention relates to the technical field of image recognition, and in particular to a method for enhancing long-tail data of wild animals by combining prior knowledge and a diffusion model. Background Art

[0002] In the process of monitoring wild animals in an open environment, it is easy to be affected by factors such as the number of animals, living habits and species characteristics. It is difficult for researchers to obtain data of all species comprehensively and evenly during the monitoring process, that is, the obtained data has a long-tail distribution phenomenon. The long-tail distribution of data has a huge impact on the model. Because the number of samples in the tail category of long-tail data is small, the model tends to be biased towards the head and middle categories with a larger number during the training process, and the tail category cannot be effectively learned, resulting in the model being unable to fully learn the characteristics of the tail category, affecting the model's recognition ability and generalization performance. In order to solve the above problems, the present invention proposes a method for enhancing long-tail data of wild animals by combining prior knowledge and diffusion model. Summary of the invention

[0003] The purpose of the present invention is to propose a wildlife long-tail data enhancement method that combines prior knowledge and diffusion model to solve the long-tail problem of data sets caused by the morphology, number of species and living habits of wild animals.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions:

[0005] The wildlife long-tail data enhancement method that combines prior knowledge and diffusion model includes the following:

[0006] S1. Build a wildlife recognition model and introduce additional prior knowledge in combination with a large-scale language model to obtain rich and effective text descriptions of environment representation and action representation;

[0007] S2, using the latent diffusion model as the basic generator, integrating the text descriptions generated in S1 to improve the quality and diversity of generated images;

[0008] S3. Use contrastive language-image pre-training model to filter negative samples to enhance wildlife long-tail data.

[0009] Preferably, the S1 specifically includes the following contents:

[0010] S1.1. Build a wildlife recognition model, combine it with the large-scale language model to introduce additional prior knowledge, set "where" and "what" as text prompts to restrict the description of the large-scale language model. Specifically, define the full prompt as "Where is the [label]" to obtain various surrounding text descriptions and enrich the feature space of environmental factors.

[0011] S1.2, collect wildlife action information related to the class label, i.e., “What is the [label] doing”;

[0012] S1.3, combine the label and text description hint of the tail category of the kth category;

[0013] S1.4. Through the above prompt settings, the wildlife recognition model obtains rich and effective text descriptions of environment representation and action representation from the large-scale language model respectively.

[0014] Preferably, based on the operation in S2:

[0015] In the forward diffusion process of the diffusion model, the latent diffusion model starts from the feature representation z0 sampled from the true representation distribution q(z), and gradually introduces Gaussian noise in a total of T steps to z0, obtaining a series of variables z1, z2, …, z t ; In the whole forward diffusion process, z t Only with the previous moment data z t-1 So z t and z t-1 The relationship is expressed as:

[0016]

[0017] Where N represents the Gaussian distribution of mean and covariance; β t represents the mean and variance of the additional noise, and satisfies β1<β2<…<β t ; I represents the identity matrix.

[0018] The latent diffusion model uses a reparameterization technique to derive z directly from z0 t In order to speed up the calculation and eliminate the iteration process, let α t =1-β t and Then t The expression is:

[0019]

[0020] Among them, ∈ t Represented as z t The introduced noise, ∈ t ~N(0,I) represents∈ t is sampled from a normal Gaussian distribution, z t The relationship between and z0 is expressed as:

[0021]

[0022] Preferably, based on the operation in S2:

[0023] The reverse generation process in the diffusion model recovers the image representation from Gaussian noise and follows a Markov chain process, that is, the generation of each step depends only on the state of the previous step, but not on the earlier state; the noise amplitude introduced in each step of the forward diffusion process is small, and the diffusion model uses the Gaussian transformation p parameterized by the neural network θ θ (z t-1 |z t ) from z t Reconstruction t-1 , its formula is expressed as:

[0024]

[0025] Among them, μ θ (z t ,t) and denote the predicted mean and covariance of time step t respectively;

[0026] According to the Bayesian principle, the quality of image synthesis is improved. θ (z t ,t) is defined as:

[0027]

[0028] The diffusion model introduces a specific text encoder in the inverse process to control the generation process of image representation, denoted as τ θ , maps h to the intermediate representation τ θ (h)∈R M×dτ ; Subsequently, the text representation is introduced into the backbone network U-Net of the denoising diffusion probability model through the combination of ResNet and cross attention. The formula of this process is expressed as:

[0029]

[0030] in, Represents the intermediate representation of U-Net; represents a learnable projection matrix;

[0031] When image-based and text-based representations are involved in the image generation process, the model is further optimized based on the input of images and text.

[0032] Preferably, S3 specifically includes the following contents:

[0033] In the negative sample screening process, we first define “A photo of {label}” as the class text description; then, we use the text encoder to encode the text descriptions of the K classes into feature vectors, which are represented as: Ti (1,2,…,K); In addition, the generated samples are encoded into vectors using the visual encoder The model evaluates the correspondence between the generated sample and the text by calculating the cosine similarity λ, which is calculated as follows:

[0034]

[0035] The mechanism for negative sample screening uses cosine similarity as a metric to evaluate the semantic similarity between text descriptions and image samples. The higher the cosine similarity, the greater the possibility that the neural network will associate the generated image with the provided text description. A threshold of cosine similarity α is established, and samples with a similarity lower than α are defined as negative samples, which are then filtered out to ensure strict screening of the generated samples, thereby enhancing the long-tail data of wild animals.

[0036] Compared with the prior art, the present invention provides a method for enhancing long-tail data of wild animals by combining prior knowledge and diffusion model, which has the following beneficial effects:

[0037] The present invention helps to accurately control the generated images using text and image representations, improves the quality and diversity of the generated images, thereby improving the classification and generalization performance of the classifier in the classification task and increasing the generalization ability. The present invention solves the impact of long-tail data of wild animals on model training, generates more effective wild animal images, enhances the recognition performance of the model for tail categories, and further improves the recognition ability of the wild animal model. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a method for enhancing long-tail data of wild animals combining prior knowledge and diffusion model according to the present invention;

[0039] Figure 2 The long-tail data image enhancement path map mentioned in Embodiment 1 of the present invention;

[0040] Figure 3 This is a comparison diagram of the generation effects mentioned in Example 3 of the present invention. DETAILED DESCRIPTION

[0041] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0043] Embodiment 1:

[0044] See also Figure 1 ,,The present invention proposes a wildlife long-tail data enhancement method combining prior knowledge and diffusion model, comprising the following steps:

[0045] S1. Combine the large-scale language model GPT-3.5 to introduce additional prior knowledge to obtain rich and effective text descriptions of environment representation and action representation, including the following:

[0046] Set "where" and "what" as text prompts to restrict the description of GPT-3.5. Specifically, the complete prompt is defined as "Where is the [label] (label representing the tail category)", so as to obtain various surrounding text descriptions and enrich the feature space of environmental factors. Correspondingly, collect wildlife action information related to the class label, that is, "What is the [label] doing". Next, combine the label and text description prompt of the tail category of the kth category. Through the above prompt settings, the model obtains rich and effective text descriptions of environmental representation and action representation from GPT-3.5 respectively;

[0047] S2. Using the latent diffusion model as the basic generator, we integrate the generated text descriptions to improve the quality and diversity of generated images, including the following:

[0048] In the forward diffusion process of the diffusion model, the latent diffusion model starts from the feature representation z0 sampled from the true representation distribution q(z), and gradually introduces Gaussian noise in a total of T steps to z0, resulting in a series of variables z0, z0, …, z T In the whole forward diffusion process, z t Only with the previous moment data z t-1 Therefore, z t and z t-1 The relationship can be expressed as:

[0049]

[0050] Among them, β t represents the mean and variance of the additional noise and satisfies β1<β2<…<β t In this process, the model learns how to preserve the structural information of the data as the noise gradually increases, and restores the original data in the subsequent denoising process.

[0051] In order to speed up the calculation and eliminate the iteration process, the latent diffusion model uses the reparameterization technique to directly derive z from z0. t . Let α t =1-β tand Then t The expression is:

[0052]

[0053] Among them, ∈ t Represented as z t The introduced noise, ∈ t ~N(0,I) represents∈ t is sampled from a normal Gaussian distribution. t The relationship between and z0 is expressed as:

[0054]

[0055] At the same time, in step S2, the reverse generation process in the diffusion model recovers the image representation from the Gaussian noise and follows the Markov chain process, that is, the generation of each step depends only on the state of the previous step, but not on the earlier state. Considering that the noise amplitude introduced in each step of the forward diffusion process is small, the model uses the Gaussian transformation p parameterized by the neural network θ θ (z t-1 |z t ) from z t Reconstruction t-1 The formula is:

[0056]

[0057] Among them, μ θ (z t ,t) and Represent the predicted mean and covariance of time step t respectively. In order to improve the synthesis quality, according to the Bayesian principle, μ θ (z t ,t) can be defined as:

[0058]

[0059] In order to better control the generation process of image representation, the diffusion model first introduces a specific text encoder in the inverse process, denoted as τ θ , maps h to the intermediate representation τ θ (h)∈R M×dτ . Subsequently, the text representation is introduced into the backbone network U-Net of the denoising diffusion probability model through a combination of ResNet and cross attention. The formula for this process is:

[0060]

[0061] in, represents the intermediate representation of U-Net, represents a learnable projection matrix.

[0062] When image-based and text-based representations are involved in the image generation process, the model is further optimized based on the input of images and text.

[0063] S3. Use the contrastive language-image pre-training model to screen negative and invalid samples to enhance the long-tail data of wild animals, including the following:

[0064] In the negative sample screening process, we first define “A photo of {label}” as the class text description. Then, we use the text encoder to encode the text descriptions of K classes into feature vectors, which are represented as: T i (1,2,…,K). In addition, the generated samples are encoded into vectors using the visual encoder The model evaluates the correspondence between the generated sample and the text by calculating the cosine similarity λ, which is calculated as follows:

[0065]

[0066] The mechanism for negative sample screening uses cosine similarity as a metric to evaluate the semantic similarity between text descriptions and image samples. The higher the cosine similarity, the greater the possibility that the neural network will associate the generated image with the provided text description. A threshold of cosine similarity α is established, and samples with a similarity lower than α are defined as negative samples, which are then filtered out to ensure strict screening of the generated samples, thereby enhancing the long-tail data of wild animals.

[0067] See also Figure 2 , Figure 2 is a long-tail data image enhancement path map according to an embodiment of the present invention; Figure 2 As shown, the long-tail data image enhancement step of the embodiment of the present invention includes the following steps:

[0068] Step 1, the label description generation stage, first introduces the text prior knowledge required by GPT-3.5 to generate tail categories to enrich its feature space.

[0069] Step 2, the wildlife image generation stage, combines the prior knowledge obtained in step S1 with the original samples, and uses the latent diffusion model (LDM) to generate new samples to expand the tail categories. However, this process will generate both positive and negative samples.

[0070] In step 3, the negative sample screening stage, in order to mitigate the adverse effects of negative samples, a discriminator mechanism based on the Contrast Language-image Pre-training (CLIP) model is proposed to screen out the generated negative samples.

[0071] Embodiment 2:

[0072] In order to verify the effectiveness of the algorithm proposed in this paper, a comparative experiment of different long-tail methods was conducted on the long-tail dataset of wildlife images. The methods involved in the comparison are as follows:

[0073] Baseline method: Only ResNet50 is used for classification tasks on the dataset. The dataset uses the original dataset without any data augmentation operations.

[0074] Focal loss: In the traditional cross entropy loss function, all samples are treated equally, which will lead to the model being more inclined to predict the category with a larger proportion in the case of category imbalance, while the prediction effect on the minority category is poor. Based on this, this algorithm controls the weight of each category on the basis of classification loss to make the model pay more attention to the classification of the tail category, so as to solve the problem of imbalanced number of samples in each category of the data set.

[0075] LDAM: Since there are fewer samples in the tail category, the model cannot learn the feature information of the tail category, and the deployed model is often large, so overfitting the tail category can enhance the model's recognition ability for the tail category. Based on this, the label distribution-aware margin (LDAM) loss is proposed, which aims to minimize the margin-based generalization boundary. During the training process, the main idea of ​​LDAM is to introduce a learnable boundary for each category in the loss function and dynamically adjust these boundaries according to the distribution of the category to balance the model's attention to each category, thereby improving the model's performance on the problem of imbalanced categories.

[0076] Over-sampling: This algorithm resamples the tail category, rotates the sampled data, randomly flips it horizontally, and samples the tail category samples multiple times to achieve the effect of expanding the tail category samples. The model can better learn the characteristics of the tail category, thereby improving its classification performance.

[0077] Manifold Mixup: This algorithm is based on Mixup, which mixes the input data and the intermediate hidden layer output to achieve the purpose of data expansion. At the same time, the mixed output of the intermediate hidden layer can smooth the decision boundary, expand the low confidence space, and flatten the value of the hidden layer output. Manifold Mixup further generalizes the operation to features. Since features have higher-order semantic information, interpolating in their dimensions may produce more meaningful samples. This method helps neural networks form smoother decision boundaries on multi-layer vector representations, allowing the model to use fewer variance directions to learn category representations during training.

[0078] CycleGAN: CycleGAN uses an adversarial loss function to train the generator network so that the generated images can realistically disguise as images of the target domain. This loss function consists of two parts: the generator loss and the discriminator loss. The generator loss aims to encourage the generated images to deceive the discriminator so that it cannot accurately distinguish the generated images from the real target domain images. The discriminator loss aims to help the discriminator correctly distinguish the generated images from the real target domain images. Secondly, the method introduces the cycle consistency loss. The purpose of this loss function is to ensure that semantic consistency is maintained during the image conversion process. Finally, CycleGAN converts images from one domain to images in another domain to expand the tail category data, thereby improving the classification performance of the model.

[0079] Table 1 shows the recognition performance of the five methods on the long-tail dataset of wild animals. From the recognition results of the tail category, different methods have improved the recognition accuracy of the tail category. Among them, the data enhancement method proposed in the present invention achieved the highest recognition accuracy of 46.2% on this indicator, which is 28.8% higher than the classification accuracy of the model trained on the original dataset without data enhancement, and 5.7% higher than CycleGAN. This shows that the data enhancement method of combining prior knowledge and diffusion model proposed in the present invention effectively improves the recognition accuracy of the tail category. Overall, the data enhancement method proposed in this chapter improves the overall recognition accuracy to 71.9%, which is 1.9% higher than the baseline network, verifying that the method in this chapter can effectively enhance the overall recognition performance of the model and balance the recognition capabilities of the head, middle and tail categories.

[0080] Table 1.1 Accuracy of different long-tail methods on the long-tail dataset of wildlife images (%)

[0081]

[0082] See also Figure 3By comparing the generation effects of CycleGAN and the method of the present invention, it can be seen that the image generated by the present invention has improved image quality compared to the original image, and compared with the CycleGAN generation method, the image generated by the present invention has more wildlife feature representations. By introducing prior knowledge through GPT-3.5, the feature space of the generated image can be enhanced, and a relatively rich wildlife feature representation can be obtained, which is beneficial to improving the generalization performance of the model. At the same time, the method of the present invention generates wildlife images with different backgrounds and different action forms under different prior knowledge, which is conducive to generating richer samples. This shows that the image generation method of combining prior knowledge and diffusion model proposed in the present invention can effectively address the problem of long-tail data and expand the images of the tail category.

[0083] The above are only preferred specific implementation modes of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for enhancing long-tail data of wild animals by combining prior knowledge and diffusion model, characterized in that: Includes the following: S1. Build a wildlife recognition model and introduce additional prior knowledge in combination with a large-scale language model to obtain rich and effective text descriptions of environment representation and action representation; S2, using the latent diffusion model as the basic generator, integrating the text descriptions generated in S1 to improve the quality and diversity of generated images; S3. Use contrastive language-image pre-training model to filter negative samples to enhance wildlife long-tail data.

2. The method for enhancing long-tail data of wild animals by combining prior knowledge and diffusion model according to claim 1, characterized in that: The S1 specifically includes the following contents: S1.

1. Build a wildlife recognition model, introduce additional prior knowledge in combination with the large-scale language model, set "where" and "what" as text prompts to restrict the description of the large-scale language model. Specifically, define the full prompt as "Where is the [label]" to obtain various surrounding text descriptions and enrich the feature space of environmental factors; S1.2, collect wildlife action information related to the class label, i.e., "What is the [label] doing"; S1.3, combine the label and text description hint of the tail category of the kth category; S1.

4. Through the above prompt settings, the wildlife recognition model obtains rich and effective text descriptions of environment representation and action representation from the large-scale language model respectively.

3. The method for enhancing long-tail data of wild animals by combining prior knowledge and diffusion model according to claim 1, characterized in that: Based on the operation described in S2: In the forward diffusion process of the diffusion model, the latent diffusion model starts from the feature representation z0 sampled from the true representation distribution q(z), and gradually introduces Gaussian noise in a total of T steps to z0, obtaining a series of variables z1, z2, …, z t ; During the entire forward diffusion process, z t Only with the previous moment data z t-1 Therefore, z t and z t-1 The relationship is expressed as: Where N represents the Gaussian distribution of mean and covariance; β t represents the mean and variance of the additional noise, and satisfies β1<β2<…<β t ; I represents the identity matrix. The latent diffusion model uses a reparameterization technique to derive z directly from z0 t In order to speed up the calculation and eliminate the iteration process, let α t =1-β t and Then t The expression is: Among them, ∈ t Represented as z t The introduced noise, ∈ t ~N(0,I) represents∈ t is sampled from a normal Gaussian distribution, z t The relationship between and z0 is expressed as:

4. The method for enhancing long-tail data of wild animals by combining prior knowledge and diffusion model according to claim 1, characterized in that: Based on the operation described in S2: The reverse generation process in the diffusion model recovers the image representation from Gaussian noise and follows a Markov chain process, that is, the generation of each step depends only on the state of the previous step, but not on the earlier state; the noise amplitude introduced in each step of the forward diffusion process is small, and the diffusion model uses the Gaussian transformation p parameterized by the neural network θ θ (z t-1 |z t ) from z t Reconstruction t-1 , its formula is expressed as: Among them, μ θ (z t ,t) and denote the predicted mean and covariance of time step t respectively; According to the Bayesian principle, the quality of image synthesis is improved. θ (z t ,t) is defined as: The diffusion model introduces a specific text encoder in the inverse process to control the generation process of image representation, denoted as τ θ , maps h to the intermediate representation τ θ (h)∈R M×dτ ; Subsequently, the text representation is introduced into the backbone network U-Net of the denoising diffusion probability model through the combination of ResNet and cross attention. The formula of this process is expressed as: in, Represents the intermediate representation of U-Net; represents a learnable projection matrix; When image-based and text-based representations are involved in the image generation process, the model is further optimized based on the input of images and text.

5. The method for enhancing long-tail data of wild animals by combining prior knowledge and diffusion model according to claim 1, characterized in that: The S3 specifically includes the following contents: In the negative sample screening process, we first define "A photo of {label}" as the class text description; then, we use the text encoder to encode the text descriptions of K classes into feature vectors, which are represented as: T i (1,2,…,K); In addition, the generated samples are encoded into vectors using the visual encoder The model evaluates the correspondence between the generated sample and the text by calculating the cosine similarity λ, which is calculated as follows: The mechanism for negative sample screening uses cosine similarity as a metric to evaluate the semantic similarity between text descriptions and image samples. The higher the cosine similarity, the greater the possibility that the neural network will associate the generated image with the provided text description. A threshold of cosine similarity α is established, and samples with a similarity lower than α are defined as negative samples, which are then filtered out to ensure strict screening of the generated samples, thereby enhancing the long-tail data of wild animals.

Images