A low-resource image management system and method based on multiple similar styles

Abstract

The application discloses a low-resource picture management system and method based on multiple similar styles, and relates to the technical field of picture processing. The low-resource picture management method comprises the following steps: obtaining an input picture set containing pictures of similar styles, extracting main features of pictures of each style in the picture set, and converting the main features into text data; inputting text data of pictures of any style and text data of pictures of other styles, and obtaining text data identical to the pictures of any style and different from the pictures of other styles through training output; performing denoising processing on the extracted text data through a positive text and a negative text training model, and generating pictures conforming to current style description; calculating the similarity between the picture vector representation of the generated pictures and the average representation of the picture set, screening out pictures similar to a specific style, and adding the pictures to the picture set of the style.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a low-resource image management system and method based on multiple similar styles. Background Technology

[0002] The significance of low-resource image augmentation lies in its ability to address the poor performance of traditional neural network models when augmenting images with missing features, thereby improving the augmentation effect and the model's adaptability. In practical applications, image datasets are often limited, while low-resource image augmentation methods can generate more data by augmenting existing images, thus expanding the scale and diversity of the dataset. This can improve the model's accuracy and robustness, making it more practical.

[0003] Traditional neural network models perform poorly when processing images with missing features, especially in low-resource environments. Existing image enhancement methods can only enhance a specific type of image, failing to address the enhancement of multiple images with similar styles, and require a large number of real-world object images as training data, thus limiting their feasibility. Summary of the Invention

[0004] The purpose of this invention is to provide a low-resource image management system and method based on multiple similar styles, so as to solve the problems mentioned in the background art.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for managing low-resource images based on multiple similar styles, the method comprising the following steps:

[0006] Step S100: Obtain the input image set, extract the main features of each image in the image set, and convert the main features into text data;

[0007] Step S200: Input text data of an image of any style and text data of images of other styles. After training, output text data that is the same as the image of any style but different from the images of other styles. Use the output text data as positive text and use the remaining text data in the image of any style as negative text.

[0008] Step S300: Train the model using positive and negative text, denoise the extracted text data, and generate an image that matches the positive text.

[0009] Step S400: Calculate the similarity between the image vector representation of the generated image and the average representation of the image set, filter out images similar to a specific style, and add the images to the image set of the style.

[0010] Furthermore, step S100 includes the following steps:

[0011] Step S101: Input the image into the image encoding module in the first model, slice the image into blocks, linearly map it to the vector space, and extract the image features;

[0012] Step S102: Describe the image features with text based on the extracted image features, and convert the image features into text data.

[0013] Furthermore, the training process in step S200 includes the following steps:

[0014] Step S201: Map the extracted text data to a vector space to obtain text vectors;

[0015] Step S202: Adjust the parameters of the text vector, fix the original training parameters, and concatenate the parameters to be trained before the text vector;

[0016] Step S203: Input text data of any style of image, train it through the GPT architecture in the model, and output positive text; when the output negative text is similar to the positive text, input the negative text into the model to continue training.

[0017] The description enhancement and filtering section can select the most suitable enhancement method based on the characteristics of the target image, while eliminating some unsuitable enhancement methods, thereby further improving the enhancement effect.

[0018] Furthermore, step S300 includes the following steps:

[0019] Step S301: Take the text description of the current style as positive text and the text description with a similar style as negative text; in the text encoder, sample the text description of the image based on the positive text to obtain the corresponding positive text, and sample the text description of the image based on the negative text to obtain the corresponding negative text.

[0020] Step S302: Use a sampler to input the initial pure noise latent variables in the text description of the image into the Unet network; combine the semantic control vector, modify the model architecture according to the received negative text, the text description in the iteration and the time step, adjust the Unet denoising intensity to remove noise in the pure noise latent variables, repeat several times, and continuously inject semantic information into the latent variables to obtain a latent space vector.

[0021] Step S303: Exclude all negative text from the latent space vector, sample the latent space vector that does not conform to negative text using a sampler, enhance the text description of the image based on positive text, and generate the image in the Unet network;

[0022] Step S304: Upscale the low-dimensional space vector obtained from the generator using an image decoder to obtain a complete image.

[0023] Furthermore, step S400 includes the following steps:

[0024] Step S401: Let V be the vector representation of the generated image, and let V be the average vector representation of the set of images with similar styles to the generated image. According to the formula:

[0025]

[0026] in, For vector V and vector The inner product, |V|, For vector V and vector The model field is given, and S represents the similarity between two vectors; the vector representation V of the generated image and the average vector representation of the set of images with similar styles are calculated. The similarity between them;

[0027] Step S402: Set the image set to have a total of j styles, where the similarity between the image of style i and the image of the generated style is S. i According to the formula:

[0028]

[0029] Among them, Score i Let K be the similarity index of the i-th style of images; select the K images with the highest similarity index and add them to the image set.

[0030] Furthermore, to better implement the above method, a low-resource image management system based on multiple similar styles is proposed. The system includes a text conversion module, a feature differentiation module, an image generation module, and a similarity filtering module.

[0031] The text conversion module is used to acquire the input image set, extract the main features of each image in the image set, and convert the main features into text data;

[0032] The feature differentiation module is used to learn the distinguishing features between different styles through the prefix module. It takes a style text description as input as positive text and outputs text that is different from the style text description as negative text. It compares the text data with the positive and negative texts and selects features with high similarity.

[0033] The image generation module uses text descriptions of a certain style as positive text and text descriptions with a similar style as negative text. It trains a model using positive and negative texts, performs noise reduction on the extracted text data, and generates images that conform to the positive text.

[0034] The similarity filtering module is used to calculate the similarity between the image vector representation of the generated image and the average representation of the image set, filter out images similar to a specific style, and add the images to the image set of that style.

[0035] Furthermore, the image generation module includes a text encoding unit, an image information generation unit, and an image decoding unit;

[0036] The text encoding unit is used to sample the text description of the image based on the positive text in the text encoder to obtain the corresponding positive text, and to sample the text description of the image based on the negative text to obtain the corresponding negative text.

[0037] The image information generation unit is used to combine semantic control vectors to dynamically adjust the Unet denoising intensity to remove noise from the pure noise latent variables, and to view the denoised image corresponding to positive text and the denoised image corresponding to negative text through a sampler, compare the differences between the two, and enhance the generated image in the direction of positive text.

[0038] The image decoding unit is used to upscale the low-dimensional space vector obtained from the generator using an image decoder to obtain a complete image.

[0039] Furthermore, the image generation unit includes a Unet network unit and a sampler algorithm unit;

[0040] The Unet network unit is used to perform the image generation process in the Unet network; the sampler algorithm unit is used to control the image generation speed.

[0041] Compared with the prior art, the beneficial effects achieved by the present invention are: (1) It can process images of various similar styles and enhance the effect better; (2) It uses the technical means of word segmentation and screening of descriptive text in order to better extract the features and characteristics of images of specific styles; (3) It adopts the multi-head attention mechanism of cross attention to establish a closer connection between texts and enhance the encoding of negative adjectives so as to better influence the direction of text generation; (4) It uses positive and negative texts to train the model so that it can better distinguish the differences between different styles and generate images that conform to various styles; (5) It effectively solves the problem that traditional neural network models are less effective when enhancing images with missing features, improves the network's ability to adapt to extreme scenes, and thus improves the enhancement effect of target images. Attached Figure Description

[0042] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0043] Figure 1 This is a step-by-step diagram of a low-resource image management method based on multiple similar styles;

[0044] Figure 2 This is a schematic diagram of a low-resource image management system based on multiple similar styles;

[0045] Figure 3 This is a schematic diagram of the negative description penalty module based on the BLIP model. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] Please see Figures 1 to 3 The present invention provides a technical solution: a method for managing low-resource images based on multiple similar styles, the method comprising the following steps:

[0048] Step S100: Obtain the input image set, extract the main features of each image in the image set, and convert the main features into text data;

[0049] Step S100 includes the following steps:

[0050] Step S101: Input the image into the image encoding module in the first model, slice the image into blocks, linearly map it to the vector space, and extract the image features;

[0051] Step S102: Describe the image features with text based on the extracted image features, and convert the image features into text data;

[0052] For a 512×512 pixel image, the vector dimension of its RGB channel representation is (3, 512, 512). First, the image is divided into multiple patches. If it is divided into 16 patches, each patch is represented as (16, 3, 32, 32). Then, a lower-dimensional image representation can be obtained through a linear mapping layer. Various features of each image are analyzed, such as color, shape, size, texture, etc. The obtained features are used to generate corresponding text descriptions to describe the objects, scenes, and people in the image.

[0053] Step S200: Input text data of an image of any style and text data of images of other styles. After training, output text data that is the same as the image of any style but different from the images of other styles. Use the output text data as positive text and use the remaining text data in the image of any style as negative text.

[0054] The training process in step S200 includes the following steps:

[0055] Step S201: Map the extracted text data to a vector space to obtain text vectors;

[0056] Step S202: Adjust the parameters of the text vector, fix the original training parameters, and concatenate the parameters to be trained before the text vector;

[0057] Step S203: Input text data of any style of image, train it through the GPT architecture in the model, and output positive text; when the output negative text is similar to the positive text, input the negative text into the model to continue training.

[0058] Step S300: Train the model using positive and negative text, denoise the extracted text data, and generate an image that matches the positive text.

[0059] For example, by inputting both positive and negative cues into `stablediffuison`, the model will output images that meet the requirements. When there are no negative cues, the diffusion model generates images through unconditional sampling. By specifying cues, negative cues are used for sampling, thus providing more precise control over image generation.

[0060] Step S300 includes the following steps:

[0061] Step S301: Take the text description of the current style as positive text and the text description with a similar style as negative text; in the text encoder, sample the text description of the image based on the positive text to obtain the corresponding positive text, and sample the text description of the image based on the negative text to obtain the corresponding negative text.

[0062] Step S302: Use a sampler to input the initial pure noise latent variables in the text description of the image into the Unet network; combine the semantic control vector, modify the model architecture according to the received negative text, the text description in the iteration and the time step, adjust the Unet denoising intensity to remove noise in the pure noise latent variables, repeat several times, and continuously inject semantic information into the latent variables to obtain a latent space vector.

[0063] Step S303: Exclude all negative text from the latent space vector, sample the latent space vector that does not conform to negative text using a sampler, enhance the text description of the image based on positive text, and generate the image in the Unet network;

[0064] Step S304: Upscale the low-dimensional space vector obtained from the generator using an image decoder to obtain a complete image;

[0065] Step S400: Calculate the similarity between the image vector representation of the generated image and the average representation of the image set, filter out images similar to a specific style, and add the images to the image set of the style;

[0066] Step S400 includes the following steps:

[0067] Step S401: Let V be the vector representation of the generated image, and let V be the average vector representation of the set of images with similar styles to the generated image. According to the formula:

[0068]

[0069] in, For vector V and vector The inner product, |V|, For vector V and vector The model field is given, and S represents the similarity between two vectors; the vector representation V of the generated image and the average vector representation of the set of images with similar styles are calculated. The similarity between them;

[0070] Step S402: Set the image set to have a total of j styles, where the similarity between the image of style i and the image of the generated style is S. i According to the formula:

[0071]

[0072] Among them, Score i Let K be the similarity index of the i-th style of images; select the K images with the highest similarity index and add them to the image set.

[0073] Let the similarity between the first style image and the generated style image be S1 = 0.6, the similarity between the second style image and the generated style image be S2 = 0.21, the similarity between the third style image and the generated style image be S3 = 0.85, and the similarity between the fourth style image and the generated style image be S4 = 0.37.

[0074] therefore,

[0075] If two images with the highest similarity index are selected, then the images of the first and third styles will be added to the image set.

[0076] One example is a low-resource image management system based on multiple similar styles, which includes a text conversion module, a feature differentiation module, an image generation module, and a similarity filtering module.

[0077] The text conversion module is used to acquire the input image set, extract the main features of each image in the image set, and convert the main features into text data;

[0078] The feature differentiation module is used to learn the distinguishing features between different styles through the prefix module. It takes a style text description as input as positive text and outputs text that is different from the style text description as negative text. It compares the text data with the positive and negative texts and selects features with high similarity.

[0079] The image generation module uses text descriptions of a certain style as positive text and text descriptions with a similar style as negative text. It trains a model using positive and negative texts, performs noise reduction on the extracted text data, and generates images that conform to the positive text.

[0080] The similarity filtering module is used to calculate the similarity between the image vector representation of the generated image and the average representation of the image set, filter out images similar to a specific style, and add the images to the image set of that style.

[0081] The image generation module includes a text encoding unit, an image information generation unit, and an image decoding unit.

[0082] The text encoding unit is used to sample the text description of the image based on the positive text in the text encoder to obtain the corresponding positive text, and to sample the text description of the image based on the negative text to obtain the corresponding negative text.

[0083] The image information generation unit is used to combine semantic control vectors to dynamically adjust the Unet denoising intensity to remove noise from the pure noise latent variables, and to view the denoised image corresponding to positive text and the denoised image corresponding to negative text through a sampler, compare the differences between the two, and enhance the generated image in the direction of positive text.

[0084] The image decoding unit is used to upscale the low-dimensional space vector obtained from the generator through the image decoder to obtain a complete image.

[0085] The image generation unit includes a Unet network unit and a sampler algorithm unit.

[0086] The Unet network unit is used to perform the image generation process in the Unet network; the sampler algorithm unit is used to control the image generation speed.

[0087] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0088] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for managing low-resource images based on multiple similar styles, characterized in that: The management method comprises the following steps: Step S100: obtaining an input picture set containing pictures of similar styles, extracting main features of pictures of each style in the picture set, and converting the main features into text data; Step S200: inputting text data of pictures of any style and text data of pictures of other styles, obtaining text data identical to the pictures of any style and different from the pictures of other styles through training output, taking the obtained text data as positive text, and taking remaining text data of the pictures of any style as negative text; Step S300: training a model through the positive text and the negative text, performing denoising processing on the extracted text data, and generating pictures conforming to the positive text; Step S400: calculating similarity between picture vector representation of the generated pictures and average representation of the picture set, screening out pictures similar to a specific style, and adding the pictures to the picture set of the style; The step S300 comprises the following steps: Step S301: sampling text description of a picture according to positive text in a text encoder to obtain corresponding positive text, and sampling the text description of the picture according to negative text to obtain corresponding negative text; Step S302: inputting initial pure noise hidden variables existing in the text description of the picture into a Unet network by using a sampler; combining a semantic control vector, modifying the architecture of the model according to received negative text, text description in iteration and time steps, adjusting Unet denoising strength to remove noise in the pure noise hidden variables, repeating for several times, and continuously injecting semantic information into the hidden variables to obtain a hidden space vector; Step S303: excluding all negative text in the hidden space vector, sampling in the hidden space vector not conforming to the negative text by using the sampler, strengthening the text description of the picture according to the positive text, and generating the picture in the Unet network; Step S304: upgrading the low-dimensional space vector obtained in the generator by using a picture decoder to obtain a complete picture.

2. The low resource image management method based on multiple similar styles according to claim 1, characterized in that: The step S100 comprises the following steps: Step S101: inputting a picture into a set picture encoding module, cutting the picture, linearly mapping the picture to a vector space, and extracting picture features; Step S102: describing the picture features in text according to the extracted picture features, and converting the picture features into text data.

3. The low resource image management method based on multiple similar styles according to claim 2, characterized in that: The training process in the step S200 comprises the following steps: Step S201: mapping the extracted text data to a vector space to obtain text vectors; Step S202: adjusting parameters of the text vectors, fixing original training parameters, and splicing parameters needing to be trained in front of the text vectors; Step S203: inputting text data of pictures of any style, training through a GPT architecture in the model, outputting positive text, and inputting negative text into the model for continuous training when the output negative text is similar to the positive text.

4. The low resource image management method based on multiple similar styles according to claim 1, characterized in that: The step S400 comprises the following steps: Step S401: Set the vector representation of a generated picture as the average vector representation of a picture set with similar style to the picture is According to the formula: ； wherein, is the inner product of the vector and the vector , , is the modulus field of the vector and the vector , is the similarity between two vectors; the similarity between the vector representation of the generated picture and the average vector representation of the picture set with similar style to the picture is calculated . Step S402: set the total number of pictures in the picture set to a style, wherein the similarity between the picture of the first style and the picture generated in the style is According to the formula: ； wherein, is the first similarity index of the picture of the style; and the picture of the style with the highest similarity index is selected to be added to the picture set.

5. A low-resource image management system for use in a low-resource image management method based on multiple similar styles according to any one of claims 1-4, characterized in that: The system comprises a text conversion module, a feature distinguishing module, a picture generation module and a similarity screening module. The text conversion module is used for obtaining an input picture set, extracting main features of each picture in the picture set, and converting the main features into text data. The feature distinguishing module is used for learning distinguishing features between different styles through a prefix module, inputting a style text description as positive text, and outputting different negative text from the style text description; comparing the text data with the positive text and the negative text, and screening features with high similarity. The picture generation module takes a certain style of text description as positive text, and a text description with similar style as negative text; trains a model through the positive text and the negative text, performs denoising processing on the extracted text data, and generates a picture consistent with the positive text. The similarity screening module is used for calculating the similarity between the picture vector representation of the generated picture and the average representation of the picture set, screening pictures similar to a specific style, and adding the pictures to the picture set of the style.

6. The low-resource picture management system of claim 5, wherein: The picture generation module comprises a text encoding unit, a picture information generation unit and a picture decoding unit. The text encoding unit is used for sampling the text description of the picture according to the positive text in the text encoder to obtain corresponding positive text, and sampling the text description of the picture according to the negative text to obtain corresponding negative text. The picture information generation unit is used for dynamically adjusting the Unet denoising intensity to remove noise in the pure noise hidden variable by combining the semantic control vector, checking the denoised image corresponding to the positive text and the denoised image corresponding to the negative text through the sampler, comparing the differences between the two, and strengthening the generated picture towards the positive text. The picture decoding unit is used for upgrading the low-dimensional space vector obtained in the generator to obtain a complete picture.

7. The low-resource picture management system of claim 6, wherein: The picture information generation unit comprises a Unet network unit and a sampler algorithm unit. The Unet network unit is used for executing the picture generation process in the Unet network; and the sampler algorithm unit is used for controlling the picture generation speed.

Images