Method and apparatus for training picture recognition model, and storage medium, electronic device and program product

Abstract

Disclosed in the embodiments of the present application are a method and apparatus for training a picture recognition model, and a storage medium, an electronic device and a program product. The method comprises: acquiring N sample pictures in a first sample picture set, and picture description text and at least one picture label of each sample picture among the N sample pictures; and for each sample picture, executing the following processing: according to the picture description text and the at least one picture label of the sample picture, generating a group of picture prompt words corresponding to the sample picture, according to the sample picture and the group of picture prompt words, generating a sample picture subset corresponding to the sample picture, wherein N sample picture subsets form a second sample picture set, and by using the first sample picture set and the second sample picture set, training a picture recognition model to be trained.

Description

Image recognition model training method, device, storage medium, electronic device and program product

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 16, 2023, with application number 202310731811.8 and application name “Training method and device for image recognition model, storage medium and electronic device”. Technical Field

[0002] The present application relates to the field of computers, and more specifically, to a method, device, storage medium, electronic device, and program product for training an image recognition model.

[0003] Background of the Invention

[0004] At present, deep learning classification methods are mainly used for image classification, which often require manual labeling, that is, correctly and accurately labeling and classifying images of different categories so that machine learning algorithms can train and predict these data.

[0005] However, for some categories, the amount of data is small (such as data involving sensitive issues). Even if the quality of data annotation is good, due to the small amount, the data in this category is often submerged in the large amount of data used for business training, resulting in the model's weak recognition ability for this category.

[0006] This shows that the training method of the image classification model in the related art has the problem of poor image recognition ability due to the inability to fully train a small amount of data.

[0007] Summary of the Invention

[0008] The embodiments of the present application provide a method, device, storage medium, electronic device and program product for training an image recognition model, so as to at least solve the problem that the image recognition model training method in the related art has poor image recognition ability due to the inability to fully train a small amount of data.

[0009] According to one aspect of an embodiment of the present application, a method for training an image recognition model is provided, comprising:

[0010] Obtain N sample pictures in a first sample picture set and picture description text and at least one picture tag of each sample picture in the N sample pictures;

[0011] For each sample image, perform the following processing:

[0012] generating a set of picture prompts corresponding to the sample picture according to the picture description text and the at least one picture tag of the sample picture;

[0013] generating a sample picture subset corresponding to the sample picture according to the sample picture and the set of picture prompts, wherein N sample picture subsets constitute a second sample picture set; and

[0014] The image recognition model to be trained is trained using the first sample picture set and the second sample picture set.

[0015] According to another aspect of an embodiment of the present application, a training device for an image recognition model is provided, comprising:

[0016] an acquiring unit, configured to acquire N sample pictures from a first sample picture set and a picture description text and at least one picture tag of each sample picture in the N sample pictures;

[0017] A first generating unit is configured to generate, for each sample image, a set of image prompts corresponding to the sample image based on the image description text and the at least one image tag of the sample image;

[0018] A second generating unit is configured to generate, for each sample picture, a sample picture subset corresponding to the sample picture according to the sample picture and the set of picture prompts, wherein the N sample picture subsets constitute a second sample picture set; and

[0019] A training unit is used to train the image recognition model to be trained using the first sample image set and the second sample image set.

[0020] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned training method of the image recognition model when running.

[0021] According to another aspect of the embodiments of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image recognition model training method.

[0022] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to execute the training method of the image recognition model through the computer program.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] FIG1 is a schematic diagram of an application environment of an optional training method for an image recognition model according to an embodiment of the present application;

[0026] FIG2 is a flow chart of an optional method for training an image recognition model according to an embodiment of the present application;

[0027] FIG3 is a flow chart of another optional method for training an image recognition model according to an embodiment of the present application;

[0028] FIG4 is a schematic diagram of an optional method for training an image recognition model according to an embodiment of the present application;

[0029] FIG5 is a schematic diagram of an optional noise addition method according to an embodiment of the present application;

[0030] FIG6 is a schematic diagram of another optional noise addition method according to an embodiment of the present application;

[0031] FIG7 is a schematic diagram of another optional noise addition method according to an embodiment of the present application;

[0032] FIG8 is a schematic diagram of an optional training method for a target text encoder and a target image encoder according to an embodiment of the present application;

[0033] FIG9 is a schematic diagram of an optional training method of a generative pre-training model according to an embodiment of the present application;

[0034] FIG10 is a schematic diagram of an optional visual question answering training method according to an embodiment of the present application;

[0035] FIG11 is a structural block diagram of an optional image recognition model training device according to an embodiment of the present application;

[0036] FIG12 is a schematic structural diagram of an optional electronic device according to an embodiment of the present application;

[0037] FIG13 is a structural block diagram of a computer system of an optional electronic device according to an embodiment of the present application.

[0038] Implementation Method

[0039] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0040] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0041] According to one aspect of an embodiment of the present application, a method for training an image recognition model is provided. Optionally, as an optional implementation, the method for training an image recognition model may be applied to, but is not limited to, the environment shown in FIG1 . The environment may include, but is not limited to, a terminal device 102, a network 110, and a server 112. The terminal device 102 may include, but is not limited to, a display 108, a processor 106, and a memory 104.

[0042] The specific process can be as follows:

[0043] In step S102 , the terminal device 102 sends a sample image and a model training request to the server 112 to request that the image recognition model on the server be trained using the sample image.

[0044] The terminal device 102 may have a client corresponding to the image recognition model, and the sample image can be uploaded through the client. In response to the detected image upload operation, the terminal device 102 can send the sample image and the model training request to the server 112 via the network 110.

[0045] Step S104, after receiving the sample image and the model training request, the server 112 trains the image recognition model according to the data of the received sample image, and recognizes the sample image according to the trained image recognition model to obtain an output result.

[0046] The processing engine 116 of the server 112 may first pull the model parameters corresponding to the model training request from the database 114 according to the model training request, determine the image recognition model, and input the sample image into the image recognition model to train the image recognition model.

[0047] In step S106 , the server 112 sends the output result of the image recognition model to the terminal device 102 via the network 110 .

[0048] In step S108 , the terminal device 102 displays the received output result on the client corresponding to the image recognition model.

[0049] Optionally, the terminal device 102 includes but is not limited to at least one of the following: a mobile phone (such as an Android phone, an iOS phone, etc.), a laptop computer, a tablet computer, a PDA, a MID (Mobile Internet Devices), a PAD, a desktop computer, a smart home appliance, a vehicle-mounted device, AR (Augmented Reality), VR (Virtual Reality) and other virtual reality devices, etc. The above-mentioned network 110 may include but is not limited to: a wired network, a wireless network, wherein the wired network includes: a local area network, a metropolitan area network and a wide area network, and the wireless network includes: Bluetooth, Wireless Fidelity (Wireless Fidelity, referred to as WIFI) and other networks that realize wireless communication. The above-mentioned server 112 may be a single server, or a server cluster composed of multiple servers, or a cloud server. The above is only an example and is not limited to this in this embodiment.

[0050] Optionally, the training method of the above-mentioned image recognition model can be executed by the server 112 alone, or can be executed by the server 112 and the terminal device 102 together, or can be executed by other electronic devices other than the terminal device 102 and the server 112.

[0051] As an optional implementation, FIG2 is a flow chart of an optional image recognition model training method according to an embodiment of the present application, which is executed by an electronic device, such as the server 112 shown in FIG1 . As shown in FIG2 , the process of the image recognition model training method may include the following steps:

[0052] Step S202: Obtain N sample pictures in the first sample picture set and a picture description text and at least one picture tag of each sample picture in the N sample pictures.

[0053] In this step, the image description texts of N sample images and Q image labels corresponding to each sample image are obtained, wherein the N sample images are sample images included in the first sample image set, and N and Q are both positive integers.

[0054] In an embodiment of the present application, the training method of the image recognition model can be applied to training an image classification model. The model used for image classification and recognition can be a model based on a deep learning classification method. Combined with the annotation of the image, by inputting a large number of images of the same type into the model, the image recognition model can have a certain recognition ability for images of that type. Generally speaking, the higher the quality of the image annotation and the more images are input, the stronger the image recognition model's ability to recognize the images. Here, the annotation of the image can be a description of the image, such as a sentence describing the image.

[0055] In related technologies, image annotation is often done manually. However, manual annotation often requires a lot of time to collect data and train annotation personnel to understand the standards. The annotation time is long and the training optimization process is slow. In addition to the long time and manpower costs, there is also the problem of uneven quality of manual annotation. Especially when the customer requires urgent time to go online, the efficiency and quality of manual annotation will be a big problem, which will affect the training effect of the model on such images. In addition, considering that most business data actually involves sensitive issues (such as content review, face review and other personal privacy issues, etc.), the number of images that can be obtained and used for model training is relatively small. Even if the quality of the image annotation is good, during the model training process, the number of images themselves is often insufficient, resulting in a long-tail distribution, making it difficult to optimize the effects of certain categories.

[0056] In related technologies, for a small number of pictures, this small amount of data is often submerged in the large amount of data used for business training, resulting in a weaker model's ability to recognize pictures of this small category.

[0057] To address the above technical issues, in an embodiment of the present application, for a relatively small number of sample images, a large number of images of the same type as the sample images can be generated by combining their own image labels to train the image recognition model on the same type of sample images. Here, the image labels of the sample images themselves can be the label information of the sample images acquired at the same time as the sample images are acquired.

[0058] In addition, for images involved in classification problems, most of their image labels only have a single label, such as knife, fork, etc., and when these few labels are input into the corresponding model, the success rate of image generation is very low. Even if the image is generated, there is a high probability that there will be a large difference between the generated image and the original image.

[0059] In this embodiment, for a small number of sample images, the image description texts of N sample images and Q image tags of each sample image may be obtained simultaneously to increase the success rate of image generation.

[0060] Here, the N sample pictures may be sample pictures included in a first sample picture set. The first sample picture set may be a set consisting of pictures of the same category.

[0061] Here, the image description text may be the descriptive text of the sample image itself, may be the descriptive text directly generated based on the sample image, or may be automatically generated by a related model.

[0062] Step S204 : For each sample image, a set of image prompts corresponding to the sample image is generated according to the image description text and at least one image tag of the sample image.

[0063] To improve the success rate of image generation, we can first generate multiple image prompts corresponding to the sample images based on their description text and the sample images' own image tags. This way, by combining multiple image prompts to generate a large number of images of the same type, we can reduce the difference between the generated new images and the original images. Here, the image description text can be similar to the image prompts, both of which are descriptive phrases, words, short sentences, etc. corresponding to the images.

[0064] Taking the image generation process of the CLIP (Contrastive Language-Image Pre-Training) model (a pre-training model that can process both text and images) in the Stable Diffusion (Sd) model (a text-to-image generation model) as an example, the prompt word in the Sd model's image generation process serves as a constraint on the conditions for image synthesis. In the large-scale dataset used by the CLIP model, each image includes information such as name, prompt word, number, source, and pixel row width, as shown in Table 1 below.

[0065] Table 1

[0066] In this embodiment, after obtaining the image description text and Q image tags for N sample images, a set of image prompts can be generated for each of the N sample images based on the image description text and Q image tags corresponding to each sample image. Here, a set of image prompts can include multiple image prompts. Correspondingly, N sample images can have N sets of image prompts.

[0067] Step S206 : For each sample picture, generate a sample picture subset corresponding to the sample picture according to the sample picture and a set of picture prompts, wherein N sample picture subsets constitute a second sample picture set.

[0068] Based on N groups of picture prompts and N sample pictures, new sample pictures corresponding to each sample picture are generated respectively, thereby forming a second sample picture set, wherein the second sample picture set includes N sample picture subsets, each sample picture subset is generated based on the text representation vector of a group of picture prompts in the N groups of picture prompts and the picture representation vector of the corresponding sample picture in the N sample pictures.

[0069] Since a set of image prompts can contain multiple image prompts, multiple new sample images of the same type but not identical can be generated based on the image representation vector of the sample image and some of the image prompts in the corresponding set of image prompts, thereby forming a sample image subset corresponding to the sample image and containing multiple images. The selected partial image prompts can be obtained by randomly inserting and combining words or sentences.

[0070] In the embodiment of the present application, the image representation vector of the sample image can be the output result obtained after inputting the sample image into the image recognition model. The image recognition model here can be the aforementioned image recognition model to be trained or another trained image recognition model, and this embodiment does not limit this.

[0071] Step 208: Use the first sample picture set and the second sample picture set to train the picture recognition model to be trained.

[0072] After generating the second sample picture set, the first sample picture set and the second sample picture set can be input together into the picture recognition model to be trained for training the picture recognition model, so that the picture recognition model has stronger generalization and higher recognition ability for pictures of the same type as the first sample picture set.

[0073] Taking the Sd model for image generation as an example, the training process of the image recognition model for a small number of sample images in this embodiment can be shown in Figure 3. In the figure, gray dashed lines indicate that the transferred parameters need to be trained, while other dashed lines indicate that the transferred parameters are frozen parameters; black dashed lines indicate that the transferred parameters are copied. The numbers 1, 2, and 3 indicate the training steps in sequence. Where D and ε are the decoder and encoder corresponding to the sd model, respectively, and DP is the diffusion process.

[0074] The original data 301 is a small amount of original data, and can also be a small number of categories in long-tail learning. The generated data 302 is a prompt generated based on the original data. Combined with the image data generated by the original data, the generated data 302 is passed through the backbone network 303 to obtain the corresponding feature vector f, and then the final classification result (cls, i.e. classification) can be obtained based on the feature vector f.

[0075] It should be noted that the dimensions of the image representation vectors fc and f generated by the generated data 302 through the backbone network 303 can be the same (e.g., 16*16*1024), and the initial latent space vector Z and other latent space vectors (such as the diffuse latent space vector Z) obtained by transforming fc can be the same. T 、Target latent space vector Z con The dimensions of the images (e.g., 64*64*3) can also be the same (e.g., 64*64*3), and the size of the generated data (picture) obtained by decoding can be 512*512. In order to reduce the amount of data calculation, in the second and third steps, the size of the picture input to the backbone network 303 can be adjusted to a preset size (e.g., 224*224).

[0076] For categories with a small amount of data, it is necessary to fuse the high-dimensional feature information obtained by the diffusion model to obtain the final loss. The overall loss mainly includes the loss generated when training the diffusion model and the classification loss generated when training the backbone network. The loss generated when training the diffusion model can be, but is not limited to, the initial latent space vector Z and the diffusion latent space vector Z T For example, the initial latent space vector Z is compared with the diffusion latent space vector Z T Input the target loss function (e.g., L1 loss function) to obtain the initial latent space vector Z and the diffuse latent space vector Z T The loss between the initial latent space vector Z and the diffusion latent space vector Z is calculated using the L1 loss function. T The sum of the absolute values ​​of the values ​​at the same position in the initial latent space vector Z and the diffusion latent space vector Z are obtained. T The loss between.

[0077] The classification loss generated when training the backbone network can be the loss between the predicted classification label generated when training the backbone network and the predetermined true classification label. As an optional example, the above loss can be determined by a cross-entropy loss function, that is, the parameter used to represent the predicted classification label (for example, the probability corresponding to the predicted classification label) and the parameter used to represent the true classification label (for example, 1 or 0) are input into the above cross-entropy loss function to obtain a cross-entropy value (that is, the above loss). As an optional example, the smaller the cross-entropy value, the better the prediction effect of the backbone network.

[0078] Through the embodiments provided by the present application, N sample pictures in a first sample picture set and picture description text and at least one picture label of each sample picture in the N sample pictures are obtained; for each sample picture, the following processing is performed: based on the picture description text and the at least one picture label of the sample picture, a group of picture prompts corresponding to the sample picture is generated; based on the sample picture and the group of picture prompts, a sample picture subset corresponding to the sample picture is generated, wherein the N sample picture subsets constitute a second sample picture set; and, using the first sample picture set and the second sample picture set, the picture recognition model to be trained is trained to complete the conversion from a small number of original pictures to a large number of pictures of the same type, and the picture recognition model is trained using the large number of high-quality pictures of the same type generated, which can avoid the situation where the picture recognition model cannot be fully trained due to the small number of pictures available for training, thereby achieving the technical effect of improving the picture recognition ability and improving the picture recognition ability of the picture recognition model.

[0079] As an optional solution, for each sample image in the N sample images, a sample image subset corresponding to the sample image is generated based on the sample image and a set of image prompts, including:

[0080] For the i-th sample image and the i-th group of image prompts, perform the following steps to obtain the i-th sample image subset, where i is a positive integer greater than or equal to 1 and less than or equal to N:

[0081] Obtain the image representation vector of the i-th sample image in advance, and determine the initial latent space vector of the i-th sample image based on the image representation vector;

[0082] Perform noise processing on the initial latent space vector to obtain the diffuse latent space vector corresponding to the i-th sample image;

[0083] In the i-th group of picture prompts, select the M to be used i A subset of picture prompts, where M i is a positive integer greater than or equal to 1;

[0084] M i Each picture prompt in the picture prompt subset is converted into a text representation vector to obtain M i Group text representation vector;

[0085] According to M i The text representation vector and the diffusion latent space vector are combined to determine the M corresponding to the i-th sample image. i A set of sample pictures, where the i-th sample picture subset includes M i Group sample pictures.

[0086] When generating a second set of sample pictures based on N groups of picture prompts and N sample pictures, a picture generation operation may be performed on the i-th group of picture prompts in the N groups of picture prompts and the i-th sample picture corresponding to the i-th group of picture prompts in the N sample pictures, to obtain an i-th sample picture subset. Here, i is a positive integer greater than or equal to 1 and less than or equal to N.

[0087] The above-mentioned picture generation operation can be the above-mentioned operation of combining the picture representation vector with part of the picture prompts in the corresponding set of picture prompts to generate multiple new sample pictures of the same type but not completely the same, that is, according to the picture representation vector of the i-th sample picture and the M in the i-th group of picture prompts i A subset of picture prompts is generated to generate M corresponding to the i-th sample picture i Group sample pictures.

[0088] Here, the i-th sample picture subset includes M i Group sample pictures, M i is a positive integer greater than or equal to 1, and the number of picture prompts included in each picture prompt subset is a positive integer greater than or equal to 1.

[0089] In the i-th group of picture prompts, select the M to be used i When selecting three picture prompt subsets, selection can be made based on the number of picture prompts included in each picture prompt subset. In order to improve the accuracy of image classification, the number of picture prompts included in each picture prompt subset can be set to 3 or more. As shown in Table 1, multiple prompts corresponding to one picture in Table 1 can constitute a picture prompt subset. For example, "small liquid sculpture", "viscous reflection", and "digital art" in Table 1 are three picture prompts that can constitute a picture prompt subset.

[0090] The above-mentioned text image generation model that generates sample images based on image representation vectors and image prompts can be a diffusion model, such as the aforementioned Sd model. Since this type of model is usually a conditional latent space diffusion model, in this embodiment, the image representation vector can be first converted into a latent space vector that can be used by the model, and the image prompt can be converted into a text representation vector that can be combined with the latent space vector. Then, a new sample image can be generated through the decoder of the diffusion model. Here, the latent space refers to the high-dimensional information of the image, which is generally used for feature alignment of the generated results.

[0091] In this embodiment, the initial latent space vector of the i-th sample image can be determined based on the image representation vector of the i-th sample image. Then, using the diffusion model, the obtained initial latent space vector of the i-th sample image is subjected to noise processing to obtain a diffuse latent space vector corresponding to the i-th sample image. Here, the process of obtaining the initial latent space vector from the image representation vector can be completed based on the convolution processing of the image recognition model. The diffuse latent space vector can be used to represent the noise map obtained by adding noise to the i-th sample image.

[0092] For the i-th group of picture prompts corresponding to the i-th sample picture, the M to be used can be selected in the i-th group of picture prompts. i A subset of picture prompts and M i Each picture prompt in the picture prompt subset is converted into a text representation vector to obtain M i Set of text representation vectors.

[0093] Here, the conversion of picture prompts into text representation vectors can be achieved through the corresponding encoding model. The conversion of picture prompts into text representation vectors can be completed by the model that generates picture prompts while the picture prompts are generated. It can also be completed after the picture prompts are generated through other models that can convert prompts into text representation vectors. This embodiment does not limit this.

[0094] According to M i The diffusion latent space vector corresponding to the text representation vector and the i-th sample image can be generated by the denoising and decoding part of the aforementioned diffusion model to generate the M corresponding to the i-th sample image. i Here, the diffusion latent space vector corresponding to the i-th sample image can be combined with M i Some text representation vectors in the group text representation vector are combined in the denoising process, and then through the decoding process, the M corresponding to the i-th sample image is generated. i Group sample pictures.

[0095] Through the embodiments provided in this application, by converting the image representation vector of each sample image into a corresponding diffuse latent space vector and combining it with the text representation vector corresponding to the image prompt to generate a new sample image, the correlation between the generated new sample image and the original sample image can be improved.

[0096] As an optional solution, according to M i The text representation vector and the diffusion latent space vector corresponding to the i-th sample image are grouped together to determine the M corresponding to the i-th sample image. i A set of sample images, including:

[0097] S21, for M iThe jth text representation vector and the diffusion latent space vector in the mth group of text representation vectors in the group text representation vector are performed to obtain the jth sample image in the mth group of sample images corresponding to the i-th sample image, where m is greater than or equal to 1 and less than M i A positive integer, j is a positive integer greater than or equal to 1:

[0098] Use the j-th text representation vector and the preset first noise value set to perform noise reduction on the diffusion latent space vector to obtain the target latent space vector;

[0099] Decode the target latent space vector to obtain the jth sample image.

[0100] In this embodiment, when determining M i After combining the text representation vector and the diffuse latent space vector corresponding to the i-th sample image, the j-th text representation vector and the diffuse latent space vector corresponding to the i-th sample image can be combined first, and then the target latent space vector obtained from the combination is decoded to obtain the j-th sample image. Here, j is a positive integer greater than or equal to 1.

[0101] The aforementioned process of combining the j-th text representation vector and the diffuse latent space vector corresponding to the i-th sample image can be performed by using the j-th text representation vector and the first noise value set to perform noise reduction on the diffuse latent space vector corresponding to the i-th sample image. Through noise reduction, the target latent space vector can be obtained, and then decoding can be performed to obtain the j-th sample image.

[0102] Here, the noise values ​​in the first noise value set may be determined based on the noise values ​​added to the sample image. The values ​​of different noise values ​​may be different, and may be gradually increasing values ​​or randomly changing values. This embodiment does not limit this.

[0103] Through the embodiments provided in the present application, the noise-added image is subjected to noise reduction processing in combination with the text representation vector of the image prompt to obtain a new sample image. This can ensure that the difference between the new sample image and the original sample image remains within an appropriate range. The training effect of the image recognition model will not be poor due to the difference being too small, nor will the new sample image and the original sample image belong to completely different categories due to the difference being too large.

[0104] As an optional solution, the diffuse latent space vector represents the noise image at time t obtained by adding noise to the i-th sample image. Using the j-th text representation vector and the preset first noise value set, the diffuse latent space vector is subjected to noise reduction processing to obtain the target latent space vector, including:

[0105] S31, using the j-th text representation vector and the first noise value set, the diffuse latent space vector is subjected to t rounds of iterative noise reduction processing to obtain the target latent space vector, where t is a positive integer greater than or equal to 2, and one noise value from the first noise value set is used in each round of iterative noise reduction processing.

[0106] The above-mentioned t-round iterative noise reduction processing process can refer to the case where the diffuse latent space vector represents the noise image at the t-th moment obtained by adding noise to the i-th sample image, using the j-th text representation vector and the noise value at the t-th moment in the first noise value set to perform noise reduction processing on the diffuse latent space vector, and then using the result of the noise reduction processing as the latent space vector to be processed, using the j-th text representation vector and the noise value at the t-1-th moment to repeatedly process the latent space vector to be processed until the noise value at the tt-th moment in the noise value set is repeated.

[0107] Through the embodiments provided in the present application, the difference in clarity between the generated sample image and the original sample image can be reduced through t rounds of iterative noise reduction processing.

[0108] As an optional solution, the j-th text representation vector and the first noise value set are used to perform t rounds of iterative noise reduction on the diffusion latent space vector to obtain the target latent space vector, including:

[0109] S41, performing the qth round of iterative noise reduction processing through the following steps, wherein the q-1th round of iterative noise reduction processing outputs the q-1th latent space vector, and the q-1th latent space vector is used to represent the noise map at time tq, where q is a positive integer greater than or equal to 2 and less than or equal to t:

[0110] The q-1th latent space vector is passed through P processing units in sequence, and the output of the qth round of iterative noise reduction processing is the qth latent space vector, where each processing unit includes a residual network and an attention network connected in sequence, the input of the attention network in each processing unit includes the jth text representation vector, and the input of the residual network in each processing unit includes the qth noise value in the first noise value set, and P is a positive integer greater than or equal to 2.

[0111] In this embodiment, during the process of performing t rounds of iterative noise reduction processing on the diffuse latent space vector corresponding to the i-th sample image, the latent space vector input in each round (the diffuse latent space vector or the latent space vector output in the previous round of this round) can be sequentially passed through P processing units. While the latent space vector passes through the P processing units in sequence, the j-th text representation vector is input in sequence, and the input text representation vector is used as a conditional constraint to obtain the target latent space vector of the j-th sample image. Here, each processing unit may include a residual network and an attention network connected in sequence, and the input of the attention network in each processing unit may include the j-th text representation vector, and P is a positive integer greater than or equal to 2.

[0112] For example, the qth round (q is a positive integer greater than or equal to 2 and less than or equal to t) of iterative noise reduction processing can be to sequentially pass the q-1th latent space vector through P processing units to obtain the output result of the qth round of iterative noise reduction processing: the qth latent space vector. The output of the q-1th round of iterative noise reduction processing is the q-1th latent space vector, which is used to represent the noise map at time tq.

[0113] Taking the sampling SD model to generate the jth sample image as an example, as shown in Figure 4, the main modules of the SD model include encoder (ε) 401, decoder (D) 402, diffusion process 403, and denoising process 404. The image representation vector (such as feature dimension fc) of a small amount of data provided by the client (such as 2-3 images provided by the client, or even a single image) passes through the encoder (ε) 401 to obtain the latent space vector Z; after the diffusion process 403, the diffuse latent space vector Z is obtained. T , from Z T Conversely, using the text representation vector, the noise is decoded in the denoising process (U-Net) 404 to obtain the target latent space vector Zcon, which is then converted to a sample image using the decoder (D) 402. The encoder (ε) converts the image into a high-dimensional latent space vector, typically the downsampling portion of the U-Net. The decoder (D) converts the high-dimensional latent space vector into an image, typically the upsampling portion of the U-Net. The U-Net (U-Shaped Network) model is based on a U-shaped encoder-decoder network and is a fully convolutional neural network model.

[0114] Through the embodiments provided in this application, the latent space vector corresponding to the original sample image is passed through P processing units in sequence, and the text representation vector is input as a conditional constraint to obtain the target latent space vector of the original sample image, and then decoded to generate a new sample image. This can improve the correlation between the target latent space vector and the original sample image, thereby improving the correlation between the generated sample image and the original sample image.

[0115] As an optional solution, the q-1th latent space vector is sequentially passed through P processing units, and the qth round of iterative noise reduction processing output is the qth latent space vector, including:

[0116] S51, processing the q-1th latent space vector according to the qth noise value through the first residual network in the first processing unit among the P processing units to obtain a residual result output by the first residual network;

[0117] S52, processing the residual result output by the first residual network and the j-th text representation vector through the first attention network in the first processing unit to obtain the q1-th latent space vector output by the first attention network;

[0118] S53, performing the following steps by the kth processing unit among the P processing units, where k is a positive integer greater than or equal to 2 and less than or equal to P, and the input of the residual network in each processing unit includes the qth noise value:

[0119] Through the kth residual network in the kth processing unit, according to the qth noise value, the qth output of the k-1th attention network in the k-1th processing unit is processed. k-1 The latent space vector is processed to obtain the residual result output by the k-th residual network;

[0120] The kth attention network in the kth processing unit processes the residual result output by the kth residual network and the jth text representation vector to obtain the qth output of the kth attention network. k latent space vectors, where when k is equal to P, the qth latent space vector is the qth latent space vector output by the Pth attention network. P A latent space vector.

[0121] In this embodiment, in the process of passing the q-1th latent space vector through the P processing units in sequence, the q-1th latent space vector may first pass through the residual network in the processing unit, and then pass through the attention network in the processing unit. The qth noise value is input through the first residual network in the first processing unit among the P processing units, and the q-1th latent space vector is processed to obtain the residual result output by the first residual network. The residual result output by the first residual network and the jth text representation vector are then processed through the first attention network in the first processing unit to obtain the q1th latent space vector output by the first attention network.

[0122] The step of executing the kth (k is a positive integer greater than or equal to 2 and less than or equal to P) processing unit after the first processing unit among the P processing units on the q-1th latent space vector may include: performing a noise calculation on the qth latent space vector output by the k-1th attention network in the k-1th processing unit according to a preset noise value through the kth residual network in the kth processing unit. k-1 The kth residual network outputs the residual result, and then the kth attention network in the kth processing unit processes the residual result of the kth residual network output and the jth text representation vector to obtain the qth output of the kth attention network. k Correspondingly, when k is equal to P, the qth latent space vector can be the qth output of the Pth attention network. p A latent space vector.

[0123] It should be noted that, in this embodiment, the residual network in the processing unit refers to the residual network module, and the attention network in the processing unit refers to the attention network module. Correspondingly, the residual network and attention network connected in sequence refer to the residual network module and the attention network module connected in sequence, that is, an attention module is added in sequence to each residual network module of the complete residual network.

[0124] Taking the U-Net model as an example for processing sample images, the residual network, as the core building block of the U-Net (i.e., the module within the processing unit), can be a ResNet (Residual Network) module, and the attention network can be an Attention module. Since the ResNet module cannot directly process text vectors, by combining each ResNet module with an Attention module that can process text vectors, the text representation vector can be fused into the image representation vector.

[0125] As shown in FIG5 , the noise compressed image Z is input into the ResNet module 504. T(i.e., the diffusion latent space vector of the sample image after diffusion processing) 501 and the noise value 502 (determined according to the noise value at time T input during the diffusion process), the residual result output by the connected ResNet module 504 is input into the attention module 505, and the text information (i.e., the text representation vector) 503 is injected. In the U-Net denoising process, the Attention mechanism is continuously used to inject the text representation vector into the denoising process. Each ResNet module is no longer directly connected to the adjacent ResNet module, but an Attention module is added in the middle, see ResNet module 506 and attention module 750. The text representation vector is processed by the Attention module to realize the continuous injection of text information, thereby completing the merging of the latent space vector and the text representation vector of the image. The results obtained by each processing unit are connected and fused to output the predicted noise sample Z T-1 508.

[0126] The above process of processing the image latent space vector and noise value in a ResNet module can be shown in Figure 6. In the ResNet module, the image latent space vector is subjected to multiple convolution processes in the convolution layer to obtain the image vector. The input noise value and the image vector are processed by the fully connected layer under the influence of the activation function to obtain a residual result.

[0127] The process of processing the residual result and the text representation vector in an Attention module can be shown in Figure 7. The Attention module can calculate the attention distribution of the residual result and the text representation vector respectively, and perform weighted averaging to obtain the predicted noise sample.

[0128] The above U-Net denoising can be a multi-cycle process. That is, the output predicted noise sample is used as the input data for denoising again, and the predicted noise sample Z is input into ResNet. T-1 And the noise value corresponding to time T-1, the predicted noise sample Z is obtained T-2 , and then predict the noise sample Z T-2 As new input data, until the predicted noise sample Z is obtained T-T , that is, the target latent space vector (Z) of the jth sample image corresponding to the i-th sample image con ).

[0129] It should be noted that the diffusion model is generally divided into a forward process (diffusion process) and a restoration process. The diffusion process is the process of adding noise. The restoration process is the process of removing noise. The steps performed in this embodiment are the noise removal process combined with the text representation vector, that is, the process of restoring (generating) the sample image combined with the text representation vector.

[0130] Through the embodiments provided in this application, the text representation vector is injected into the denoising process through the attention network, so that the text representation vector and the image latent space vector can be combined in the denoising process, thereby generating new sample images corresponding to the sample images based on different text representation vectors, thereby improving the generation efficiency of sample images.

[0131] As an optional solution, the initial latent space vector of the i-th sample image is determined according to the image representation vector of the i-th sample image, including:

[0132] S61: Process the image representation vector of the i-th sample image through the residual network to obtain the initial latent space vector of the i-th sample image.

[0133] The image representation vector obtained for the i-th sample image can be converted into the initial latent space vector for the i-th sample image required by the diffusion model through a residual network. This conversion process can be performed by processing the image representation vector of the i-th sample image through a convolutional module and a fully connected module of the residual network. After averaging the image representation vectors of the i-th sample image, the initial latent space vector is obtained through the fully connected module.

[0134] Taking the residual network resnet50 (residual network, residual neural network, including 49 convolution modules and one fully connected module) as an example, the i-th sample image is input into resnet50, and the output result can be the image representation vector of the i-th sample image, whose feature dimension fc can be 16*16*1024. The features are averaged and the initial latent space vector Z is obtained through the fully connected module. The feature dimension of Z can be 64*64*3.

[0135] Through the embodiments provided in this application, the image representation vector is converted into the initial latent space vector required by the diffusion model through the residual network, which can improve the success rate of the diffusion model in generating new sample images.

[0136] As an optional solution, noise processing is performed on the initial latent space vector of the i-th sample image to obtain the diffuse latent space vector corresponding to the i-th sample image, including:

[0137] S71, decoding the initial latent space vector of the i-th sample picture to obtain the i-th picture to be processed corresponding to the i-th sample picture;

[0138] S72, encoding the i-th picture to be processed to obtain a picture representation vector of the i-th picture to be processed;

[0139] S73 , performing noise processing on the image representation vector of the i-th image to be processed to obtain a diffusion latent space vector corresponding to the i-th sample image.

[0140] Considering that the model used for denoising and denoising may be a different model from the model used for feature extraction of the sample image, after obtaining the image representation vector of the sample image according to the model used for feature extraction of the sample image and then obtaining the initial latent space vector, the initial latent space vector may be decoded and then encoded to convert the initial latent space vector into a spatial vector that can be used for model recognition and processing of denoising and denoising.

[0141] In this embodiment, the initial latent space vector of the i-th sample image obtained can be first decoded by a decoder to obtain the i-th image to be processed corresponding to the i-th sample image, and then the i-th image to be processed is encoded to obtain the image representation vector of the i-th image to be processed.

[0142] Since the initial latent space vector can be converted from the image representation vector generated by the aforementioned backbone network based on the image after the size reduction, in order to convert the initial latent space vector into a space vector that can be used for model recognition and processing of denoising and denoising, so that the sample image generated based on the diffusion latent space vector and the text representation vector has a larger resolution and is more similar to the original sample image type, in the above-mentioned decoding process of the initial latent space vector, the size of the corresponding image can be expanded, that is, the size of the image to be processed can be larger than the size of the input sample image (that is, the sample image input to the aforementioned backbone network), that is, after the obtained sample image is adjusted to a preset size (e.g., 224*224) according to the description of the aforementioned embodiment and input into the backbone network, it is decoded by the decoder, and the size of the image to be processed obtained will become larger (e.g., 512*512). The process of encoding the image to be processed to obtain the image representation vector can be the same as the process of encoding the sample image to obtain the image representation vector of the sample image, and can be the same encoder.

[0143] The image representation vector of the i-th image to be processed can be subjected to noise processing (i.e., diffusion processing) to obtain the diffused latent space vector corresponding to the i-th sample image. The noise value input during the diffusion process can be used as the noise value input to the residual network in each processing unit.

[0144] Through the embodiments provided in the present application, by first converting the initial latent space vector of the sample image into the image to be processed, and then generating the diffused latent space vector through a noise addition process, it is possible to avoid the problem of directly performing diffusion processing on the initial latent space vector, resulting in the inability to generate the image or the generated image being significantly different from the original image, thereby improving the generation accuracy of the new sample image.

[0145] As an optional solution, the image representation vector of the i-th image to be processed is subjected to noise processing to obtain the diffusion latent space vector corresponding to the i-th sample image, including:

[0146] S81, using a preset second noise value set, performs t rounds of iterative noise processing on the image representation vector of the i-th image to be processed to obtain a diffuse latent space vector corresponding to the i-th sample image, where t is a positive integer greater than or equal to 2, and a corresponding noise value from the second noise value set is used in each round of iterative noise processing.

[0147] In this embodiment, the noise addition processing performed on the image representation vector of the i-th image to be processed may be t rounds of iterative noise addition processing using the second noise value set. Here, t may be a positive integer greater than or equal to 2, and the second noise value set may be composed of different noise values, which may be the same as the noise values ​​in the first noise value set. The second noise value may be a randomly sampled noise value or a Gaussian noise predicted by a corresponding neural network learning model.

[0148] After obtaining the image representation vector of the i-th image to be processed, a noise value corresponding to the second noise value set may be used to add noise to the image representation vector of the i-th image to be processed to obtain a noisy image representation vector. In each subsequent round of iterative noise addition processing, a noise value corresponding to the second noise value set may be used to add noise to the noisy image representation vector obtained after the previous round of noise addition processing. It should be noted that the noise value used in the t-round noise addition processing may be increased as the number of rounds increases.

[0149] Optionally, the t rounds of noise addition processing performed on the picture representation vector of the i-th picture to be processed using the second noise value set may not be iterative, that is, each round of noise addition processing is performed on the picture representation vector of the i-th picture to be processed.

[0150] Taking the image representation vector x0 of the i-th image to be processed as an example, the t-round noise processing of x0 can be shown as formula (1):

[0151] Among them, q(x t |x0) refers to the image x0 obtained by adding Gaussian noise t Formula (1) can be a mean Variance σ 2 =1-β t Gaussian distribution.

[0152] Formula (1) can be transformed into the following formulas (2), (3), and (4):

[0153] When calculating each step, we can first sample a two-dimensional standard Gaussian distribution ∈~N(0,I), and then use the parameter α t Get x from x0 t .

[0154] Through the embodiments provided in the present application, by performing multiple rounds of noise processing on the image representation vector of the i-th image to be processed, the image data can be completely converted into a pure noise image, so that the image can be generated through the reverse denoising process.

[0155] As an alternative, M i Each picture prompt in the picture prompt subset is converted into a text representation vector to obtain M i A set of text representation vectors, including:

[0156] S91, through the target text encoder M i Encode each picture prompt in the picture prompt subset to obtain M i Set of text representation vectors.

[0157] As an optional solution, the image representation vector of the i-th sample image is obtained in advance, including:

[0158] S111: Use the target picture encoder to encode the i-th sample picture to obtain a picture representation vector.

[0159] The target text encoder and the target image encoder are encoders obtained by jointly training the text encoder to be trained and the image encoder to be trained;

[0160] During training, the target text encoder and target image encoder meet the following conditions:

[0161] The similarity between the first text representation vector and the first image representation vector is less than or equal to a preset first threshold, the first text representation vector is a vector obtained by encoding the text information in the first group of information by the target text encoder, the first image representation vector is a vector obtained by encoding the image in the first group of information by the target image encoder, and the text information in the first group of information does not match the image;

[0162] The similarity between the second text representation vector and the second image representation vector is greater than or equal to a preset second threshold, the second text representation vector is a vector obtained by the target text encoder encoding the text information in the second group of information, the second image representation vector is a vector obtained by the target image encoder encoding the image in the second group of information, the text information in the second group of information matches the image, and the second threshold is greater than the first threshold.

[0163] In this embodiment, M i Each image prompt in the image prompt subset is converted into a text representation vector, which can be obtained by encoding each image prompt using a target text encoder. The target text encoder and the target image encoder are encoders obtained by jointly training the text encoder to be trained and the image encoder to be trained. Here, the target image encoder can be the encoder that encodes the sample image to obtain the image representation vector.

[0164] The above-mentioned target text encoder and target image encoder can be different encoders belonging to the same model, such as the text encoder and image encoder of the CLIP model. As shown in Figure 8, the text encoder 801 can convert text into a text representation vector (Text embedding), and the image encoder 802 can convert an image into an image representation vector (Image embedding).

[0165] After the prompt generation module completes the conversion from label to prompt, CLIP's text encoder 801 is first used to compress it into an image representation vector 803, which is then used as text input for vector comparison in the second step (here based on the diffusion model, taking the commonly used LDM model stable diffusion model), so that the diffusion model has stronger conditional constraint information.

[0166] Taking the CLIP model as an example, the CLIP model includes an image encoder and a text encoder. Its training process may include: first, randomly extracting an image and a piece of text from the training set (note that the text and image do not necessarily match; the CLIP model's task is to predict whether the image and text match, thereby starting training). After randomly extracting the text and image, the image encoder and text encoder respectively compress them into two representation vectors: image representation vector 803 and text representation vector 804 (as shown in Figure 8 as two 3*1 vectors).

[0167] Cosine similarity is used to compare the similarity of two representation vectors to determine whether randomly selected text and images match. At the beginning of training, even if the image and text actually match well, the parameters of the two encoders are newly initialized and chaotic, resulting in a chaotic representation vector. The calculated similarity is often close to 0. This means that the image and text are a data pair with similar labels, but the predictions calculated by cosine similarity are dissimilar. This mismatch between similar labels and dissimilar predictions can be used to update the parameters of the two encoders in reverse order.

[0168] By continuously repeating the above backpropagation process, the two encoders can be trained. For matching images and text, at the end of training, the two encoders will output similar representation vectors, and the calculated cosine similarity can be close to 1. For mismatched images and text, at the end of training, the two encoders will output significantly different representation vectors, making the calculated cosine similarity close to 0. After training is complete, for example, if the CLIP model is fed a picture of a puppy and given the text description "puppy photo", the CLIP model will generate two similar representation vectors, thus determining that the text and image match.

[0169] Using this training method, CLIP connects two previously unrelated areas of information: computer vision and human language, creating a unified mathematical representation. In CLIP, text can be converted into images using a text encoder, and vice versa. This is the core of the diffusion model's ability to generate images from text.

[0170] Through the embodiments provided in this application, the generation of picture prompts to text representation vectors is completed by jointly training the text encoder and the text encoder in the picture encoder, which can improve the success rate of generating matching pictures through text representation vectors.

[0171] As an optional solution, N sets of image prompts are generated based on the image description texts of N sample images and Q image tags, including:

[0172] S121, performing the following steps on the image description text of the i-th sample image among the N sample images and the i-th image label among the Q image labels to obtain the i-th group of image description text prompts, where i is a positive integer greater than or equal to 1 and less than or equal to N;

[0173] S123: Input the image description text and the i-th image label as the target question into the generative pre-trained model to obtain the i-th image prompt, wherein the generative pre-trained model is used to generate the corresponding answer based on the input question, and i is a positive integer greater than or equal to 1.

[0174] In this embodiment, when generating N sets of picture prompts based on the picture description texts of N sample pictures and Q picture labels, the target question corresponding to the i-th sample picture can be input into the generative pre-training model, and the corresponding picture prompt can be generated based on the picture description text of the i-th sample picture and the i-th picture label. Here, i is a positive integer greater than or equal to 1 and less than or equal to N. The generative pre-training model can be used to generate the corresponding answer based on the input question. The target question corresponding to the i-th sample picture can be used to request the generation of the corresponding picture prompt based on the picture description text of the i-th sample picture and the i-th picture label.

[0175] Optionally, the image description text of the i-th sample image may be the image description text obtained by inputting the i-th sample image into a text generation model. The text generation model may be used to generate image description text based on an input image.

[0176] The generative pre-trained model can be a language model used to generate sentence dialogue information, such as the ChatGPT (chat Generative Pre-trained Transformer) model. Taking the ChatGPT model as an example, as shown in Figure 9, the ChatGPT model is optimized in three steps, including:

[0177] Step 1: Fine-tune the model by collecting demonstration data and training the supervision strategy;

[0178] Step 2: Train the reward model by collecting comparative data;

[0179] Step 3: Optimize the strategy based on the reward model through reinforcement learning.

[0180] Since it is limited to text-based queries and answers, it is difficult to understand and process visual information. It can be achieved by iteratively processing the generative pre-training model and the input text generation model to achieve visual understanding of the sample images.

[0181] The generative pre-trained model can serve as a conversion tool, receiving input sample image labels and generating corresponding prompts based on textual queries and answers. The language model used for vision and language pre-training, as a model that can understand and process vision, can generate corresponding visual descriptions (i.e., image description text) based on the input sample images. By combining these two models, it is possible to achieve the goal of generating more accurate image description text based on sample images.

[0182] Taking the BLIP-2 model as an example of a text generation model, as shown in Figure 10, the BLIP-2 model introduces the Large Language Model (LLM) model. A lightweight query transformer (Q-Former) model 1002 is added between the frozen pre-trained image encoder 1001 and the frozen pre-trained LLM 1003 to bridge the modality gap between vision and language models. In the entire model, the Q-Former 1002 is the only trainable module, while the image encoder 1001 and the LLM remain frozen. An image is input into the image encoder 1001, and the output is fused with the text 1004 in the Q-Former 1002. Finally, the output is fed into the LLM 1003 to generate an image description 1005.

[0183] Chatgpt is used as a conversion tool. Unpopular labels corresponding to a small number of sample images are input. Multiple rounds of iterative interaction are performed through BLIP-2. Chatgpt summarizes all the information to obtain the final prompt.

[0184] To improve the accuracy of image description text generation, corresponding guidance rules can be pre-established for the generative pre-trained model and the language and vision pre-trained model to guide the iterative interaction between the generative pre-trained model and the language and vision pre-trained model to generate image description text. Here, the guidance rules can include setting tasks and response rules.

[0185] Taking the generative pre-trained model as the Chatgpt model as an example, the tasks and response rules pre-set for the Chatgpt model can be:

[0186] I have an image.Ask me questions about the content of this image.Carefully asking me informative questions to maximize your information about this image content.Each time ask one question only without giving an answer.Avoid asking yes / no questions.I'll put my answer beginning with "Answer:"

[0187] The reply rule can be: Question:<****>Answer:<******>.

[0188] A rule for asking questions could be: Next Question.Avoid asking yes / no questions.Question:

[0189] Taking the BLIP-2 model as an example, since the large-scale language model used in the bottom module of BLIP-2 is almost also GPT-2, it is also necessary to give the BLIP-2 model clear tasks in advance. The set tasks can be: Answer given questions. If you are not sure about the answer, say you don't know honestly. Don't imagine any contents that are not in the image. (Answer given questions. If you are not sure about the answer, say you don't know honestly. Don't imagine any contents that are not in the image.)

[0190] The reply rule can be the same as the reply rule of the aforementioned Chatgpt model, or it can be: Question:<****>Answer:<******>.

[0191] After the interaction is over, Chatgpt can make a final summary. The rules for summarizing can be: Now summarize the information you get in a few sentences. Ignore the questions with answers no or not sure. Don't add information. Don't miss information. Summary: (Now summarize the information you get in a few sentences. Ignore the questions with answers no or not sure. Don't add information. Don't miss information.).

[0192] Optionally, the prompts generated by the generative pre-trained model can be augmented at the text level to increase the amount of text describing the image. This augmentation can include, but is not limited to, replacing words or sentences in the prompts, and performing multiple iterative translations of the prompts into different languages ​​before translating them into Chinese.

[0193] For example, the obtained prompts are subjected to textual data augmentation in the following manner to obtain a series of prompts:

[0194] Perform random synonym replacement on words in a sentence:

[0195] Randomly replace antonyms of words in sentences;

[0196] Perform random homophone replacements on words in sentences;

[0197] Randomly replace typos in words in a sentence:

[0198] Randomly swap the positions of words in a sentence;

[0199] Generate sentences with similar meanings to the following sentences;

[0200] The sentence is translated into English, then back into German, and then into Chinese.

[0201] Through the embodiments provided in this application, the efficiency of generating picture description text can be improved by generating picture description text based on the picture labels of sample pictures through a generative pre-training model.

[0202] It should be noted that for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0203] According to another aspect of the embodiments of the present application, a training device for an image recognition model is provided for implementing the above-mentioned training method for an image recognition model. FIG11 is a block diagram of an optional training device for an image recognition model according to an embodiment of the present application. As shown in FIG11 , the device may include:

[0204] An acquiring unit 1102 is configured to acquire N sample pictures in a first sample picture set and a picture description text and at least one picture tag of each sample picture in the N sample pictures;

[0205] The first generating unit 1104 is connected to the acquiring unit 1102 and is configured to generate, for each sample image, a set of image prompts corresponding to the sample image based on the image description text and the at least one image tag of the sample image;

[0206] A second generating unit 1106 is connected to the first generating unit 1104 and is configured to generate, for each sample picture, a sample picture subset corresponding to the sample picture based on the sample picture and the set of picture prompts, wherein the N sample picture subsets constitute a second sample picture set;

[0207] The training unit 1108 is connected to the second generating unit 1106 and uses the first sample picture set and the second sample picture set to train the picture recognition model to be trained.

[0208] It should be noted that the acquisition unit 1102 in this embodiment can be used to execute the above step S202, the first generation unit 1104 in this embodiment can be used to execute the above step S204, the second generation unit 1106 in this embodiment can be used to execute the above step S206, and the training unit 1108 in this embodiment can be used to execute the above step S208.

[0209] As an optional solution, the second generation unit includes:

[0210] The first execution unit is configured to perform the following steps for the i-th sample picture and the i-th group of picture prompts to obtain the i-th sample picture subset, where i is a positive integer greater than or equal to 1 and less than or equal to N: pre-acquire a picture representation vector of the i-th sample picture, and determine an initial latent space vector of the i-th sample picture based on the picture representation vector; perform noise processing on the initial latent space vector to obtain a diffuse latent space vector corresponding to the i-th sample picture; select M to be used in the i-th group of picture prompts i A subset of picture prompts, where M i is a positive integer greater than or equal to 1; i Each picture prompt in the picture prompt subset is converted into a text representation vector to obtain M i Group text representation vector; according to the M i The text representation vector and the diffusion latent space vector are combined to determine the M corresponding to the i-th sample image. i A set of sample pictures, wherein the i-th sample picture subset includes the M i Group sample pictures.

[0211] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0212] As an optional solution, the first execution unit includes:

[0213] The first execution module is used to execute the M i The j-th text representation vector in the m-th group of text representation vectors and the diffusion latent space vector in the group text representation vector are performed to obtain the j-th sample image in the m-th group of sample images corresponding to the i-th sample image, wherein m is greater than or equal to 1 and less than M i A positive integer, j is a positive integer greater than or equal to 1: using the j-th text representation vector and a preset first noise value set, performing noise reduction processing on the diffusion latent space vector to obtain a target latent space vector; decoding the target latent space vector to obtain the j-th sample image.

[0214] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0215] As an optional solution, the diffuse latent space vector represents a noise image at time t obtained by adding noise to the i-th sample image, and the first execution module includes:

[0216] The first execution submodule is used to use the j-th text representation vector and the first noise value set to perform t rounds of iterative noise reduction processing on the diffusion latent space vector to obtain the target latent space vector, where t is a positive integer greater than or equal to 2, and one noise value from the first noise value set is used in each round of iterative noise reduction processing.

[0217] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0218] As an optional solution, the first execution submodule includes:

[0219] The execution subunit is configured to perform the qth round of iterative noise reduction processing through the following steps, wherein the q-1th round of iterative noise reduction processing outputs the q-1th latent space vector, and the q-1th latent space vector is used to represent the noise map at the tqth time, where q is a positive integer greater than or equal to 2 and less than or equal to t:

[0220] The q-1th latent space vector is passed through P processing units in sequence to obtain the qth latent space vector, wherein each processing unit includes a residual network and an attention network connected in sequence, the input of the attention network in each processing unit includes the jth text representation vector, and the input of the residual network in each processing unit includes the qth noise value in the first noise value set, and P is a positive integer greater than or equal to 2.

[0221] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0222] As an optional solution, the execution subunit includes:

[0223] a first execution sub-module, configured to process the q-1th latent space vector according to the qth noise value through a first residual network in a first processing unit among the P processing units, to obtain a residual result output by the first residual network;

[0224] A second execution sub-module is configured to process the residual result output by the first residual network and the j-th text representation vector through the first attention network in the first processing unit to obtain the q1-th latent space vector output by the first attention network;

[0225] The third execution sub-module is configured to execute the following steps by the kth processing unit among the P processing units, where k is a positive integer greater than or equal to 2 and less than or equal to P:

[0226] Through the kth residual network in the kth processing unit, according to the qth noise value, the qth output of the k-1th attention network in the k-1th processing unit is processed. k-1 The latent space vector is processed to obtain the residual result output by the k-th residual network;

[0227] The residual result output by the kth residual network and the jth text representation vector are processed by the kth attention network in the kth processing unit to obtain the qth text representation vector output by the kth attention network. k latent space vectors, where when k is equal to P, the qth latent space vector is the qth output of the Pth attention network. p A latent space vector.

[0228] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0229] As an optional solution, the first execution unit includes:

[0230] The second execution module is used to process the image representation vector through the convolution module and the fully connected module in the residual network to obtain the initial latent space vector of the i-th sample image.

[0231] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0232] As an optional solution, the first execution unit includes:

[0233] A decoding module, configured to decode the initial latent space vector to obtain an i-th image to be processed corresponding to the i-th sample image;

[0234] A first encoding module is used to encode the i-th image to be processed to obtain a picture representation vector of the i-th image to be processed;

[0235] The third execution module is configured to perform noise processing on the image representation vector of the i-th image to be processed to obtain the diffusion latent space vector corresponding to the i-th sample image.

[0236] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0237] As an optional solution, the third execution module includes:

[0238] A second execution submodule is configured to perform t rounds of iterative noise processing on the image representation vector of the i-th image to be processed using a preset second noise value set to obtain the diffuse latent space vector corresponding to the i-th sample image, where t is a positive integer greater than or equal to 2, and a noise value corresponding to the second noise value set is used in each round of iterative noise processing.

[0239] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0240] As an optional solution, the first execution unit includes:

[0241] The second encoding module is used to encode M i Encode each picture prompt in the picture prompt subset to obtain M i Set of text representation vectors.

[0242] As an optional solution, the second execution module is configured to encode the i-th sample picture using a target picture encoder to obtain the picture representation vector;

[0243] The target text encoder and the target image encoder are encoders obtained by jointly training the text encoder to be trained and the image encoder to be trained;

[0244] Among them, the target text encoder and the target picture encoder meet the following conditions: the similarity between the first text representation vector and the first picture representation vector is less than or equal to a preset first threshold, the first text representation vector is a vector obtained by the target text encoder to encode the text information in the first group of information, the first picture representation vector is a vector obtained by the target picture encoder to encode the picture in the first group of information, and the text information in the first group of information does not match the picture; the similarity between the second text representation vector and the second picture representation vector is greater than or equal to a preset second threshold, the second text representation vector is a vector obtained by the target text encoder to encode the text information in the second group of information, the second picture representation vector is a vector obtained by the target picture encoder to encode the picture in the second group of information, the text information in the second group of information matches the picture, and the second threshold is greater than the first threshold.

[0245] Optional examples of this embodiment can refer to the examples shown in the training method of the above-mentioned image recognition model, which will not be repeated here in this embodiment.

[0246] As an optional solution, the first generating unit includes:

[0247] The fourth execution module is used to input the image description text and the i-th image label as the target question into the generative pre-trained model to obtain the i-th image prompt, wherein the generative pre-trained model is used to generate the corresponding answer based on the input question, and i is a positive integer greater than or equal to 1.

[0248] According to another aspect of the embodiment of the present application, an electronic device for implementing the training method of the above-mentioned image recognition model is also provided. The electronic device can be the terminal device or server shown in Figure 1. This embodiment is illustrated by taking the electronic device as an example of a terminal device. As shown in Figure 12, the electronic device includes a memory 1202 and a processor 1204. The memory 1202 stores a computer program, and the processor 1204 is configured to execute the steps in any of the above-mentioned method embodiments through the computer program.

[0249] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0250] Optionally, in this embodiment, the processor may be configured to execute the method described in the following embodiments through a computer program.

[0251] Alternatively, those skilled in the art will appreciate that the structure shown in FIG12 is merely illustrative, and the electronic device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, or a terminal device such as an MID or PAD. FIG12 does not limit the structure of the electronic device. For example, the electronic device may include more or fewer components (such as a network interface, etc.) than those shown in FIG12, or may have a configuration different from that shown in FIG12.

[0252] Among them, the memory 1202 can be used to store software programs and modules, such as the program instructions / modules corresponding to the training method and device of the image recognition model in the embodiment of the present application. The processor 1204 executes various functional applications and data processing by running the software programs and modules stored in the memory 1202, that is, realizing the above-mentioned training method of the image recognition model. The memory 1202 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1202 may further include a memory remotely arranged relative to the processor 1204, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include but are not limited to the Internet, corporate intranet, local area network, mobile communication network and combinations thereof. Among them, the memory 1202 can be specifically but not limited to being used for information such as serialized files and compiled files.

[0253] As an example, as shown in FIG12 , the memory 1202 may include, but is not limited to, the acquisition unit 1102, the first generation unit 1104, the second generation unit 1106, and the training unit 1108 in the training device for the image recognition model. Furthermore, the memory 1202 may also include, but is not limited to, other module units in the training device for the image recognition model, which will not be described in detail in this example.

[0254] Optionally, the transmission device 1206 is configured to receive or send data via a network. Specific examples of the network may include a wired network and a wireless network. In one embodiment, the transmission device 1206 includes a network interface controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In one embodiment, the transmission device 1206 is a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0255] In addition, the electronic device further includes: a display 1208 for displaying a display interface of a game application; and a connection bus 1210 for connecting various module components in the electronic device.

[0256] In other embodiments, the terminal device or server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting multiple nodes through network communication. The nodes may form a peer-to-peer (P2P) network, and any computing device, such as a server, terminal, or other electronic device, may become a node in the blockchain system by joining the peer-to-peer network.

[0257] According to one aspect of the present application, a computer program product is provided, comprising a computer program / instruction containing program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1309 and / or installed from a removable medium 1311. When the computer program is executed by the central processing unit 1301, the various functions provided in the embodiments of the present application are performed. The serial numbers of the embodiments of the present application are for descriptive purposes only and do not represent the merits of the embodiments.

[0258] Figure 13 schematically shows a block diagram of a computer system for an optional electronic device for realizing an embodiment of the present application. As shown in Figure 13, computer system 1300 includes a central processing unit 1301 (Central Processing Unit, CPU), which can perform various appropriate actions and processes according to the program stored in a read-only memory 1302 (Read-Only Memory, ROM) or the program loaded from a storage portion 1308 into a random access memory 1303 (Random Access Memory, RAM). In the random access memory 1303, various programs and data required for system operation are also stored. The central processing unit 1301, the read-only memory 1302 and the random access memory 1303 are connected to each other via a bus 1304. An input / output interface 1305 (Input / Output interface, i.e., an I / O interface) is also connected to the bus 1304.

[0259] The following components are connected to the input / output interface 1305: an input section 1306 including a keyboard, a mouse, and the like; an output section 1307 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1308 including a hard disk; and a communication section 1309 including a network interface card such as a local area network card or a modem. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the input / output interface 1305 as needed. Removable media 1311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1310 as needed, so that computer programs read therefrom can be installed into the storage section 1308 as needed.

[0260] In particular, according to an embodiment of the present application, the processes described in the various method flow charts can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flow charts. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1309 and / or installed from a removable medium 1311. When the computer program is executed by the central processing unit 1301, the various functions defined in the system of the present application are performed.

[0261] It should be noted that the computer system 1300 of the electronic device shown in FIG13 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present application.

[0262] According to one aspect of the present application, a computer-readable storage medium is provided, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations of the above embodiments.

[0263] Optionally, in this embodiment, the computer-readable storage medium may be configured to store a computer program for executing the steps described in the above embodiment.

[0264] Optionally, in this embodiment, a person of ordinary skill in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0265] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above-mentioned computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which can be personal computers, servers or network devices, etc.) to execute all or part of the steps of the methods described in each embodiment of the present application.

[0266] In the above embodiments of the present application, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0267] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0268] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0269] In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or at least two units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional units.

[0270] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for training an image recognition model, executed by an electronic device, comprising: Obtain N sample pictures in a first sample picture set and a picture description text and at least one picture label of each sample picture in the N sample pictures; For each sample image, perform the following processing: Generate a set of picture prompts corresponding to the sample picture according to the picture description text and the at least one picture tag of the sample picture; Generate a sample picture subset corresponding to the sample picture according to the sample picture and the set of picture prompts, wherein the N sample picture subsets constitute a second sample picture set; and, The picture recognition model to be trained is trained using the first sample picture set and the second sample picture set.

2. The method according to claim 1, wherein: The step of generating a sample picture subset corresponding to the sample picture according to the sample picture and the set of picture prompts includes: For the i-th sample image and the i-th group of image prompts, perform the following steps to obtain the i-th sample image subset, where i is a positive integer greater than or equal to 1 and less than or equal to N: Pre-acquire a picture representation vector of the i-th sample picture, and determine an initial latent space vector of the i-th sample picture according to the picture representation vector; Performing noise processing on the initial latent space vector to obtain a diffuse latent space vector corresponding to the i-th sample image; In the i-th group of picture prompts, select the M to be used i A subset of picture prompts, where M i is a positive integer greater than or equal to 1; The M i Each image prompt in the image prompt subset is converted into a text representation vector to obtain M i Group text representation vector; According to the M i The text representation vector and the diffusion latent space vector are combined to determine the M corresponding to the i-th sample image. i A set of sample pictures, wherein the i-th sample picture subset includes the M i Group sample pictures.

3. The method according to claim 2, wherein: According to the M i The text representation vector and the diffusion latent space vector are combined to determine the M corresponding to the i-th sample image. i A set of sample images, including: For the M i The j-th text representation vector in the m-th group of text representation vectors and the diffusion latent space vector are used to perform the following steps to obtain the j-th sample image in the m-th group of sample images corresponding to the ith sample image, wherein m is greater than or equal to 1 and less than M i A positive integer, j is a positive integer greater than or equal to 1: Using the j-th text representation vector and a preset first noise value set, performing noise reduction processing on the diffusion latent space vector to obtain a target latent space vector; The target latent space vector is decoded to obtain the j-th sample picture.

4. The method according to claim 3, wherein: The diffuse latent space vector represents a noise image at the tth moment obtained after adding noise to the i-th sample image, and the use of the j-th text representation vector and a preset first noise value set to perform noise reduction processing on the diffuse latent space vector to obtain a target latent space vector includes: Using the j-th text representation vector and the first noise value set, perform t rounds of iterative noise reduction processing on the diffusion latent space vector to obtain the target latent space vector, wherein t is a positive integer greater than or equal to 2, and one noise value from the first noise value set is used in each round of iterative noise reduction processing.

5. The method according to claim 4, wherein: The step of using the j-th text representation vector and the first noise value set to perform t rounds of iterative noise reduction processing on the diffusion latent space vector to obtain the target latent space vector includes: The qth round of iterative noise reduction processing is performed through the following steps, wherein the q-1th round of iterative noise reduction processing outputs the q-1th latent space vector, and the q-1th latent space vector is used to represent the noise map at the tqth time, where q is a positive integer greater than or equal to 2 and less than or equal to t: The q-1th latent space vector is sequentially passed through P processing units to obtain the qth latent space vector, wherein each processing unit includes a residual network and an attention network connected in sequence, the input of the attention network in each processing unit includes the jth text representation vector, and the input of the residual network in each processing unit includes the qth in the first noise value set. Noise value, P is a positive integer greater than or equal to 2.

6. The method according to claim 5, wherein: The step of sequentially passing the q-1th latent space vector through P processing units to obtain the qth latent space vector comprises: Processing the q-1th latent space vector according to the qth noise value through the first residual network in the first processing unit among the P processing units to obtain a residual result output by the first residual network; Processing the residual result output by the first residual network and the j-th text representation vector through the first attention network in the first processing unit to obtain the q1-th latent space vector output by the first attention network; The following steps are performed by the kth processing unit among the P processing units, wherein k is a positive integer greater than or equal to 2 and less than or equal to P: Through the kth residual network in the kth processing unit, according to the qth noise value, the qth output of the k-1th attention network in the k-1th processing unit is processed. k-1 The k-th residual network outputs a residual result by processing the latent space vector; The residual result output by the kth residual network and the jth text representation vector are processed by the kth attention network in the kth processing unit to obtain the qth text representation vector output by the kth attention network. k latent space vectors, where when k is equal to P, the qth latent space vector is the qth latent space vector output by the Pth attention network. p A latent space vector.

7. The method according to claim 2, wherein: The determining, according to the picture representation vector, the initial latent space vector of the i-th sample picture comprises: The image representation vector is processed through a convolution module and a fully connected module in a residual network to obtain an initial latent space vector of the i-th sample image.

8. The method according to claim 2, wherein: The performing noise addition processing on the initial latent space vector to obtain the diffuse latent space vector corresponding to the i-th sample image includes: Decoding the initial latent space vector to obtain an i-th picture to be processed corresponding to the i-th sample picture; Encoding the i-th picture to be processed to obtain a picture representation vector of the i-th picture to be processed; Noise processing is performed on the image representation vector of the i-th image to be processed to obtain the diffuse latent space vector corresponding to the i-th sample image.

9. The method according to claim 8, wherein: The step of adding noise to the image representation vector of the i-th image to be processed to obtain the diffusion latent space vector corresponding to the i-th sample image includes: Using a preset second noise value set, perform t rounds of iterative noise processing on the image representation vector of the i-th image to be processed to obtain the diffuse latent space vector corresponding to the i-th sample image, wherein t is a positive integer greater than or equal to 2, and a noise value corresponding to the second noise value set is used in each round of iterative noise processing.

10. The method according to claim 2, wherein: The M i Each image prompt in the image prompt subset is converted into a text representation vector to obtain M i A set of text representation vectors, including: Through the target text encoder, the M i Encode each picture prompt in the picture prompt subset to obtain the M i Set of text representation vectors.

11. The method according to claim 10, wherein: Pre-acquiring the picture representation vector of the i-th sample picture includes: Using a target picture encoder, encoding the i-th sample picture to obtain the picture representation vector; The target text encoder and the target image encoder are encoders obtained by jointly training a text encoder to be trained and an image encoder to be trained; During training, the target text encoder and the target image encoder meet the following conditions: The similarity between the first text representation vector and the first picture representation vector is less than or equal to a preset first threshold, the first text representation vector is a vector obtained by the target text encoder encoding the text information in the first group of information, the first picture representation vector is a vector obtained by the target picture encoder encoding the picture in the first group of information, and the text information in the first group of information does not match the picture; The similarity between the second text representation vector and the second picture representation vector is greater than or equal to a preset second threshold, the second text representation vector is a vector obtained by the target text encoder encoding the text information in the second group of information, the second picture representation vector is a vector obtained by the target picture encoder encoding the picture in the second group of information, the text information in the second group of information matches the picture, and the second threshold is greater than the first threshold.

12. The method according to any one of claims 1 to 11, wherein: The step of generating a set of picture prompts corresponding to the sample picture according to the picture description text and the at least one picture tag of the sample picture includes: The image description text and the i-th image label are used as the target question and input into the generative pre-trained model to obtain the i-th image prompt, wherein the generative pre-trained model is used to generate a corresponding answer according to the input question, and i is a positive integer greater than or equal to 1.

13. A training device for an image recognition model, comprising: An acquiring unit, configured to acquire N sample pictures in a first sample picture set and a picture description text and at least one picture label of each sample picture in the N sample pictures; A first generating unit, configured to generate, for each sample image, a set of image prompts corresponding to the sample image according to the image description text and the at least one image tag of the sample image; A second generating unit is used to generate, for each sample picture, a sample picture subset corresponding to the sample picture according to the sample picture and the set of picture prompts, wherein the N sample picture subsets constitute a second sample picture set; and, A training unit is used to train the image recognition model to be trained using the first sample image set and the second sample image set.

14. The device according to claim 13, wherein: The second generating unit comprises: The first execution unit is used to perform the following steps for the i-th sample picture and the i-th group of picture prompts to obtain the i-th sample picture subset, wherein i is a positive integer greater than or equal to 1 and less than or equal to N: pre-acquire the picture representation vector of the i-th sample picture, and determine the initial latent space vector of the i-th sample picture according to the picture representation vector; perform noise addition processing on the initial latent space vector to obtain the diffusion latent space vector corresponding to the i-th sample picture; select M to be used in the i-th group of picture prompts i A subset of picture prompts, where M i is a positive integer greater than or equal to 1; i Each image prompt in the image prompt subset is converted into a text representation vector to obtain M i Group text representation vector; according to the M i The text representation vector and the diffusion latent space vector are combined to determine the M corresponding to the i-th sample image. i A set of sample pictures, wherein the i-th sample picture subset includes the M i Group sample pictures.

15. The device according to claim 14, wherein: The first execution unit comprises: The first execution module is used to execute the M i The j-th text representation vector in the m-th group of text representation vectors and the diffusion latent space vector are used to perform the following steps to obtain the j-th sample image in the m-th group of sample images corresponding to the ith sample image, wherein m is greater than or equal to 1 and less than M i A positive integer, j is a positive integer greater than or equal to 1: using the j-th text representation vector and a preset first noise value set, performing noise reduction processing on the diffusion latent space vector to obtain a target latent space vector; decoding the target latent space vector to obtain the j-th sample image.

16. The device according to claim 14, wherein: The second generating unit is used to decode the initial latent space vector to obtain the i-th image to be processed corresponding to the i-th sample image; encode the i-th image to be processed to obtain the image representation vector of the i-th image to be processed; and perform noise processing on the image representation vector of the i-th image to be processed to obtain the diffuse latent space vector corresponding to the i-th sample image.

17. The device according to any one of claims 13 to 16, wherein: The first generation unit is used to input the image description text and the i-th image label as target questions into the generative pre-trained model to obtain the i-th image prompt, wherein the generative pre-trained model is used to generate a corresponding answer based on the input question, and i is a positive integer greater than or equal to 1.

18. A computer-readable storage medium comprising a stored program, wherein: When the program is executed, the method according to any one of claims 1 to 12 is executed.

19. A computer program product, comprising a computer program / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 12.

20. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the method according to any one of claims 1 to 12 through the computer program.

Images