Image classification, generative model training and image generation methods and apparatuses

Abstract

The present application relates to the technical field of image processing. Disclosed are an image classification method and apparatus. A particular embodiment of the method comprises: performing feature fusion on the basis of an initial image of an item, a target image of the item and auxiliary information of the target image to obtain a fused feature; and classifying the fused feature to generate a classification result. The embodiment effectively improves the accuracy of a classification result.

Description

Image classification, generative model training and image generation method and device

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410865204.5, filed on June 28, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the technical field of computers, in particular to the technical field of image processing, and more particularly to an image classification, generative model training and image generation method and device. BACKGROUND

[0004] Since an attractive advertisement image can bring a higher click rate, making a beautiful advertisement image is crucial to the success of e-commerce. However, relying on human designers to make advertisements requires a lot of labor costs, so the demand for automatic advertisement image making based on generative models is rising.

[0005] However, generative models often produce images that do not meet the advertisement standards. These non-compliant images will lead to user misunderstanding of the product and bring a bad shopping experience, so human intervention is needed to check whether the generated image is compliant, but human intervention is prone to low accuracy of the review. SUMMARY

[0006] Embodiments of the present application provide an image classification, generative model training and image generation method, device, equipment and storage medium.

[0007] According to a first aspect, embodiments of the present application provide an image classification method, which comprises: performing feature fusion based on an initial image of an item, a target image of the item and auxiliary information of the target image to obtain fused features; and classifying the fused features to generate a classification result.

[0008] According to a second aspect, embodiments of the present application provide a generative model training method, which comprises: performing feature fusion based on an initial image of an item, a target image of the item and auxiliary information of the target image to obtain fused features; classifying the fused features to generate a classification result; constructing a target loss function based on the classification result and a target vector; and adjusting parameters of a generative model based on the target loss function.

[0009] According to a third aspect, embodiments of the present application provide an image generation method, which comprises: obtaining an initial image of an item and a prompt text; and inputting the initial image and the prompt text into a generative model to generate a target image, wherein the generative model is obtained by the method described in any of the implementations of the second aspect above.

[0010] According to a fourth aspect, the embodiments of the present application provide an image classification device, the device comprising: a fusion module and a classification module, wherein the fusion module is configured to perform feature fusion based on an initial image of an article, a target image of the article and auxiliary information of the target image, to obtain fused features; and the classification module is configured to classify the fused features to generate a classification result.

[0011] According to a fifth aspect, the embodiments of the present application provide a generative model training device, the device comprising: an acquisition module, an indication module, a construction module and an adjustment module, wherein the acquisition module is configured to perform feature fusion based on an initial image of an article, a target image of the article and auxiliary information of the target image, to obtain fused features; the indication module is configured to classify the fused features to generate a classification result; the construction module is configured to construct a target loss function based on the classification result and a target vector; and the adjustment module is configured to adjust parameters of the generative model based on the target loss function.

[0012] According to a sixth aspect, the embodiments of the present application provide an image generation device, the device comprising: an obtaining module and a generation module, wherein the obtaining module is configured to obtain an initial image of an article and a prompt text; and the generation module is configured to input the initial image and the prompt text into a generative model to generate a target image, wherein the generative model is obtained by the method described in any of the implementations of the second aspect.

[0013] According to a seventh aspect, the embodiments of the present application provide an electronic device, comprising one or more processors; a storage device having one or more programs stored thereon, when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the implementations of the first aspect, the second aspect or the third aspect.

[0014] According to an eighth aspect, the embodiments of the present application provide a computer readable medium having a computer program stored thereon, when the program is executed by a processor, the method described in any of the implementations of the first aspect, the second aspect or the third aspect is implemented.

[0015] According to a ninth aspect, the embodiments of the present application provide a computer program product comprising a computer program, when the computer program is executed by a processor, the method described in any of the embodiments of the first aspect, the second aspect or the third aspect is implemented.

[0016] The application obtains fusion features by performing feature fusion based on an initial image of an object, a target image of the object, and auxiliary information of the target image; and classifies the fusion features to generate a classification result. In the process of classifying the images according to the initial image and the target image of the object, the influence of the auxiliary information of the target image on the classification result is fully considered, and the accuracy of the generated classification result is effectively improved.

[0017] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0018] FIG. 1 is an exemplary system architecture diagram in which the present application can be applied;

[0019] FIG. 2a is a flowchart of one embodiment of an image classification method according to the present application;

[0020] FIG. 2b is a model structure diagram of one embodiment of an image classification method according to the present application;

[0021] FIG. 3 is a flowchart of one embodiment of a generative model training method according to the present application;

[0022] FIG. 4 is a flowchart of one embodiment of an image generation method according to the present application;

[0023] FIG. 5 is a schematic diagram of one embodiment of an image classification device according to the present application;

[0024] FIG. 6 is a schematic diagram of one embodiment of a generative model training device according to the present application;

[0025] FIG. 7 is a schematic diagram of one embodiment of an image generation device according to the present application;

[0026] FIG. 8 is a structural schematic diagram of a computer system of a server suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments of the present application are described below with reference to the accompanying drawings, which include various details of the embodiments of the present application to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0028] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0029] FIG. 1 shows an exemplary system architecture 100 to which embodiments of the image classification method of the present application can be applied.

[0030] As shown in FIG. 1, the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 serves as a medium to provide a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0031] The terminal devices 101, 102, 103 interact with the server 105 through the network 104 to receive or send messages, etc. The terminal devices 101, 102, 103 can have various communication client applications installed thereon, such as image classification applications, communication applications, etc.

[0032] The terminal devices 101, 102, 103 can be hardware or software. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to mobile phones and laptops. When the terminal devices 101, 102, 103 are software, they can be installed in the above-mentioned electronic devices. They can be implemented as multiple software or software modules (e.g., to provide image classification services), or as a single software or software module. No specific limitation is made herein.

[0033] The server 105 can be a server that provides various services, such as feature fusion based on the initial image of an item, the target image of the item, and the auxiliary information of the target image; classification of the fused features to generate a classification result.

[0034] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (e.g., to provide image classification services), or as a single software or software module. No specific limitation is made herein.

[0035] It should be noted that the image classification method provided by the embodiments of the present disclosure can be executed by the server 105, or can be executed by the terminal device 101, 102, or 103, or can be executed by the server 105 and the terminal device 101, 102, or 103 in cooperation with each other. Accordingly, each part (for example, each unit, subunit, module, and sub-module) included in the image classification apparatus can be all arranged in the server 105, or can be all arranged in the terminal device 101, 102, or 103, or can be arranged in the server 105 and the terminal device 101, 102, or 103, respectively.

[0036] It should be understood that the number of terminal devices, networks, and servers in FIG. 1 is merely illustrative. According to the needs of implementation, there can be any number of terminal devices, networks, and servers.

[0037] FIG. 2a shows a flow 200 that can be applied to an embodiment of the image classification method of the present application. In this embodiment, the image classification method includes the following steps:

[0038] In step 201, feature fusion is performed based on the initial image of the item, the target image of the item, and the auxiliary information of the target image, to obtain a fused feature.

[0039] In this embodiment, the execution subject (such as the server 105 or the terminal device 101, 102, or 103 shown in FIG. 1) can obtain the initial image of the item, the target image of the item, and the auxiliary information of the target image locally or from a remote server that stores the initial image of the item, the target image of the item, and the auxiliary information of the target image.

[0040] The initial image of the item can be an image of the item that does not contain any background, does not contain any decoration, and does not deform the item, such as a transparent background image of the item, a white background image of the item, and the like. The target image of the item can be an image generated based on a preset prompt word and / or the initial image of the item, such as a promotional image of the item, an animation effect image of the item, an ink effect image of the item, and the like.

[0041] The auxiliary information of the target image can be any modality of information corresponding to the target object, such as image modality information, text modality information, video modality information, and the like, which is not limited in the present application.

[0042] Further, the execution subject can use a fusion algorithm or a fusion network in existing technologies or future developed technologies to perform feature fusion based on the initial image of the item, the target image, and the auxiliary information of the target image, to obtain a fused feature.

[0043] In step 202, the fused feature is classified to generate a classification result.

[0044] In the embodiment, after obtaining the fused features, the execution subject can use a classification algorithm or a classification network in the prior art or future developed technology to classify the fused features and generate a classification result.

[0045] The classification result is used to indicate whether the target image meets the preset standard and the respective probability.

[0046] Here, the number of categories included in the classification result can be two or more than two, which is not limited in the application.

[0047] For example, the classification result can include two categories, i.e., meeting the preset standard and not meeting the preset standard, and the respective probabilities are 0.7 and 0.3. The classification result can also include five categories, i.e., a category meeting the preset standard and four categories not meeting the preset standard (for example, spatial misalignment, size misalignment, insignificant object, and shape illusion), and the respective probabilities are 0.6, 0.2, 0.1, 0.2, and 0.4.

[0048] In some optional manners, the fused features are obtained based on the initial image of the object, the target image of the object, and the auxiliary information of the target image, including: the fused features are obtained by using a fusion network based on the initial image of the object, the target image of the object, and the auxiliary information of the target image; and the classification result is generated by classifying the fused features, including: the classification result is generated by using a classification network to classify the fused features.

[0049] In the implementation, the execution subject can classify the target image based on a classification model, where the classification model can include a fusion network and a classification network.

[0050] The execution subject can directly use the fusion network to fuse the features based on the initial image of the object, the target image of the object, and the auxiliary information of the target image to obtain the fused features, and use the classification network to classify the fused features to generate the classification result.

[0051] Further, the classification model can also include an encoding network and a splicing network.

[0052] Specifically, the execution subject can first use the encoding network to extract features from the initial image of the object, the target image of the object, and the auxiliary information of the target image respectively to obtain extracted features, then use the splicing network to splice the extracted features to obtain spliced features, and then input the spliced features into the fusion network to obtain the fused features.

[0053] The fusion network can be a network for fusing features in the prior art or future developed technology, such as a convolution network, a self-attention network, etc., which is not limited in the application.

[0054] The classification network can be a network for classification in the prior art or future development technology, for example, a fully connected network, etc.

[0055] The implementation fully utilizes the advantages of high generalization ability and high feature extraction efficiency of the model, and further improves the accuracy of the classification result.

[0056] In some optional manners, the fusion network is utilized to perform feature fusion based on the initial image of the article, the target image of the article, and the at least one auxiliary image of the target image to obtain the fusion feature.

[0057] In the implementation, the auxiliary information can include at least one auxiliary image, and the auxiliary image is different from the target image in the type of recorded information. For example, the target image is a color image, the type of recorded information is pixel value information, and the auxiliary image is a depth image of the target image, and the type of recorded information is depth value information.

[0058] The execution subject can directly utilize the fusion network to perform feature fusion based on the initial image of the article, the target image of the article, and the at least one auxiliary image of the target image to obtain the fusion feature, and utilize the classification network to classify the fusion feature to generate the classification result.

[0059] Further, the classification model can further include an encoding network and a splicing network.

[0060] Specifically, the execution subject can first utilize the encoding network to respectively perform feature extraction on the initial image of the article, the target image of the article, and the at least one auxiliary image of the target image to obtain extracted features, then utilize the splicing network to splice the extracted features to obtain spliced features, and then input the spliced features into the fusion network to obtain the fusion feature.

[0061] The implementation fully considers the influence of the auxiliary image of the target image on the classification result, and further improves the accuracy of the classification result.

[0062] In some optional manners, the fusion network is used to perform feature fusion based on the initial image of the item, the target image and at least one auxiliary image of the target image to obtain the fusion feature, including: the fusion network is used to perform feature fusion based on the initial image of the item, the target image and at least two auxiliary images of the target image to obtain the fusion feature.

[0063] In the implementation manner, the auxiliary information can include at least two auxiliary images, and the auxiliary images are different from the information type recorded by the target image. The information types recorded by the at least two auxiliary images are also different from each other. The at least two auxiliary images include a depth image and a saliency image, the depth image includes a depth estimation result of the target image, and the saliency image includes saliency region information of the target image.

[0064] The depth image can represent the positional relationship between the item and the background, and the saliency image can represent whether the item is located in a saliency region in the image.

[0065] Here, the depth image can be obtained by inputting the target image into a preset depth estimation model, and the saliency image can be obtained by inputting the target image into a preset saliency detection model.

[0066] The execution subject can directly use the fusion network to perform feature fusion based on the initial image of the item, the target image of the item and the at least two auxiliary images to obtain the fusion feature, and use the classification network to classify the fusion feature to generate the classification result.

[0067] Further, the classification model can further include an encoding network and a splicing network.

[0068] Specifically, the execution subject can first use the encoding network to respectively extract features from the initial image of the item, the target image of the item and the at least two auxiliary images to obtain extracted features, then use the splicing network to splice the extracted features to obtain spliced features, and finally input the spliced features into the fusion network to obtain the fusion feature.

[0069] The implementation manner fully considers the influence of the auxiliary images of the target image on the classification result by using the fusion network to perform feature fusion based on the initial image of the item, the target image and the at least two auxiliary images of the target image to obtain the fusion feature, and using the classification network to classify the fusion feature to generate the classification result, thereby further improving the accuracy of the classification result.

[0070] In some optional manners, the fusion network is used to perform feature fusion based on the initial image of the item, the target image of the item and the auxiliary information of the target image to obtain the fusion feature, including: a filtering network is used to extract visual features in the auxiliary text based on the initial image; and the fusion network is used to perform feature fusion based on the visual features and the target image to obtain the fusion feature.

[0071] In the implementation, the auxiliary information includes auxiliary text, the auxiliary text is text information describing the item, and the classification model further includes a filtering network.

[0072] The execution subject can first extract visual features in the auxiliary text based on the initial image by using the filtering network, and then perform feature fusion based on the visual features and the target image by using the fusion network to obtain the fused features.

[0073] Here, the filtering network can be implemented based on a FFM (Field-aware Factorization Machine) network that adopts a cross-attention mechanism.

[0074] Further, the classification model can further include a text feature extraction network, an encoding network, and a splicing network.

[0075] Here, the execution subject can first extract text features of the auxiliary text by using the text feature extraction network, then extract visual features in the text features based on the initial image by using the filtering network, and finally perform feature fusion based on the visual features and the target image by using the fusion network.

[0076] The text feature extraction network can be a network for extracting text features in existing technologies or future technologies, such as a BERT (Bidirectional Encoder Representations from Transformers) network, an EMLo (Embeddings from Language Models) network, etc., which is not limited in the present application.

[0077] Specifically, the execution subject can first extract features of the initial image of the item and the target image of the item by using the encoding network respectively to obtain extracted features, the extracted features including features of the initial image and features of the target image, input the features of the initial image and text features of the auxiliary text into the filtering network to obtain visual features; then splice the visual features and the features of the target image by using the splicing network to obtain spliced features, and then input the spliced features into the fusion network to obtain the fused features.

[0078] Here, the filtering network can be represented by the following formula: e f = Conv(CrossAttention(e o ,e c )) × Conv(e o ) + e o

[0079] Wherein, eo is taken as the query, ec is taken as the key and value, and is obtained by input mapping of the filtering network, and Conv represents a 1x1 convolution.

[0080] Since the auxiliary text often contains noise information irrelevant to the shape of the article, such as the brand of the article, the usage, etc., the implementation manner extracts the visual features in the auxiliary text based on the initial image by using the filtering network; and performs feature fusion based on the visual features and the target image by using the fusion network to obtain the fusion features, fully considers the influence of the features in the auxiliary text related to vision on the classification result, and improves the accuracy of the classification result.

[0081] In some optional manners, the fusion network is used to perform feature fusion based on the visual features and the target image to obtain the fusion features, including: the fusion network is used to perform feature fusion based on the visual features, the target image, a depth image and a saliency image to obtain the fusion features.

[0082] In the implementation manner, the auxiliary information can further include at least two kinds of auxiliary images, and the at least two kinds of auxiliary images include a depth image and a saliency image, the depth image includes a depth estimation result of the target image, and the saliency image includes saliency region information of the target image.

[0083] The execution subject can use the fusion network to perform feature fusion based on the visual features, the target image, a depth image and a saliency image to obtain the fusion features.

[0084] Further, the classification model can further include a text feature extraction network, an encoding network and a splicing network.

[0085] Specifically, as shown in FIG. 2b, the article is a bowl, the initial image of the article is a see-through image of the article, the target image of the article is an advertisement image of the article, and the auxiliary text is the title of the article, for example, “A brand ceramic bowl with cover and double ears 850ml household large size soup bowl”. The classification model can include an encoding network, a text feature extraction network, a filtering network, a splicing network, a fusion network and a classification network. The execution subject can first use the encoding network to respectively extract features of the initial image of the article, the target image of the article, a depth image and a saliency image, to obtain the features of the initial image, the features of the target image, the features of the depth image and the features of the saliency image; extract the text features of the auxiliary text based on the text feature extraction network; then input the features of the initial image and the text features of the auxiliary text into the filtering network to obtain the visual features; use the splicing network to splice the visual features, the features of the target image, the features of the depth image and the features of the saliency image to obtain spliced features, and then input the spliced features into the fusion network to obtain the fusion features.

[0086] Further, the fusion feature is input into a classification network to obtain a classification result, which is used to indicate whether the advertising image of the item meets the preset standard, i.e., whether it is compliant.

[0087] The implementation fully considers the influence of the auxiliary text and the auxiliary image on the classification result by using the fusion network to perform feature fusion based on the visual feature, the target image, the depth image, and the saliency image, and by using the classification network to classify the fusion feature and generate a classification result, thereby improving the accuracy of the classification result.

[0088] With reference to FIG. 3, a flow 300 according to one embodiment of the generation model training method in the present application is shown. The generation model training method includes the following steps:

[0089] At step 301, feature fusion is performed based on the initial image of the item, the target image of the item, and the auxiliary information of the target image to obtain a fusion feature.

[0090] In the present embodiment, the execution subject (such as the server 105 or the terminal devices 101, 102, and 103 shown in FIG. 1) can obtain the initial image of the item, the target image of the item, and the auxiliary information of the target image locally or from a remote server that stores the initial image of the item, the target image of the item, and the auxiliary information of the target image.

[0091] The target image of the item is generated by a generation model. The generation model can be a model for generating images in existing technologies or future technologies, such as a generative adversarial model, a diffusion model, etc., which are not limited in the present application.

[0092] Specifically, the generation model can be a diffusion generation model, which can specifically include Stable Diffusion and ControlNet. The process of generating the target image of the item based on the diffusion generation model can be that first, a prompt word is input into the Stable Diffusion, and then the initial image of the item is input into the ControlNet through edge detection. In order to keep the details of the item unchanged, the following operation is performed on the latent vector x t during the denoising process: x t + M x o

[0093] wherein x o is the latent variable of the initial image of the item, M is the mask of the item, and I is an all-1 matrix. Finally, after t steps of denoising, the latent variable x o is converted into the target image of the item. Since the initial image of the item is used in combination with the ControlNet, a natural and harmonious background can be generated for the item while keeping the details of the item unchanged.

[0094] Here, the implementation details and technical effects of the execution subject performing feature fusion based on the initial image of the item, the target image of the item, and the auxiliary information of the target image to obtain the fused features can be referred to the descriptions of the related embodiments in FIGS. 2a and 2b, and will not be described here again.

[0095] In step 302, the fused features are classified to generate a classification result.

[0096] In this embodiment, the implementation details and technical effects of the execution subject classifying the fused features to generate the classification result can be referred to the descriptions of the related embodiments in FIGS. 2a and 2b, and will not be described here again.

[0097] In step 303, a target loss function is constructed based on the classification result and a target vector.

[0098] In this embodiment, the execution subject can construct the target loss function based on the classification result and the target vector.

[0099] The target vector is used to increase the probability value indicating compliance with the preset standard in the classification result and reduce the probability value indicating non-compliance with the preset standard in the classification result.

[0100] Specifically, the target loss function F AC can be expressed by the following formula:

[0101] where y d is a one-hot vector, where the probability of the category complying with the preset standard is 1, and the probability of the category not complying with the preset standard is 0. O i is the probability obtained after inputting the target image into the classification model, i.e., the classification result, and N is the total number of samples in one training. The obtained gradient, i.e., the value of the target loss function, is back-propagated to the generation model to optimize it in the direction of improving the probability of generating images complying with the preset standard.

[0102] In step 304, the parameters of the generation model are adjusted based on the target loss function.

[0103] In this embodiment, after determining the target loss function, the execution subject can adjust the parameters of the generation model according to the target loss function to obtain the generation model after parameter adjustment.

[0104] Specifically, the generation model is a diffusion generation model, including Stable Diffusion and ControlNet, and the execution subject can adjust the parameters of the diffusion generation model according to the target loss function to obtain the adjusted diffusion generation model.

[0105] Here, the execution subject can adjust the parameters of the ControlNet part only according to the target loss function, and the parameters of the Stable Diffusion part remain unchanged.

[0106] In some optional manners, adjusting the parameters of the generation model based on the target loss function includes: adjusting the parameters of the generation model based on the target loss function and the divergence loss function.

[0107] In the implementation manner, although the parameters of the generation model are adjusted according to the target loss function, the probability that the generated target image meets the preset standard, i.e., the availability, can be improved, but improving the availability of the image will lead to a decrease in the aesthetic degree. In order to maintain the aesthetic degree of the image under the condition of improving the availability of the image, the execution subject can adjust the parameters of the generation model according to the target loss function and the divergence loss function.

[0108] The divergence loss function is used to make the similarity between the adjusted output distribution and the output distribution before adjustment meet a preset condition.

[0109] Specifically, the divergence loss function L KL can be represented by the following formula: L KL = KL (p θ (x t |z, c) ||p ref (x t |z, c)

[0110] Wherein, c and z are control conditions of the image and text of the generated target image, p θ and p ref represent the output of the fine-tuned model and the model before fine-tuning.

[0111] The implementation manner adjusts the parameters of the generation model based on the target loss function and the divergence loss function, maintains the aesthetic degree of the image under the condition of improving the availability of the image.

[0112] In some optional manners, adjusting the parameters of the generation model based on the target loss function includes: adjusting the parameters of the generation model based on the target loss function and the conditional constraint loss function.

[0113] In the implementation manner, since the purpose of the divergence loss function is to maintain the image unchanged, and the target loss function hopes that the image changes in the direction of higher availability, i.e., changes the image, such a confrontation is difficult to produce a win-win result. In order to achieve a win-win result of the confrontation, the execution subject can determine a total loss function according to the target loss function and the conditional constraint loss function, and adjust the parameters of the generation model according to the total loss function to obtain an adjusted generation model.

[0114] wherein the conditional constraint loss function is used to keep the text condition for generating the image unchanged.

[0115] Specifically, the conditional constraint loss function L CC which can be represented by the following formula:

[0116] wherein, to represent the influence direction of the text condition on the image generation process, can be derived based on a classifier-free training method, which can be specifically represented by the following formula:

[0117] Here, (1-α t ) is a geometric sequence between 0 and 1, ε θ represents the model prediction noise, c and z are the control conditions of the image and text for generating the target image, and t represents the step number in the training process.

[0118] Further, the total loss function F total can be represented by the following formula: F total = F AC + βL CC

[0119] wherein β is a hyperparameter.

[0120] The implementation adjusts the parameters of the generation model based on the target loss function and the conditional constraint loss function, realizes a win-win strategy, and allows the generation model to adjust the parameters in a direction with a higher availability rate under the condition of maintaining the gradient of the text condition.

[0121] As can be seen from FIG. 3, the flow 300 of the generation model training method in the embodiment embodies feature fusion based on the initial image of the item, the target image of the item, and the auxiliary information of the target image; classifies the fusion features to generate a classification result; constructs a target loss function based on the classification result and a target vector; and adjusts the parameters of the generation model based on the target loss function. This way realizes classifying the target image generated by the generation model according to the classification model, and fine-tuning the gradient according to the classification result to adjust the parameters of the generation model, effectively improving the probability that the target image generated by the generation model meets the preset standard.

[0122] Referring still to FIG. 4, it illustrates a flow 400 of an embodiment of the image generation method according to the present application. The image generation method comprises the following steps:

[0123] Step 401, obtaining an initial image of an item and a prompt text.

[0124] In the embodiment, the execution subject (such as the server 105 or the terminal device 101, 102 or 103 shown in FIG. 1) can obtain the initial image and the prompt text of the article by wired or wireless connection.

[0125] The wireless connection can include, but is not limited to, 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other now known or future developed wireless connection.

[0126] In step 402, the initial image and the prompt text are input into the generation model to generate a target image.

[0127] In the embodiment, after obtaining the initial image and the prompt text, the execution subject can input the initial image and the prompt text into the generation model to generate a target model.

[0128] The generation model is obtained by the method described in the embodiment corresponding to FIG. 3, which is not repeated here.

[0129] As can be seen from FIG. 4, the generation model training method of the embodiment 400 embodies obtaining the initial image and the prompt text of the article, inputting the initial image and the prompt text into the generation model, generating a target image, and improving the accuracy of the generated target image.

[0130] Further referring to FIG. 5, as an implementation of the method shown in the above figures, the present application provides an embodiment of an image classification device, which corresponds to the method embodiment shown in FIG. 2a. The device embodiment can be applied to various electronic devices.

[0131] As shown in FIG. 5, the image classification device 500 of the embodiment includes a fusion module 501 and a classification module 502.

[0132] The fusion module 501 can be configured to perform feature fusion based on the initial image of the article, the target image of the article, and the auxiliary information of the target image to obtain fusion features.

[0133] The classification module 502 can be configured to classify the fusion features to generate a classification result.

[0134] In some optional manners of the embodiment, the fusion module is further configured to perform feature fusion based on the initial image of the article, the target image of the article, and the auxiliary information of the target image by using a fusion network to obtain fusion features; and the classification module is further configured to classify the fusion features by using a classification network to generate a classification result.

[0135] In some optional manners of the present embodiment, the fusion module is further configured to perform feature fusion based on the initial image of the item, the target image of the item, and at least one auxiliary image of the target image, by using the fusion network, to obtain the fused features.

[0136] In some optional manners of the present embodiment, the fusion module is further configured to perform feature fusion based on the initial image of the item, the target image of the item, and at least two auxiliary images of the target image, by using the fusion network, to obtain the fused features.

[0137] In some optional manners of the present embodiment, the fusion module is further configured to extract visual features in the auxiliary text based on the initial image, by using the filtering network; and perform feature fusion based on the visual features and the target image, by using the fusion network, to obtain the fused features.

[0138] In some optional manners of the present embodiment, the fusion module is further configured to perform feature fusion based on the visual features, the target image, the depth image, and the saliency image, by using the fusion network, to obtain the fused features.

[0139] Further referring to FIG. 6, as an implementation of the method shown in the above figures, the present application provides an embodiment of a generation model training apparatus, which can be applied in various electronic devices.

[0140] As shown in FIG. 6, the generation model training apparatus 600 of the present embodiment includes an acquisition module 601, an indication module 602, a construction module 603, and an adjustment module 604.

[0141] The acquisition module 601 can be configured to perform feature fusion based on the initial image of the item, the target image of the item, and auxiliary information of the target image, to obtain the fused features.

[0142] The indication module 602 can be configured to classify the fused features to generate a classification result.

[0143] The construction module 603 can be configured to construct a target loss function based on the classification result and a target vector.

[0144] The adjustment module 604 can be configured to adjust parameters of the generation model based on the target loss function.

[0145] In some optional manners of the present embodiment, the adjustment module is further configured to adjust the parameters of the generation model based on the target loss function and the divergence loss function.

[0146] In some optional manners of the present embodiment, the adjustment module is further configured to adjust the parameters of the generation model based on the target loss function and the conditional constraint loss function.

[0147] With further reference to FIG. 7, as an implementation of the method shown in the above figures, the present application provides an embodiment of an image generation device, which can be specifically applied to various electronic devices.

[0148] As shown in FIG. 7, the image generation device 700 of the present embodiment includes an obtaining module 701 and a generating module.

[0149] The obtaining module 701 can be configured to obtain an initial image of an item and prompt text.

[0150] The generating module 702 can be configured to input the initial image and the prompt text into a generation model to generate a target image.

[0151] It should be noted that in the technical solutions of the present disclosure, the collection, collection, update, analysis, processing, use, transmission, storage, etc. of user personal information involved in the technical solutions comply with

[0152] the relevant laws and regulations, are used for legal purposes, and do not violate public order and good customs.

[0153] The user personal information is taken necessary measures to prevent illegal access to user personal information data, and to maintain user personal information security, network security and national security.

[0154] The user personal information is taken necessary measures to prevent illegal access to user personal information data, and to maintain user personal information security, network security and national security.

[0155] According to the embodiments of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0156] As shown in FIG. 8, it is a block diagram of an electronic device according to the image classification method of the embodiment of the present application.

[0157] 800 is a block diagram of an electronic device according to the image classification method of the embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit implementations of the present application described and / or claimed in this document.

[0158] As shown in FIG. 8, the electronic device includes one or more processors 801, memory 802, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected using different buses, and can be mounted on a common main board or otherwise mounted as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device, such as a display device coupled to the interface. In other embodiments, multiple processors and / or buses can be used with multiple memories and multiple memory, if desired. Also, multiple electronic devices can be connected, each device providing part of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). One processor 801 is shown in FIG. 8 as an example.

[0159] The memory 802 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by at least one processor, so that the at least one processor executes the image classification method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the image classification method provided by the present application.

[0160] The memory 802, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the image classification method in the embodiments of the present application (for example, the fusion module 501 and the classification module 502 shown in FIG. 5). The processor 801 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions and modules stored in the memory 802, that is, implements the image classification method in the above method embodiments.

[0161] The memory 802 can include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required by at least one function; the data storage area can store data created by use of the image classification electronic device, etc. In addition, the memory 802 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 802 can optionally include a memory disposed remotely with respect to the processor 801, which can be connected to the image classification electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0162] The electronic device of the image classification method can further include an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803, and the output device 804 can be connected through a bus or other means, and are connected through a bus in FIG. 8 as an example.

[0163] The input device 803 can receive input digital or character information, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 804 can include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor), etc. The display device can include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, and a plasma display. In some embodiments, the display device can be a touch screen.

[0164] Various embodiments of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0165] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0166] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0167] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0168] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0169] According to the technical scheme of the embodiment of the present application, the accuracy of the classification result is effectively improved.

[0170] It should be understood that various forms of flow shown above can be used, with reordering, adding or removing steps. For example, the steps recited in the specification can be performed in parallel, or in a different order, or can be omitted, as long as the desired result of the technology disclosed in the present application is achieved, which is not limited herein.

[0171] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. An image classification method, comprising: performing feature fusion based on an initial image of an item, a target image of the item, and auxiliary information of the target image to obtain fused features; performing classification on the fused features to generate a classification result, the classification result being used to indicate whether the target image meets a preset standard and a respective corresponding probability.

2. The method of claim 1, wherein, The feature fusion based on the initial image of the item, the target image of the item, and the auxiliary information of the target image to obtain the fused features comprises: performing feature fusion based on the initial image of the item, the target image of the item, and the auxiliary information of the target image by using a fusion network to obtain the fused features; and The classification on the fused features to generate the classification result comprises: performing classification on the fused features by using a classification network to generate the classification result, the fusion network and the classification network being included in a preset classification model.

3. The method of claim 2, wherein, The auxiliary information comprises at least one auxiliary image, the auxiliary image being different from the information type recorded by the target image, and the feature fusion based on the initial image of the item, the target image of the item, and the auxiliary information of the target image by using the fusion network to obtain the fused features comprises: performing feature fusion based on the initial image of the item, the target image, and the at least one auxiliary image of the target image by using the fusion network to obtain the fused features.

4. The method of claim 3, wherein, The feature fusion based on the initial image of the item, the target image of the item, and the at least one auxiliary image of the target image by using the fusion network to obtain the fused features comprises: performing feature fusion based on the initial image of the item, the target image of the item, and the at least two auxiliary images of the target image by using the fusion network to obtain the fused features, the at least two auxiliary images comprising a depth image and a saliency image, the depth image comprising a depth estimation result of the target image, and the saliency image comprising saliency region information of the target image.

5. The method of claim 2, wherein, The auxiliary information comprises auxiliary text, the auxiliary text being text information describing the item, the classification model further comprising a filtering network, and the feature fusion based on the initial image of the item, the target image of the item, and the auxiliary information of the target image by using the fusion network to obtain the fused features comprises: extracting visual features in the auxiliary text based on the initial image by using the filtering network; performing feature fusion based on the visual features and the target image by using the fusion network to obtain the fused features.

6. The method of claim 5, wherein, The auxiliary information further comprises the at least two auxiliary images, the at least two auxiliary images comprising the depth image and the saliency image, the depth image comprising the depth estimation result of the target image, and the saliency image comprising the saliency region information of the target image, and the feature fusion based on the visual features and the target image by using the fusion network to obtain the fused features comprises: performing feature fusion based on the visual features, the target image, the depth image, and the saliency image by using the fusion network to obtain the fused features.

7. A model training method, comprising: The feature fusion is performed based on an initial image of an object, a target image of the object, and auxiliary information of the target image, to obtain fused features, and the target image of the object is generated by a generation model; The fused features are classified to generate a classification result, and the classification result is used to indicate whether the target image meets a preset standard and respective corresponding probabilities; A target loss function is constructed based on the classification result and a target vector, and the target vector is used to increase a probability value indicating that the preset standard is met in the classification result and to reduce a probability value indicating that the preset standard is not met in the classification result; Parameters of the generation model are adjusted based on the target loss function.

8. The method of claim 7, wherein, The adjusting of the parameters of the generation model based on the target loss function comprises: The parameters of the generation model are adjusted based on the target loss function and a divergence loss function, and the divergence loss function is used to make a similarity between an adjusted output distribution of the model and an unadjusted output distribution meet a preset condition.

9. The method of claim 7, wherein, The adjusting of the parameters of the generation model based on the target loss function comprises: The parameters of the generation model are adjusted based on the target loss function and a conditional constraint loss function, and the conditional constraint loss function is used to keep a text condition used for generating an image unchanged.

10. An image generation method, comprising: obtaining an initial image of an object and prompt text; inputting the initial image and the prompt text into a generation model to generate a target image, and the generation model is a generation model obtained by the method in any one of claims 7-9.

11. An image classification apparatus, comprising: a fusion module configured to perform feature fusion based on an initial image of an object, a target image of the object, and auxiliary information of the target image to obtain fused features; a classification module configured to classify the fused features to generate a classification result, and the classification result is used to indicate whether the target image meets a preset standard and respective corresponding probabilities.

12. A generation model training apparatus, comprising: an obtaining module configured to perform feature fusion based on an initial image of an object, a target image of the object, and auxiliary information of the target image to obtain fused features, and the target image of the object is generated by a generation model; an indicating module configured to classify the fused features to generate a classification result, and the classification result is used to indicate whether the target image meets a preset standard and respective corresponding probabilities; a constructing module configured to construct a target loss function based on the classification result and a target vector, and the target vector is used to increase a probability value indicating that the preset standard is met in the classification result and to reduce a probability value indicating that the preset standard is not met in the classification result; an adjusting module configured to adjust parameters of the generation model based on the target loss function.

13. An image generation apparatus, comprising: an obtaining module configured to obtain an initial image of an object and prompt text; a generating module configured to input the initial image and the prompt text into a generation model to generate a target image, and the generation model is a generation model obtained by the method in any one of claims 7-9.

14. An electronic device, comprising: comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory has stored computer instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-10.

15. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, the computer instructions are for causing a computer to perform the method of any one of claims 1-10.

16. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-10.

Images