Image processing method and device, electronic equipment, readable storage medium and program product

Abstract

The invention relates to an image processing method and device, electronic equipment, a readable storage medium and a program product, and the method comprises the steps: generating text prompt information for controlling a commodity background of a target commodity based on a first commodity image of the target commodity; and obtaining a mask image of a target commodity, and inputting the mask image, a second commodity image of the target commodity and the text prompt information into an image generation model to obtain a target image with the target commodity as a foreground and an image generated based on the text prompt information as a background. According to the method, the text prompt information and the mask image of the target commodity are taken as conditions, the background is generated through the image generation model, the background of the target commodity can be automatically replaced, and the display effect of the commodity is improved. Moreover, the second commodity image is introduced as an explicit image semantic condition, so that harmonious transition between the background area and the commodity main body can be realized, and the problem that the background and the commodity boundary are obvious in the prior art is solved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of e-commerce, how to stand out from the crowd has become the goal pursued by manufacturers and e-commerce platforms. The display effect of products directly affects consumers' willingness to buy, especially the background of product pictures. Traditional product picture background processing usually requires manual operation by designers, which is time-consuming and labor-intensive and difficult to meet diverse needs. Summary of the invention

[0003] The present disclosure provides an image processing method, device, electronic device, storage medium and program product to at least solve the problem that the method for processing the background of commodity pictures in the related art is time-consuming and labor-intensive and difficult to meet diverse needs. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0005] Generating text prompt information based on the first product image of the target product; the text prompt information is used to control the generation of a product background of the target product;

[0006] A mask image of the target product is obtained, and the mask image, a second product image of the target product, and the text prompt information are input into an image generation model to obtain a target image; the target image has the target product as a foreground and an image generated based on the text prompt information as a background.

[0007] In an exemplary embodiment, inputting the mask image, the second product image of the target product, and the text prompt information into an image generation model to obtain the target image includes:

[0008] Performing encoding processing on the mask image and the second product image respectively to obtain a mask image code and a product image code;

[0009] The mask image code, the product image code and the text prompt information are input into the image generation model to obtain the target image.

[0010] In an exemplary embodiment, the image generation model includes a text encoder, a Unet network, and a generation network;

[0011] The step of inputting the mask image code, the product image code and the text prompt information into the image generation model to obtain a target image includes:

[0012] Inputting the text prompt information into the text encoder in the image generation model to obtain a text code;

[0013] Inputting the text code and the product image code into the encoding module and the decoding module of the Unet network, and inputting the mask image code into the decoding module of the Unet network, to obtain a potential image representation corresponding to the text prompt information;

[0014] The latent image representation is input into the generative network to generate the target image.

[0015] In an exemplary embodiment, the step of generating text prompt information based on the first product image of the target product includes:

[0016] The first product image of the target product is input into a multimodal large language model for processing to obtain the text prompt information.

[0017] In an exemplary embodiment, the acquiring the mask image of the target product includes:

[0018] The first product image or the second product image of the target product is subjected to segmentation processing based on the target product to obtain the mask image; the mask image contains complete features of the target product.

[0019] In an exemplary embodiment, both the first product image and the second product image are images with the target product as the foreground and an arbitrary image as the background.

[0020] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0021] An information generating unit is configured to generate text prompt information based on a first product image of a target product; the text prompt information is used to control the generation of a product background of the target product;

[0022] The image generation unit is configured to execute acquisition of a mask image of the target product, input the mask image, a second product image of the target product and the text prompt information into an image generation model, and obtain a target image; the target image has the target product as a foreground and an image generated based on the text prompt information as a background.

[0023] In an exemplary embodiment, the image generation unit is further configured to perform encoding processing on the mask image and the second product image respectively to obtain a mask image code and a product image code; and input the mask image code, the product image code and the text prompt information into the image generation model to obtain the target image.

[0024] In an exemplary embodiment, the image generation model includes a text encoder, a Unet network and a generation network; the image generation unit is further configured to execute the input of the text prompt information into the text encoder in the image generation model to obtain a text encoding; input the text encoding and the product image encoding into the encoding module and the decoding module of the Unet network, and input the mask image encoding into the decoding module of the Unet network to obtain a potential image representation corresponding to the text prompt information; input the potential image representation into the generation network to generate the target image.

[0025] In an exemplary embodiment, the information generating unit is further configured to input the first product image of the target product into a multimodal large language model for processing to obtain the text prompt information.

[0026] In an exemplary embodiment, the image generation unit is further configured to perform segmentation processing on the first product image or the second product image of the target product based on the target product to obtain the mask image; the mask image contains complete features of the target product.

[0027] In an exemplary embodiment, both the first product image and the second product image are images with the target product as the foreground and an arbitrary image as the background.

[0028] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0029] processor;

[0030] a memory for storing instructions executable by the processor;

[0031] The processor is configured to execute the instructions to implement the method as described in any one of the above items.

[0032] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute any of the methods described above.

[0033] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein the computer program product includes instructions, and when the instructions are executed by a processor of an electronic device, the electronic device is enabled to execute any of the methods described above.

[0034] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0035] Based on the first product image of the target product, text prompt information for controlling the product background of the target product is generated; the mask image of the target product is obtained, and the mask image, the second product image of the target product and the text prompt information are input into the image generation model to obtain a target image with the target product as the foreground and the image generated based on the text prompt information as the background. The method uses the text prompt information and the mask image of the target product as conditions to realize the generation of the background through the image generation model, thereby realizing the automatic replacement of the background of the target product and improving the display effect of the product. Furthermore, by automatically generating prompt information based on the product image, the diversity and specificity of the prompt can be effectively improved, and the flexible adaptation of the product and the background can be realized. Furthermore, the introduction of the second product image as an explicit image semantic condition can realize the harmonious transition between the background area and the product body, solving the problem of obvious boundaries between the background and the product in the prior art.

[0036] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0038] Fig. 1 is a schematic flow chart of an image processing method according to an exemplary embodiment.

[0039] Fig. 2 is a schematic flow chart of an image processing method according to another exemplary embodiment.

[0040] Fig. 3 is a schematic diagram showing a method of generating prompt information based on a first commodity image of a target commodity according to an exemplary embodiment.

[0041] Fig. 4 is a schematic diagram showing effects of a random mask and a mask based on subject segmentation according to an exemplary embodiment.

[0042] Fig. 5 is a schematic flow chart of an image processing method according to yet another exemplary embodiment.

[0043] Fig. 6 is a structural block diagram showing an image processing apparatus according to an exemplary embodiment.

[0044] Fig. 7 is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0045] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0046] It should be noted that the embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. It should also be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in the present disclosure are all information and data authorized by the user or fully authorized by all parties.

[0048] It is understandable that in the related art, although there are some semi-automatic product background replacement technologies based on neural network models, this technology uses neural network models to achieve a certain degree of automatic product background replacement, but there are also many problems. For example, the boundary between the background and the product is obvious, and background redrawing may cause damage to the integrity of the product. In addition, manually designed text prompts often lack diversity, and the generated background may not meet the needs of actual applications.

[0049] Therefore, the present invention aims to solve the problems of obvious boundary between products and background, damage to product integrity caused by background redrawing, and flexible adaptation of products and background in related technologies. By designing an effective control strategy, the automatic replacement of product background can be realized, the overall beauty and attractiveness of the product page can be improved, and the product display effect can be improved.

[0050] In an exemplary embodiment, as shown in FIG1 , an image processing method is provided. This embodiment is illustrated by applying the method to a terminal. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. Among them, the terminal can be but is not limited to various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server can be implemented as an independent server or a server cluster consisting of multiple servers. In this embodiment, the method includes the following steps:

[0051] In step S110 , text prompt information is generated based on the first product image of the target product; the text prompt information is used to control the generation of the product background of the target product.

[0052] The target product can be any type of product, and the first product image is an image with the target product as the foreground and any image as the background. The foreground refers to the most conspicuous and main focus part of the image, which can be understood as the object or element closer to the camera, specifically the product itself. The background is the thing that is farther away from the camera than the foreground and the subject, specifically the part behind the product, which can be any image or color. The background content can be diverse, such as blurred urban landscapes, natural scenes, patterns, or monochrome backgrounds (such as white).

[0053] Among them, text prompt information, namely prompt, is used to provide key information, which can help the model understand the task faster.

[0054] In a specific implementation, after obtaining the first product image of the target product, feature extraction can be performed on the first product image to extract features of the target product, such as product category, color, shape, texture, etc. Text prompt information for controlling the product background of the target product is generated based on the extracted features of the target product.

[0055] In practical applications, natural language generation technology can be used to build a model that can generate text prompt information based on product images. After training, it can understand how to link product images with text prompt information. Therefore, after obtaining the first product image of the target product, the first product image can be input into the model for processing and output text prompt information.

[0056] It can be understood that the traditional method generally manually designs text prompts, which are relatively simple, and the resulting background may not meet the needs of actual applications. Therefore, in this step, by generating text prompt information based on product images, the diversity and specificity of the text prompt information can be effectively improved, and then a background that better meets the needs of actual applications can be generated, thereby helping to optimize the product display effect.

[0057] In step S120, a mask image of the target product is obtained, and the mask image, the second product image of the target product and the text prompt information are input into an image generation model to obtain a target image; the target image has the target product as the foreground and an image generated based on the text prompt information as the background.

[0058] The mask image is a binary image that can be used to control the position and shape of the product in the image. The foreground of the mask image is the product area, and the background area is black. As shown in the structural condition diagram in Figure 2, it is a mask image with an apple in the foreground and a black background.

[0059] Similar to the first product image, the second product image is also an image with the target product as the foreground and an arbitrary image as the background. The first product image and the second product image may be the same, that is, they belong to the same image, or they may be different.

[0060] The image generation model is a text-to-image model that uses text prompt information as a condition to guide image generation so as to generate an image that matches the text prompt information. For example, the image generation model may be a SD model (Stable Diffusion, which uses the principle of the diffusion model to gradually convert random noise into a clear image).

[0061] It is understandable that, in view of the problem of too obvious boundary between the product and the background, after in-depth analysis and research, it was found that, since the semantic information of the main body of the product is implicitly learned through the Unet network in the image generation model, the background generation process relies too much on the text prompt, making it difficult to achieve a harmonious fusion of the background and the main body of the product. Therefore, in order to solve this problem, this application introduces an explicit image semantic conditional branch, which can be regarded as an image prompt (i.e., introducing a second product image) as explicit image reference information to enhance the background area to achieve a harmonious transition between the background area and the main body of the product.

[0062] In a specific implementation, the mask image of the target product can be directly obtained, or the mask image can be generated by performing image segmentation on the first product image or the second product image of the target product. After that, the mask image, the second product image of the target product and the text prompt information are input into a predetermined image generation model for processing to obtain a target image with the target product as the foreground and the image generated based on the text prompt information as the background.

[0063] More specifically, the mask image is another condition for controlling the generation of the target image, which can be regarded as a control map. As an additional adjustment condition for the text prompt information, it controls the position and shape of the product in the image. The second product image is used as explicit image reference information for enhancing the background area to achieve a harmonious transition between the background area and the main body of the product. The text prompt information is used to guide the image generation model to generate a background that conforms to the content described by the text prompt information. These three contents are input into the image generation model, which uses the mask image to determine the area of ​​the product, uses the second product image to fill the product details into the specified area, and uses the text prompt information to generate a background image that matches the characteristics of the target product, thereby outputting a complete and harmonious image with the target product as the foreground combined with the generated background.

[0064] In the above-mentioned image processing method, based on the first product image of the target product, text prompt information for controlling the product background of the target product is generated; the mask image of the target product is obtained, and the mask image, the second product image of the target product and the text prompt information are input into the image generation model to obtain the target image with the target product as the foreground and the image generated based on the text prompt information as the background. This method uses the text prompt information and the mask image of the target product as conditions to realize the generation of the background through the image generation model, thereby realizing the automatic replacement of the background of the target product and improving the display effect of the product. Furthermore, by automatically generating prompt information based on the product image, the diversity and specificity of the prompt can be effectively improved, and the flexible adaptation of the product and the background can be realized. Furthermore, the introduction of the second product image as an explicit image semantic condition can realize the harmonious transition between the background area and the product body, solving the problem of obvious boundaries between the background and the product in the prior art.

[0065] In an exemplary embodiment, in step S120, the mask image, the second product image of the target product, and the text prompt information are input into the image generation model to obtain the target image, which specifically includes: encoding the mask image and the second product image respectively to obtain the mask image code and the product image code; and inputting the mask image code, the product image code, and the text prompt information into the image generation model to obtain the target image.

[0066] In a specific implementation, the encoding process of the mask image and the second product image can be realized by an image encoder. Specifically, the mask image can be input into the first image encoder to obtain the mask image code; the second product image of the target product can be input into the second image encoder to obtain the product image code. Furthermore, the mask image code, the product image code and the text prompt information are input into the image generation model for processing to obtain the target image.

[0067] In this embodiment, by effectively inputting the mask image, the second product image of the target product and the text prompt information into the image generation model, the quality and accuracy of the generated image can be guaranteed.

[0068] In an exemplary embodiment, an image generation model includes a text encoder, a Unet network and a generation network; a mask image code, a product image code and text prompt information are input into the image generation model to obtain a target image, specifically comprising: inputting the text prompt information into the text encoder in the image generation model to obtain a text code; inputting the text code and the product image code into the encoding module and the decoding module of the Unet network, and inputting the mask image code into the decoding module of the Unet network to obtain a potential image representation corresponding to the text prompt information; and inputting the potential image representation into the generation network to generate a target image.

[0069] In the specific implementation, as shown in Figure 2, the image generation model can be a diffusion model-based model, for example, a StableDiffusion model, including a text encoder, a Unet network and a generation network, and the generation network can be a variational autoencoder (VAE). In the process of Stable Diffusion, the main function of U-Net is to combine the features generated by the text encoder with the fixed potential noise, and generate a potential image representation corresponding to the input prompt through multi-layer convolution processing.

[0070] Referring to Figure 2, the process of processing mask image encoding, product image encoding and text prompt information through the image generation model is as follows: the text prompt information is first input into the text encoder in the image generation model to obtain the text encoding. At the same time, a noise latent variable (noise latent) is generated from the latent space (an abstract space used by the generation model to implicitly represent the features of the data), and the text encoding, noise latent variable, product image encoding and mask image encoding are input into the Unet network together to obtain a latent image representation. Furthermore, the latent image representation is passed through the decoder of the VAE (variational autoencoder) to convert the feature map in the latent space into a visible image and output the target image. Among them, the Unet network includes an encoding module and a decoding module. The text encoding, noise latent variable and product image encoding are simultaneously input into the encoding module and decoding module of the Unet network, while the mask image encoding only needs to be input into the decoding module of the Unet network.

[0071] In this embodiment, by combining the details of the product image, the semantics of the text description, and the specific area information of the mask image, the generated target image can reflect the characteristics of these inputs with high quality, ensuring that its content is accurate and attractive. In addition, the structure of the Unet network can effectively capture local features and global context, ensuring the efficiency and accuracy of the image generation process.

[0072] It is understandable that although manually designed text prompts have great controllability, their production efficiency is relatively low and may result in a lack of diversity in the generated prompt templates. Therefore, in order to improve efficiency and enrich the diversity of prompts, the present disclosure improves the method of generating text prompt information. Correspondingly, in an exemplary embodiment, in step S110, text prompt information is generated based on the first product image of the target product, including: inputting the first product image of the target product into a multimodal large language model for processing to obtain text prompt information.

[0073] Specifically, this embodiment uses a multimodal large language model (MLLM) to assist in the generation of text prompt information. The first product image of the target product is input into the multimodal large language model, the first product image is converted into a feature vector through the multimodal large language model, the key features and content of the target product in the first product image are extracted, and then the feature vector of the first product image is converted into a natural language description, and a text prompt information is automatically generated according to the features of the target product in the image.

[0074] The generated text prompt information may include a background design description of the product, display features of the target product, image quality, atmosphere requirements, etc. For example, as shown in FIG3, it is a schematic diagram of generating prompt information based on the first product image of the target product according to an embodiment. The shoes in the figure can be regarded as the target product, and the original image can be regarded as the first product image of the target product. The text prompt information generated based on the first product image is "In a dense forest, a pair of sneakers lies quietly on the fallen leaves, surrounded by towering trees and mottled sunlight. Using high resolution and octane rendering technology, the details and texture of the shoes are highlighted, while creating a natural and tranquil atmosphere, ultra-high definition, octane rendering, high-grade texture, details, 32k", and the target image generated based on the text prompt information is shown in the rightmost image in FIG3, and the background of the target image matches the text prompt information.

[0075] In this embodiment, a multimodal large language model is used to assist in the generation of text prompt information. The multimodal large language model can deeply understand the characteristics of the commodity body and generate diversified and adaptable commodity display scenes based on this, thereby improving the generation efficiency of text prompt information and the quality of commodity background description generation.

[0076] It can be understood that if a random mask is directly used when obtaining the mask image of the target product, the image generation model will add new components to the product, thereby destroying its integrity. Because, in order to avoid this situation, this embodiment improves the mask used for image generation. Correspondingly, in an exemplary embodiment, in step S120, the mask image of the target product is obtained, including: performing segmentation processing based on the target product on the first product image or the second product image of the target product to obtain a mask image; the mask image contains the complete features of the target product.

[0077] Specifically, an image segmentation algorithm can be used to perform segmentation processing on the first product image or the second product image of the target product based on the target product, for example, a segmentation model (SAM) can be used to perform segmentation on the first product image or the second product image of the target product based on the target product, identify the target product for segmentation, and obtain a corresponding mask image. The area of ​​the target product in the mask image is white or represents a specific value, while the background area is black. This mask will completely retain the characteristics of the target product, such as the color, shape, edge, etc. of the product, so that the integrity of the product is not lost during subsequent generation.

[0078] In practical applications, the mask image can be obtained through ControlNet (a neural network structure model for controlling stable diffusion). ControlNet is used to control the diffusion model by adding additional conditions. It obtains additional input images and converts them into control maps (mask images) through different preprocessors, which are then used as additional conditions for Stable Diffusion diffusion to guide the diffusion process and obtain more stable diffusion results.

[0079] Referring to FIG4 , which is a schematic diagram of the effects of a random mask and a mask based on subject segmentation according to an embodiment, it can be seen from the figure that the random mask cannot completely segment the subject, while the mask based on subject segmentation can completely extract the area where the subject is located.

[0080] It should be noted that different segmentation algorithms can also be used to segment the first product image or the second product image of the target product based on the target product. For example, the edge detection method is used to identify the location where the gray value in the image suddenly changes to detect the edge, thereby achieving image segmentation. The clustering-based segmentation method is used to cluster the image pixels and classify similar pixels into one category to achieve image segmentation. The deep learning-based segmentation method is used to train a deep neural network to achieve pixel-level classification, thereby achieving image segmentation, etc.

[0081] In this embodiment, the mask image used in image generation is replaced by a mask based on subject segmentation instead of a random mask, thereby only adding reference context information of the background area and ensuring the integrity of the product body. In this way, when the background is redrawn, the integrity of the product body can be more effectively avoided from being destroyed.

[0082] In one embodiment, the method further includes a training process of an image generation model, which can be a diffusion model (such as Stable Diffusion), which is essentially a process of predicting noise and gradually denoising an input with noise by predicting noise until it is restored. It includes two processes: a forward process and a reverse process, wherein the forward process (or forward process) is to continuously add random noise (p) according to a certain step interval (T) on the basis of the original input (X0) until a pure noise image (XT) that conforms to the standard normal distribution is finally obtained. The reverse process is a denoising process, which gradually (according to the step interval T) eliminates noise (p) from a pure noise (XT) by training a neural network, and restores the pure noise image (XT) to X0. The noisy data formed based on the forward process is used for the training of the diffusion model. The learning goal of the neural network is to make T denoising operations just offset the corresponding noise adding operations. After the training is completed, it is only necessary to randomly sample a noise from the standard normal distribution, and then use the neural network in the reverse process to restore the noise to an image to generate an image.

[0083] In practical applications, the first sample product image, the second sample product image and the sample mask image corresponding to the sample product can be obtained as a training set, and the image generation model and the two image encoders other than the multimodal large language model can be trained together through the training set. Specifically, based on the first sample product image of the sample product, the text prompt information is first generated; the second sample product image, the sample mask image and the text prompt information are used as input variables to train the image generation model and the two image encoders together. In each training process, a noise is randomly generated, and the noise is used to perform the forward process to generate a noisy image Xt. After that, Xt is passed to the neural network to let the neural network predict the random noise. The loss function of the training is the mean square error between the predicted noise and the actual noise. Therefore, the loss function can use the gradient descent method to optimize the network to achieve training. After the training is completed, the reverse process can be performed to denoise any noisy image to achieve image generation. To generate different images, it is only necessary to change the sampled noise.

[0084] In another exemplary embodiment, as shown in FIG5 , it is a flow chart of another image processing method according to an exemplary embodiment. In this embodiment, the method includes the following steps:

[0085] Step S510 , obtaining a first product image and a second product image of a target product; the first product image and the second product image are both images with the target product as the foreground and an arbitrary image as the background.

[0086] The first product image and the second product image may be the same or different.

[0087] Step S520: Input the first product image of the target product into the multimodal large language model for processing to obtain text prompt information.

[0088] Step S530 , performing segmentation processing based on the target product on the first product image or the second product image of the target product to obtain a mask image; the mask image contains complete features of the target product.

[0089] Step S540 , encoding the mask image and the second product image respectively to obtain a mask image code and a product image code.

[0090] Step S550: input the text prompt information into a text encoder in the image generation model to obtain a text code.

[0091] Step S560: input the text code and the product image code into the encoding module and the decoding module of the Unet network, and input the mask image code into the decoding module of the Unet network to obtain a potential image representation corresponding to the text prompt information.

[0092] Step S570, input the latent image representation into the generative network to generate a target image; the target image has the target product as the foreground and the image generated based on the text prompt information as the background.

[0093] In this embodiment, the image restoration technology is used to realize the automatic replacement of the product background, improve the overall beauty and attractiveness of the product page, and bring huge commercial value to e-commerce companies. In this process, the image generation model (StableDiffusion) works together with ControlNet restoration, and the extracted products are regarded as the foreground part of the image, and the remaining area is regarded as the background to be processed. The precise prompt is further used to guide the generation of the background to ensure that the generated background and the product are harmoniously adapted. In addition, the introduction of the second product image as an explicit image semantic condition can achieve a harmonious transition between the background area and the main body of the product, and solve the problem of obvious boundaries between the background and the product in the prior art. By replacing the mask image used in image generation with a mask based on subject segmentation, only the reference context information of the background area is increased, and the integrity of the main body of the product is ensured, avoiding the destruction of the integrity of the product, and obtaining a richer and more vivid visual effect.

[0094] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0095] It can be understood that the same / similar parts between the various embodiments of the above method in this specification can refer to each other, and each embodiment focuses on the differences from other embodiments. For related points, please refer to the description of other method embodiments.

[0096] Based on the same inventive concept, the embodiment of the present disclosure also provides an image processing device for implementing the above-mentioned image processing method.

[0097] Fig. 6 is a structural block diagram of an image processing device according to an exemplary embodiment. Referring to Fig. 6, the device includes: an information generating unit 610 and an image generating unit 620, wherein:

[0098] The information generating unit 610 is configured to generate text prompt information based on the first product image of the target product; the text prompt information is used to control the generation of the product background of the target product;

[0099] The image generation unit 620 is configured to execute acquisition of a mask image of a target product, input the mask image, a second product image of the target product, and text prompt information into an image generation model to obtain a target image; the target image has the target product as a foreground and an image generated based on the text prompt information as a background.

[0100] In an exemplary embodiment, the image generation unit 620 is further configured to perform encoding processing on the mask image and the second product image respectively to obtain the mask image code and the product image code; input the mask image code, the product image code and the text prompt information into the image generation model to obtain the target image.

[0101] In an exemplary embodiment, the image generation model includes a text encoder, a Unet network and a generation network; the image generation unit 620 is further configured to execute the input of text prompt information into the text encoder in the image generation model to obtain a text encoding; input the text encoding and the product image encoding into the encoding module and the decoding module of the Unet network, and input the mask image encoding into the decoding module of the Unet network to obtain a potential image representation corresponding to the text prompt information; and input the potential image representation into the generation network to generate a target image.

[0102] In an exemplary embodiment, the information generating unit 610 is further configured to input the first product image of the target product into the multimodal large language model for processing to obtain text prompt information.

[0103] In an exemplary embodiment, the image generating unit 620 is further configured to perform segmentation processing based on the target product on the first product image or the second product image of the target product to obtain a mask image; the mask image contains complete features of the target product.

[0104] In an exemplary embodiment, the first product image and the second product image are both images with the target product as the foreground and an arbitrary image as the background.

[0105] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0106] Fig. 7 is a block diagram of an electronic device 700 for implementing an image processing method according to an exemplary embodiment. For example, the electronic device 700 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0107] 7 , electronic device 700 may include one or more of the following components: a processing component 702 , a memory 704 , a power component 706 , a multimedia component 708 , an audio component 710 , an input / output (I / O) interface 712 , a sensor component 714 , and a communication component 716 .

[0108] The processing component 702 generally controls the overall operation of the electronic device 700, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 702 may include one or more modules to facilitate the interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate the interaction between the multimedia component 708 and the processing component 702.

[0109] The memory 704 is configured to store various types of data to support operations on the electronic device 700. Examples of such data include instructions for any application or method operating on the electronic device 700, contact data, phone book data, messages, pictures, videos, etc. The memory 704 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, an optical disk, or a graphene memory.

[0110] The power supply component 706 provides power to the various components of the electronic device 700. The power supply component 706 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 700.

[0111] The multimedia component 708 includes a screen that provides an output interface between the electronic device 700 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 708 includes a front camera and / or a rear camera. When the electronic device 700 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0112] The audio component 710 is configured to output and / or input audio signals. For example, the audio component 710 includes a microphone (MIC), and when the electronic device 700 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in the memory 704 or sent via the communication component 716. In some embodiments, the audio component 710 also includes a speaker for outputting an audio signal.

[0113] I / O interface 712 provides an interface between processing component 702 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0114] The sensor assembly 714 includes one or more sensors for providing various aspects of status assessment for the electronic device 700. For example, the sensor assembly 714 can detect the open / closed state of the electronic device 700, the relative positioning of the components, such as the display and keypad of the electronic device 700, and the sensor assembly 714 can also detect the position change of the electronic device 700 or the components of the electronic device 700, the presence or absence of contact between the user and the electronic device 700, the orientation or acceleration / deceleration of the device 700, and the temperature change of the electronic device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0115] The communication component 716 is configured to facilitate wired or wireless communication between the electronic device 700 and other devices. The electronic device 700 can access a wireless network based on a communication standard, such as WiFi, a carrier network (such as 2G, 3G, 4G or 5G), or a combination thereof. In an exemplary embodiment, the communication component 716 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 716 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0116] In an exemplary embodiment, the electronic device 700 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0117] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 704 including instructions, and the above instructions can be executed by the processor 720 of the electronic device 700 to complete the above method. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0118] In an exemplary embodiment, a computer program product is further provided. The computer program product includes instructions. The instructions can be executed by the processor 720 of the electronic device 700 to complete the above method.

[0119] It should be noted that the above-mentioned devices, electronic devices, computer-readable storage media, computer program products, etc. may also include other implementation methods according to the description of the method embodiments. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.

[0120] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0121] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Generate text prompt information based on the first product image of the target product; The text prompt information is used to control the generation of the commodity background of the target commodity; A mask image of the target product is obtained, and the mask image, a second product image of the target product, and the text prompt information are input into an image generation model to obtain a target image; the target image has the target product as a foreground and an image generated based on the text prompt information as a background.

2. The method according to claim 1, characterized in that The step of inputting the mask image, the second product image of the target product, and the text prompt information into an image generation model to obtain a target image includes: Performing encoding processing on the mask image and the second product image respectively to obtain a mask image code and a product image code; The mask image code, the product image code and the text prompt information are input into the image generation model to obtain the target image.

3. The method according to claim 2, characterized in that The image generation model includes a text encoder, a Unet network and a generation network; The step of inputting the mask image code, the product image code and the text prompt information into the image generation model to obtain a target image includes: Inputting the text prompt information into the text encoder in the image generation model to obtain a text code; Inputting the text code and the product image code into the encoding module and the decoding module of the Unet network, and inputting the mask image code into the decoding module of the Unet network, to obtain a potential image representation corresponding to the text prompt information; The latent image representation is input into the generative network to generate the target image.

4. The method according to claim 1, characterized in that: The step of generating text prompt information based on the first product image of the target product includes: The first product image of the target product is input into a multimodal large language model for processing to obtain the text prompt information.

5. The method according to claim 1, characterized in that The step of obtaining the mask image of the target product includes: The first product image or the second product image of the target product is subjected to segmentation processing based on the target product to obtain the mask image; the mask image contains complete features of the target product.

6. The method according to any one of claims 1 to 5, characterized in that: The first product image and the second product image are both images with the target product as the foreground and an arbitrary image as the background.

7. An image processing device, characterized in that: include: An information generating unit is configured to generate text prompt information based on a first product image of a target product; The text prompt information is used to control the generation of the commodity background of the target commodity; The image generation unit is configured to execute acquisition of a mask image of the target product, input the mask image, a second product image of the target product and the text prompt information into an image generation model, and obtain a target image; the target image has the target product as a foreground and an image generated based on the text prompt information as a background.

8. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the image processing method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 6 is implemented.

Images