Image generation method, model determination method, and image generation system

Abstract

The invention discloses an image generation method, a model determination method and an image generation system, and relates to the technical fields of large model technology and image processing. The method comprises the steps that an initial object image to be processed and a mask image are determined, the image content of the initial object image comprises at least one to-be-displayed object located on a pure color background, and the mask image is used for representing an image area to which the to-be-displayed object is to be displayed in an initial scene image; using the image processing model to take the initial object image as visual reference information of the mask image, and predicting to obtain a predicted object image corresponding to the mask image; and superposing the predicted object image to the initial scene image to obtain a target scene image, the target scene image being used for representing a display result of the to-be-displayed object in the image area. The technical problem that the image generation effect is poor in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the fields of large model technology and image processing technology, and in particular, to an image generation method, a model determination method and an image generation system. Background Art

[0002] At present, in order to facilitate users to view the matching effect of the objects to be displayed when placed in their own homes, the objects to be displayed can be integrated into the user scene, so that the user can view the matching effect of the objects to be displayed in his own home.

[0003] In the related art, the method of cutting out images is directly used to make the white-background product image into a transparent layer, and then the transparent layer is pasted into the background scene to achieve the purpose of integrating the object to be displayed into the background scene. However, this image generation method lacks the ability to extract detailed information, resulting in unnatural light and shadow of the generated image and poor effect.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide an image generation method, a model determination method and an image generation system, so as to at least solve the technical problem of poor image generation effect in the related art.

[0006] According to one aspect of an embodiment of the present application, a method for generating an image is provided, the method comprising: determining an initial object image and a mask image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to represent the image region to be displayed of the object to be displayed in the initial scene image; using an image processing model to use the initial object image as visual reference information of the mask image, and predicting a predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, the image content of the predicted object image includes the object to be displayed on a transparent background, and the object to be displayed on the transparent background is associated with the image region represented by the mask image; and superimposing the predicted object image on the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image region.

[0007] According to another aspect of an embodiment of the present application, a method for determining a model is also provided, the method comprising: obtaining an object image sample; using the object image sample and a visual encoder to train a diffusion model to obtain an image processing model, wherein the image processing model is used to use the initial object image to be processed as visual reference information of a mask image, and predict a predicted object image corresponding to the mask image, the image content of the initial object image includes at least one object to be displayed on a solid color background, the mask image is used to represent the image area to be displayed in the initial scene image of the object to be displayed, the image content of the predicted object image includes the object to be displayed on a transparent background, the object to be displayed on the transparent background is associated with the image area represented by the mask image, the predicted object image is used to be superimposed on the initial scene image to obtain a target scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image area.

[0008] According to another aspect of an embodiment of the present application, a method for generating an image is also provided, the method comprising: identifying an initial product image to be processed from an e-commerce platform, wherein the image content of the initial product image includes at least one product to be displayed located on a solid color background; using an image processing model to use the initial product image as visual reference information of a mask image, and predicting a predicted product image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using product image samples and a visual encoder, the mask image is used to represent an image area to be displayed in the initial scene image where the product to be displayed is to be displayed, the image content of the predicted product image includes the product to be displayed located on a transparent background, and the product to be displayed located on the transparent background is associated with the image area represented by the mask image; superimposing the predicted product image onto the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the product to be displayed in the image area; and returning the target scene image to the e-commerce platform.

[0009] According to another aspect of an embodiment of the present application, a method for generating an image is also provided, the method comprising: in response to an input instruction acting on an operation interface, displaying an initial object image and a mask image to be processed on the operation interface, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to represent the image region to be displayed of the object to be displayed in the initial scene image; in response to an image generation instruction acting on the operation interface, displaying a predicted object image corresponding to the mask image on the operation interface, wherein the predicted object image is obtained by using an image processing model to use the initial object image as visual reference information of the mask image, and predicting the visual reference information, the image content of the predicted object image includes the object to be displayed on a transparent background, the object to be displayed on the transparent background is associated with the image region represented by the mask image, and the image processing model is obtained by training a diffusion model using object image samples and a visual encoder; on the operation interface, displaying a target scene image corresponding to the predicted object image, wherein the target scene image is obtained by superimposing the predicted object image on the initial scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image region.

[0010] According to another aspect of an embodiment of the present application, there is also provided an image generation system, which includes: a client, for uploading an initial object image to be processed, wherein the image content of the initial object image includes at least one object to be displayed located on a solid color background; a server, for using an image processing model to use the initial object image as visual reference information of a mask image, and predicting a predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, the mask image is used to represent an image area to be displayed in the initial scene image where the object to be displayed is located, and the image content of the predicted object image includes an object to be displayed located on a transparent background, and the object to be displayed located on the transparent background is associated with the image area represented by the mask image; the predicted object image is superimposed on the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area; and the target scene image is returned to the client for display.

[0011] According to another aspect of the embodiments of the present application, a computer terminal is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.

[0012] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0013] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0014] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0015] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0016] In an embodiment of the present application, an initial object image and a mask image to be processed are determined, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to represent the image region to be displayed in the initial scene image of the object to be displayed; an image processing model is used to use the initial object image as visual reference information of the mask image, and a predicted object image corresponding to the mask image is predicted, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, and the image content of the predicted object image includes the object to be displayed on a transparent background, and the object to be displayed on the transparent background is associated with the image region represented by the mask image; the predicted object image is superimposed on the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image region. That is to say, in an embodiment of the present application, the image content of the initial object image includes at least one object to be displayed located on a solid color background, and the image processing model is a model pre-trained using object image samples and a visual encoder. Based on this, the image processing model can effectively capture the detail information in the initial object image, and the mask image is used to represent the image area to be displayed in the initial scene image of the object to be displayed. Based on this, the mask image can be used to specify the area where the object to be displayed is placed in the initial scene, so that the generated image effect is more harmonious and natural, thereby solving the technical problem of poor image effect generated by using the cutout and overlay method in the related art.

[0017] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application scenario of an image generation method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for generating an image according to an embodiment of the present application;

[0021] Figure 3 is a flow chart of a method for determining a model according to an embodiment of the present application;

[0022] Figure 4 is a flow chart of another method for generating an image according to an embodiment of the present application;

[0023] Figure 5 is a flowchart of another method for generating an image according to an embodiment of the present application;

[0024] Figure 6 is a flow chart of an image generation system according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a cutout image according to an embodiment of the present application;

[0026] Figure 8 is a schematic diagram of end-to-end generation of a transparent layer according to an embodiment of the present application;

[0027] Fig. 9 is a schematic diagram of a model training process in end-to-end generation of a transparent layer according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of a furniture implantation scenario according to an embodiment of the present application;

[0029] Fig.11 is a schematic diagram of a network structure of a diffusion model for generating a high-definition transparent layer according to an embodiment of the present application;

[0030] Fig.12 is a schematic diagram of a transparent graph VAE encoder / decoder according to an embodiment of the present application;

[0031] Fig.13 is a schematic diagram of an object group obtained after a white background image is expanded according to an embodiment of the present application;

[0032] Fig.14 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0033] Fig.15 is a schematic diagram of a device for determining a model according to an embodiment of the present application;

[0034] Fig.16 is a schematic diagram of another image generating device according to an embodiment of the present application;

[0035] Fig.17 is a schematic diagram of another image generating device according to an embodiment of the present application;

[0036] Fig.18 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion model parameters. The large model can also be called the foundation model / foundation model (Foundation Model l). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model l, referred to as LLM), multi-modal pre-training model (multi-modal pre-training model l), etc.

[0040] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing through an image processing model in an intelligent matching scenario is taken as an example for explanation,

[0041] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0042] Stable diffusion (SD) refers to a method of image generation through a diffusion model. Compared with Generative Adversarial Networks (GANs) or Variational Autoencoders (VAE), it can usually generate images with higher quality and fewer artifacts, and has a more stable training process.

[0043] A multimodal pre-trained model (DINO-V2), consisting of an image encoder that can extract features of images;

[0044] U-net, a specific convolutional neural network architecture, is mainly used for image segmentation tasks and is used as the basic network structure in SD;

[0045] Variational Autoencoder (VAE), a generative model that combines probabilistic graphical models and deep learning techniques to learn latent representations of data and generate new data samples;

[0046] Low-Rank Adaptation of Large Language Models (Lora), a method for efficiently fine-tuning large models.

[0047] According to an embodiment of the present application, a method for generating an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned image generation method provided in the embodiment of the present application can be applied to the following Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0049] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: The client device performs: uploading an initial object image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background. The server performs the following steps in the image processing method: Step S101, using an image processing model to use the initial object image as visual reference information of a mask image, and predicting a predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, the mask image is used to represent the image area to be displayed in the initial scene image, and the image content of the predicted object image includes the object to be displayed on a transparent background, and the object to be displayed on the transparent background is associated with the image area represented by the mask image; Step S102, superimposing the predicted object image on the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area; Step S103, returning the target scene image to the client for display.

[0050] Under the above operating environment, this application provides Figure 2 The method for generating the image shown. Figure 2 FIG. 1 is a flow chart of a method for generating an image according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0051] Step S201, determining an initial object image and a mask image to be processed.

[0052] In the technical solution provided in the above step S201 of the present application, the image content of the initial object image includes at least one object to be displayed on a solid color background. For example, the initial object image can be a high-definition white background image of the object to be displayed, wherein the object to be displayed can be a product in an e-commerce platform, and the product can be a commodity to be traded, such as a sofa, a coffee table and other furniture commodities, which are only illustrative examples. The mask image is used to represent the image area to be displayed in the initial scene image of the object to be displayed, wherein the mask image can be a binary image, which is used to indicate the placement position of the object to be displayed in the initial scene image (such as the user scene image).

[0053] In this embodiment, the initial object image to be processed is used to provide detailed information on the appearance of the product, and the mask image is used to provide indicative information on the placement of the product in the scene. The initial object image to be processed can be obtained from the e-commerce platform, or the object to be displayed is placed in a solid color background and the object to be displayed is photographed from multiple angles to obtain the initial object image to be processed. The mask image can be generated by the user manually drawing or selecting the area where the product is placed on the image, or by using deep learning technology. This is only an illustrative example, and the specific method of obtaining the initial object image to be processed and the mask image is not limited.

[0054] Optionally, in order to realize the function of adaptive change of perspective, the initial object image to be processed may include pictures of the object to be displayed taken from different angles. This helps the image processing model learn the shape and light and shadow changes of the product at different perspectives, so that during the image synthesis process, appropriate adjustments can be made according to the scene right angles, so that the light and shadow of the generated image at different perspectives are more natural.

[0055] Optionally, the mask image provides position information of the commodity to be displayed in the final composite image, and based on the mask image, the object to be displayed can be integrated into a specified area / position in the image scene.

[0056] Step S202: using the image processing model to use the initial object image as visual reference information of the mask image, and predicting a predicted object image corresponding to the mask image.

[0057] In the technical solution provided in the above step S202 of the present application, the image processing model is obtained by training the diffusion model using the object image sample and the visual encoder, the image content of the object image sample includes at least one display object sample, and the image content of the predicted object image includes the object to be displayed located on the transparent background, and the object to be displayed located on the transparent background is associated with the image area represented by the mask image. Among them, the visual encoder can be a high-resolution visual encoder, and the diffusion model can be a Unet structure.

[0058] In this embodiment, the image processing model is obtained by training a diffusion model (e.g., stab led iffusion) based on object image samples and a visual encoder. The object image samples may be a transparent layer data set, which includes at least one object to be displayed. These object image samples are used to train the image processing model so that it can understand information such as features, shapes, and textures of the object to be displayed. The visual encoder is used to extract deep features of the object image samples to provide a richer visual information basis for the diffusion model.

[0059] Optionally, as can be seen from the introduction of step S201 above, the initial object image is usually a high-definition white background image, and the object to be displayed in the image is clearly visible and contrasts with the solid color background. During the prediction process, the image processing model can use the details and features of the initial object image as visual guidance information when generating the predicted object image. In this way, the image processing model can ensure that the generated object to be displayed is consistent with the initial object image in shape, texture and visual style.

[0060] Optionally, the mask image is a binary image that indicates the position that the object to be displayed should occupy in the final image. The image processing model will use the mask image to locate the object to be displayed, ensuring that the position of the object to be displayed in the generated predicted object image matches the position indicated by the mask image.

[0061] Optionally, the image processing model will predict and generate an image of the object to be displayed based on the features of the initial object image and the position information of the mask image to obtain a predicted object image. The predicted object image places the object to be displayed on a transparent background, consistent with the area specified by the mask image, which means that the generated image can be directly superimposed on the user's real scene image without background interference or position misalignment.

[0062] In this step, since the image content of the initial object image includes at least one object to be displayed located on a solid color background, the initial object image is used as the visual reference information of the image processing model. The appearance of the object to be displayed in the predicted object image generated by the image processing model can be controlled to be consistent with the appearance of the object to be displayed included in the initial object image. That is, a transparent layer of a specified product can be generated based on the initial object image instead of generating a random product.

[0063] Step S203: superimpose the predicted object image onto the initial scene image to obtain the target scene image.

[0064] In the technical solution provided in the above step S203 of the present application, the process of superimposing the predicted object image onto the initial scene image to obtain the target scene image is to fuse the generated predicted object image (e.g., the product image) with the initial scene image (e.g., the actual home environment image provided by the user) to achieve a natural display of the product in the user scene.

[0065] In this embodiment, as can be seen from the introduction of the above step S202, the predicted object image is generated by the above image processing model, and the predicted object image includes the product that the user hopes to see in the scene, and the product image is located on a transparent background, so that the product can be integrated with any background.

[0066] Optionally, the predicted object image usually contains a transparency channel (Alpha channel), which provides transparency information of the product pixels, that is, which parts are completely opaque and which parts are semi-transparent or transparent. This allows the correct transparency relationship between the product and the background to be maintained during the superposition process, achieving a more natural fusion effect between the product and the background.

[0067] Optionally, the role of the mask image in this process is to guide the specific placement position of the predicted object image in the initial scene image. Through the mask image, the object to be displayed in the predicted object image can be accurately placed at a specified position in the initial scene, thereby achieving position control.

[0068] Optionally, the predicted object image is superimposed with the initial scene image at the pixel level. Optionally, each product pixel is blended with the pixel at the corresponding position in the scene according to the transparency value of the alpha channel. Fully opaque product pixels will completely cover the scene pixels, while partially transparent product pixels will be blended with the scene pixels in proportion to show the correct hierarchical relationship between the product and the background.

[0069] Optionally, when superimposing the predicted object image with the initial scene image, the predicted object image can be adaptively adjusted in perspective and rendered in illumination according to the perspective and lighting conditions in the initial scene image, so as to ensure that the visual effect of the predicted object image is consistent with other elements in the initial scene image, thereby further improving the naturalness and authenticity of the image.

[0070] In this step, the predicted object image is superimposed on the initial scene image to obtain the target scene image, which can achieve the integration of the object to be displayed in the predicted object image into the initial scene image, and achieve seamless fusion of the object to be displayed and the initial scene image. For example, assuming that the image content of the predicted object image is a certain furniture product on a transparent background, and the initial scene image is a home environment image taken by the user, based on this, the predicted object image is superimposed on the initial scene image, which can achieve seamless fusion of the furniture product and the home environment, and the user can see the natural effect of the furniture product placed in his own home, helping the user to make a purchase decision or interior design choice.

[0071] In the above steps S201 to S203 of the present application, the image content of the initial object image includes at least one object to be displayed on a solid color background, and the image processing model is a model pre-trained using object image samples and a visual encoder. Based on this, the image processing model can effectively capture the detail information in the initial object image, and the mask image is used to represent the image area to be displayed in the initial scene image of the object to be displayed. Based on this, the mask image can be used to specify the position of the object to be displayed in the initial scene, so that the generated image effect is more harmonious and natural, thereby solving the technical problem of poor image effect generated by using the cutout and overlay method in the related art.

[0072] The above method of this embodiment is further introduced below.

[0073] As an optional implementation, step S202, using an image processing model to use the initial object image as visual reference information of the mask image to predict a predicted object image corresponding to the mask image, includes: inputting initial noise information into the image processing model; using the image processing model to use the initial object image as visual reference information to predict the initial noise information to obtain predicted noise information; and generating a predicted object image based on the predicted noise information and the initial noise information.

[0074] In this embodiment, the initial noise information can be obtained by adding random noise to the foreground layer of the initial object image, that is, the initial noise information is used to indicate the noisy foreground layer. Since the image processing model uses the initial object image as visual reference information, based on this, the image processing model can learn the object features of the initial object image, and then compare the learned object features of the initial object image with the noisy foreground layer to obtain predicted noise information, wherein the predicted noise information can characterize the part of the noisy foreground layer that does not match the object features of the initial object image.

[0075] Optionally, after obtaining the predicted noise information, a predicted object image may be generated according to the predicted noise information and the initial noise information. For example, the predicted noise information may be subtracted from the initial noise information to obtain the predicted object image. The more accurate the predicted noise is, the smaller the image difference between the generated predicted object image and the initial object image is.

[0076] Optionally, the image processing model uses the initial object image as visual reference information, and the process of the image processing model generating predicted noise information will be guided and constrained by the initial object image, so that the predicted object image generated finally has similar features to the initial object image and is placed on a transparent layer. For example, when predicting noise information, the image processing model will make the generated image content match the initial object image in terms of shape, color, texture, etc.

[0077] In this step, starting from random noise, the noise information is predicted and the initial object image is used as a visual reference to gradually remove the noise, and finally a clear predicted object image is generated. The predicted object image has similar features to the initial object image, but is placed on a transparent background to facilitate superposition with the user scene image, thereby realizing the integrated display of the product in a specific scene.

[0078] As an optional implementation, generating a predicted object image based on the predicted noise information and the initial noise information includes: obtaining difference noise information between the predicted noise information and the initial noise information; and converting the difference noise information into the predicted object image.

[0079] In this embodiment, after generating the predicted noise information, the image processing model compares the predicted noise information with the initial noise information and calculates the difference between the two. This difference is called difference noise information, wherein the difference noise information reflects the noise portion that needs to be adjusted by the image processing model in the current iteration step. Therefore, based on the difference noise information, part of the noise in the initial noise information can be removed so as to generate the predicted object image more quickly.

[0080] Optionally, the process of the image processing model generating the predicted object image is an iterative process, and in each iteration, the image processing model predicts the noise distribution of the next step based on the current noise information, and then calculates the difference noise information between the predicted noise information and the current noise information. This difference noise information guides the model to adjust the noise in the next iteration to gradually generate the predicted object image.

[0081] Optionally, at each step of iterative denoising, the image processing model applies the difference noise information to the current noisy image, and gradually restores the image content through the denoising operation. Optionally, the image processing model removes the noise portion indicated by the difference noise information from the current noisy image, and retains or enhances those portions that match the features of the initial object image. As the number of iterations increases, the noise in the noisy image gradually decreases, and the image content gradually becomes clear, eventually transforming from a completely random noise image to a clear predicted object image with specific content.

[0082] In this step, the image processing model generates predicted noise information each time through a series of iterations, calculates the difference with the current noise information, and then converts these differences into actual adjustments to the image, gradually reducing random noise until a predicted object image is generated. This predicted object image contains both the specific object features of the object to be displayed and is placed on a transparent background to facilitate subsequent superposition operations with the initial scene image.

[0083] As an optional implementation, an image processing model is used to take the initial object image as visual reference information to predict initial noise information to obtain predicted noise information, including: transforming the initial object image to obtain a transformed initial object image; and using the image processing model to take the transformed initial object image as visual reference information to predict initial noise information to obtain predicted noise information.

[0084] In this embodiment, before generating the predicted noise information, the initial object image needs to be transformed, and these transformations may include but are not limited to size scaling, directional cropping, color adjustment, etc. The purpose of this transformation is to provide the image processing model with richer visual information and help the image processing model better understand the characteristics of the object to be displayed. Especially when processing high-resolution images, appropriate transformations can retain more details and improve the fidelity of the generated image.

[0085] Optionally, after obtaining the transformed initial object image, the transformed initial object image will become visual reference information of the image processing model to predict the initial noise information and obtain predicted noise information. That is, the image processing model will use the transformed initial object image as a visual reference to predict the initial noise information and obtain predicted noise information.

[0086] In this step, the initial object image is transformed to obtain a transformed initial object image. Since the changed initial object image can include richer visual information of the object to be displayed, the transformed initial object image is used as visual reference information of the image processing model to help the image processing model better understand the characteristics of the product. Especially when processing high-resolution images, appropriate transformation can retain more details and improve the fidelity of the generated image.

[0087] As an optional implementation, the initial object image is transformed to obtain a transformed initial object image, including: scaling the size of the initial object image and cropping the initial object image from different cropping directions; and determining the scaled initial object image and the cropped initial object image as the transformed initial object image.

[0088] In this embodiment, the transformation operation on the initial object image may include but is not limited to: a resizing operation and a cropping operation in different directions. Among them, resizing refers to changing the size of the initial object image to make it smaller or larger. The resizing operation on the initial object image is mainly to adapt to the input requirements of the image processing model while retaining the visual details of the object to be displayed. Resizing can help the image processing model avoid detail loss when processing high-resolution images and ensure the quality of the generated image. Directional cropping refers to cropping from different directions of the initial object image to obtain local details of the initial object image. This includes but is not limited to cropping the initial object image from four directions: upper left, lower left, upper right, and lower right. The purpose of directional cropping is to provide views of the object to be displayed from different perspectives, to help the image processing model to more comprehensively understand the shape and texture of the object to be displayed, so that these details can be more accurately restored in the process of generating the predicted object image.

[0089] Optionally, when performing a transformation operation on the initial object image, the initial object image may be expanded into five object groups, wherein the five object groups may include: the initial object image after resizing, and images obtained by cropping the initial object image in four directions (e.g., the upper left corner, the lower left corner, the upper right corner, and the lower right corner). After obtaining the five object groups, the five object groups may be input into the image processing model as new initial object images. Compared with the method of directly resizing the initial object image, this processing method can retain more detailed information.

[0090] In this step, by scaling the size of the initial object image and cropping the initial object image from different cropping directions, and determining the scaled initial object image and the cropped initial object image as the transformed initial object image, the image processing model can learn the features of the object to be displayed under different perspectives, and thus the object to be displayed in the predicted object image finally generated by the image processing model can be adaptively adjusted according to the perspective of the scene.

[0091] As an optional implementation, an image processing model is used to take the transformed initial object image as visual reference information to predict initial noise information to obtain predicted noise information, including: encoding the transformed initial object image to obtain object features of the initial object image; marking the object features to obtain marking results, wherein the marking results include detailed information of the transformed initial object image; and using the image processing model to take the marking results as visual reference information to predict initial noise information to obtain predicted noise information.

[0092] In this embodiment, after obtaining the transformed initial object image, each transformed initial object image can be encoded by a pre-trained image encoder (e.g., DINO-V2 model) to obtain the object features of the initial object image, wherein the size of the object feature can be 325x1024, and such a size design is to ensure that the image processing model can capture sufficient detail information while maintaining the compactness and computational efficiency of the feature. Generally, a larger feature size can contain richer image details and contextual information, which helps the image processing model to more accurately restore key features such as shape, texture, and color of the object to be displayed in the process of generating the predicted object image.

[0093] Optionally, after obtaining the object features of the initial object image, the object features can be marked. The marking process is to further process the key information in the object features. The marking here can be to convert the object features into another form of representation through a specific neural network layer (such as Multi-Layer Perceptron, MLP layer for short). The marking result contains detailed information of the transformed initial object image, such as the specific shape, texture and color of the object to be displayed. This information is crucial for generating and predicting the object image.

[0094] For example, from the above introduction, it can be seen that by using the pre-trained image encoder to encode each transformed initial object image, an object feature of size 325x1024 can be obtained. That is, each transformed initial object image is encoded as an object feature of size 325x1024. Based on this, these object features are stacked and converted into a 325x768 labeling result through the MLP layer. The labeling result can effectively integrate information from different initial object images, thereby utilizing multi-perspective product details in the generation process to improve the quality and fidelity of the generated image.

[0095] Optionally, after obtaining the labeling result, the image processing model is used to use the labeling result as visual reference information to predict the initial noise information to obtain the predicted noise information.

[0096] As an optional implementation, the image generation method also includes: determining a joint feature between the mask image and a background image of the object to be displayed, wherein the image content of the background image includes the background corresponding to the object to be displayed in the initial scene image; inputting the joint feature into an image processing model; wherein, using the image processing model to use the initial object image as visual reference information of the mask image to predict a predicted object image corresponding to the mask image, including: using the image processing model to use the initial object image as visual reference information of the joint feature to predict a predicted object image.

[0097] In this embodiment, a variational autoencoder (VAE) can be used to extract features from the mask image and the background image respectively, and then the features of the two images are fused to obtain a joint feature. This joint feature contains not only the position information of the object to be displayed (mask image feature), but also the details of the background environment (background image feature).

[0098] Optionally, after the joint feature is determined, the joint feature will be input into an image processing model, wherein the image processing model may be a pre-trained diffusion model that naturally integrates the object to be displayed into the scene image. The image processing model will use this joint feature, and the initial object image as visual reference information of the mask image, to generate a predicted object image, wherein the joint feature effectively integrates the information of the mask image and the background image, helping the image processing model to understand the environment and position corresponding to the object to be displayed.

[0099] Optionally, the joint feature includes position information of the mask image and environmental information of the background image. By using the initial object image as visual reference information of the joint feature, the image processing model can match the object features of the object to be displayed with the position and background environment in other scenes when generating the predicted object image, thereby ensuring that the generated predicted object image looks natural at a specific position while maintaining the details and visual characteristics of the object to be displayed.

[0100] Through the above steps, a predicted object image can be generated, which includes the object to be displayed on a transparent background. By superimposing the predicted object image on the initial scene image, the implantation effect of the object to be displayed in the initial object scene can be made more natural. Compared with the direct use of the cutout technology, there will be no problems of cutout defects and missing shadows after cutout, and the details of the object to be displayed can be better preserved, while optimizing the light and shadow effects.

[0101] Furthermore, by providing the position information of the object to be displayed in the initial scene through the mask image, the position of the object to be displayed in the target scene can be accurately controlled when generating the target scene image, avoiding the unreasonable layout problem caused by random generation, and providing users with more accurate product display location selection.

[0102] Under the above operating environment, this application provides Figure 3 The method for determining the model shown. Figure 3 FIG. 1 is a flow chart of a method for determining a model according to an embodiment of the present application. Figure 3 As shown, the method may include the following steps:

[0103] Step S301, obtaining an object image sample.

[0104] In the technical solution provided in the above step S301 of the present application, the image content of the object image sample includes at least one display object, wherein the at least one object to be displayed may be a commodity, furniture, etc., depending on the application scenario.

[0105] In this embodiment, when obtaining the object image samples, multiple images of the object to be displayed can be captured by an image acquisition device to obtain multiple object image samples. In order to obtain high-quality images, it is usually necessary to perform the operation in a specific environment, such as using uniform lighting, a clean background (such as a white background), and capturing the object from multiple angles, so as to collect enough and diverse samples.

[0106] Optionally, when obtaining the object image samples, the images of the objects to be displayed may also be directly obtained from the e-commerce platform, the Internet, or professional gallery. This is only an illustrative example and does not limit the method of obtaining the object image samples.

[0107] Step S302: training the diffusion model using the object image sample and the visual encoder to obtain an image processing model.

[0108] In the technical solution provided in the above step S302 of the present application, the image processing model is used to use the initial object image to be processed as the visual reference information of the mask image, and predict the predicted object image corresponding to the mask image, the image content of the initial object image includes at least one object to be displayed on a solid color background, the mask image is used to represent the image area to be displayed of the object to be displayed in the initial scene image, the image content of the predicted object image includes the object to be displayed on a transparent background, the object to be displayed on the transparent background is associated with the image area represented by the mask image, the predicted object image is used to be superimposed on the initial scene image to obtain the target scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image area.

[0109] In this embodiment, object image samples containing at least one object to be displayed are used as training data, and these object image samples generally show the object features of the object to be displayed on a solid color background (e.g., white). A visual encoder (e.g., DINO-V2) is used to extract the object features of these object image samples, and the extracted object features will serve as the basis for model training.

[0110] Optionally, after extracting the object features of the image sample, the diffusion model is trained using the extracted object features and the corresponding mask image. The mask image is a binary image that indicates the area or position where the object to be displayed should be displayed in the initial scene image. During the training process, the diffusion model learns how to accurately place the object to be displayed on a solid color background in the object image sample in the form of a transparent background into the area indicated by the mask image, while ensuring that the viewing angle and lighting effect of the object to be displayed match the initial scene image.

[0111] Optionally, after training, the diffusion model can predict and generate a predicted object image with a transparent background corresponding to the mask area based on the input initial object image and mask image. This prediction process requires the diffusion model to not only remove the background, but also adjust the object image according to the characteristics of the initial scene image, such as viewing angle, lighting, etc., so that it can be integrated with the initial scene image. Finally, the trained diffusion model is the image processing model.

[0112] In the above steps S301 to S302, the image processing model obtained by training the diffusion model to understand and process the relationship between the object image and the scene mask can generate a high-quality, naturally integrated predicted object image, which solves the technical problems of inaccurate position, mismatched perspective, and unnatural lighting effects that may occur when the image is implanted into the scene in the prior art, greatly improving the user experience and the visual effect of the object to be displayed implanted in the scene.

[0113] The above method of this embodiment is further introduced below.

[0114] As an optional implementation, the object image samples include initial object image samples under different viewing angles, and the image content of the initial object image samples includes at least one display object sample located on a solid color background sample. Step S302, using the object image samples and the visual encoder, trains the diffusion model to obtain the image processing model, including: transforming the initial object image samples to obtain transformed initial object image samples; and training the diffusion model using the transformed object image samples and the visual encoder to obtain the image processing model.

[0115] In this embodiment, the initial object image samples at different viewing angles are respectively used to display the appearance features of the object at different viewing angles, so as to provide a data basis for subsequent viewing angle adaptive processing.

[0116] Optionally, before training the diffusion model, the collected initial object image samples are transformed to generate transformed object image samples. The transformation may include but is not limited to: scaling, cropping in different directions, etc. The purpose of the transformation is to simulate and increase the diversity of perspectives so that the diffusion model can learn image features at different perspectives during the training process.

[0117] Optionally, the diffusion model is trained using the object image samples and the visual encoder. The visual encoder (e.g., DINO-V2) is used to extract visual features from the transformed object image samples. These object features include information such as the shape, texture, color, and feature performance under perspective changes. Based on these features and perspective information, the diffusion model learns how to generate an image showing the object with a transparent background, i.e., predict the object image, while ensuring that the generated image matches the scene area indicated by the mask image, and that the perspective and lighting effects are adaptive to the scene image.

[0118] Optionally, through the above training process, the diffusion model learns how to generate high-quality predicted object images based on the input perspective information and features. These predicted object images not only remove the solid color background, but also adjust the perspective and lighting of the displayed object to coordinate with the visual effect of the specified area in the specific scene image. The finally trained diffusion model can be used as an image processing model to process and generate perspective-adaptive transparent background object images, thereby realizing the natural implantation of objects in various scene images.

[0119] As an optional implementation, the initial object image samples are transformed to obtain transformed initial object image samples, including: scaling the size of the initial object image samples, and cropping the initial object image samples from different cropping directions; and determining the scaled initial object image samples and the cropped initial object image samples as transformed initial object image samples.

[0120] In this embodiment, the transformation operation on the initial object image may include, but is not limited to: a resizing operation and a cropping operation in different directions. Among them, resizing refers to changing the size of the initial object image to make it smaller or larger. The resizing operation on the initial object image is mainly to adapt to the input requirements of the image processing model while retaining the visual details of the object to be displayed. The cropping operation in different directions refers to cropping from different directions of the initial object image to obtain local details of the initial object image. For example, the initial object image is cropped from four directions: the upper left, the lower left, the upper right, and the lower right. The purpose of cropping the initial object image in different directions is to provide views of the object to be displayed from different perspectives, to help the image processing model to more comprehensively understand the shape and texture of the object to be displayed, so that these details can be more accurately restored in the process of generating the predicted object image.

[0121] Optionally, by determining the scaled initial object image samples and the cropped initial object image samples as the transformed initial object image samples, five object groups can be obtained, namely, the scaled initial object image, the initial object image cropped at the lower left corner, the initial object image cropped at the upper left corner, the initial object image cropped at the lower right corner, and the initial object image cropped at the upper right corner.

[0122] As an optional implementation, the diffusion model is trained using the transformed object image samples and the visual encoder to obtain the image processing model, including: using the visual encoder to encode the transformed initial object image samples to obtain object feature samples of the initial object image samples; calling a multilayer perceptron to label the object feature samples to obtain labeled result samples, wherein the labeled result samples include detailed information of the transformed initial object image samples; and using the labeled result samples to train the diffusion model to obtain the image processing model.

[0123] In this embodiment, after obtaining the transformed object image samples, the transformed initial object image samples can be input into the visual encoder. Since the visual encoder (such as DINO-V2) is a deep learning model, it can extract object feature samples from the initial image samples, where the object feature samples may include: the shape, texture, color and more abstract semantic information of the object. For the transformed object image samples, the visual encoder not only extracts the object feature samples in the initial object image samples, but also captures the feature changes under different viewing angles, which provides a basis for the subsequent viewing angle adaptive generation.

[0124] Optionally, after extracting object feature samples from the transformed initial image samples using a visual encoder, the extracted object feature samples can be input into a multilayer perceptron (e.g., MLP) for processing, wherein the multilayer perceptron is a feedforward neural network that can establish complex nonlinear relationships between features. The role of the multilayer perceptron here is to further label and transform the extracted object feature samples to generate labeled result samples containing richer detail information. These labeled result samples not only contain the object feature samples of the object, but may also incorporate perspective information, so that the diffusion model can understand the detailed performance of the object at different perspectives.

[0125] For example, suppose a visual encoder is used to extract features from the transformed initial object image, and object feature samples of size 325x1024 are obtained. These object feature samples are stacked and converted into labeled result samples of size 325x768 through a multi-layer perceptron (MLP). The labeled result samples can effectively integrate information from different object image samples, thereby utilizing multi-perspective image sample information in the generation process to improve the quality and fidelity of the generated image.

[0126] Optionally, after obtaining the labeled result samples, the diffusion model can be trained using the labeled result samples. The diffusion model is a generative model that can gradually recover image details from noise and generate high-quality images. In this training process, the diffusion model learns how to generate predicted object images corresponding to the labeled result samples but on a transparent background. These predicted object images visually match the masked areas in the initial scene image while maintaining the object's detail information and the characteristics of adaptive perspective.

[0127] Optionally, through the above training process, the diffusion model can understand how to generate a transparent background object image that matches a specific scene area and is perspective-adaptive based on the object's features and perspective information. The trained model is an image processing model, which can be used to process the initial object image at any perspective and generate a transparent background predicted object image that meets the scene requirements, facilitating subsequent scene fusion and display.

[0128] As an optional implementation, the object image sample includes a predicted object image sample, and the image content of the predicted object image sample includes at least one display object sample located on a transparent background sample. The method for determining the model also includes: encoding layers of different color channels in the predicted object image sample to obtain encoding features, and reconstructing the predicted object image sample based on the encoding features to obtain a reconstruction result; using the labeled result samples, training the diffusion model to obtain an image processing model, including: using the reconstruction results and the labeled result samples to train the diffusion model to obtain the image processing model.

[0129] In this embodiment, different color channels refer to independent channels corresponding to color information. A color image is usually composed of three main color channels, namely red, green and blue channels, referred to as RGB. Each channel independently stores the brightness information of the corresponding color, and when combined together, a complete color representation of the image is formed. Since the image content of the predicted object image sample includes at least one display object sample located on a transparent background sample, based on this, the predicted object image sample, in addition to the RGB channels, usually also includes a transparency (Alpha) channel, wherein the Alpha channel is used to record the transparency of each pixel in the image.

[0130] Optionally, since the RGB channel and the Alpha channel carry different visual information respectively, based on this, when encoding the predicted object image sample, the information of the RGB color channel and the Alpha channel of the predicted object image sample can be encoded respectively to obtain the encoded features, so as to convert the color and transparency information into a compact feature representation, which is convenient for model learning and image reconstruction.

[0131] For example, a variational autoencoder (VAE) can be used to encode the information of the RGB color channels and the alpha channel of the predicted object image sample to obtain the encoded features. However, since VAE is mainly used for image encoding and decoding of the RGB channel, when processing images with transparent backgrounds, additional processing of transparency information is required. Therefore, the model is extended to add an encoding layer specifically for encoding the alpha layer of the image, that is, the transparency information. This additional encoding layer converts the information of the alpha channel into a set of feature vectors that complement the feature vectors of the RGB channel.

[0132] Optionally, after obtaining the encoding features, the predicted object image samples can be reconstructed based on the encoding features to obtain a reconstruction result. For example, the image can be reconstructed by a decoder (usually the decoder part of the VAE) to restore the predicted object image samples. This process restores the RGB information of the image and takes into account the influence of the Alpha channel, so that the reconstructed predicted object image samples are as close as possible to the original predicted object image samples in color and details.

[0133] Optionally, after obtaining the reconstruction results, the diffusion model can be trained using the reconstruction results and the labeled result samples to obtain an image processing model. For example, the reconstruction results are compared with the labeled result samples to evaluate the accuracy and quality of the reconstruction. Any errors or omissions in the reconstruction process will be used for subsequent model optimization.

[0134] As an optional implementation, layers of different color channels in the predicted object image sample are encoded to obtain encoding features, including: on the encoding layer, encoding the layer of the transparency channel in the predicted object image sample, and encoding the layer of the color information channel in the predicted object image sample to obtain encoding features.

[0135] In this embodiment, the coding layer is a coding layer added in the variational autoencoder (VAE) to encode the layer of the transparency channel (Alpha channel) and the layer of the color information channel (RGB channel). The layer of the transparency channel in the prediction object image sample is used to provide the transparency information of the prediction object image sample, and the layer of the color information channel in the prediction object image sample is used to provide the three basic color information of red, green and blue in the prediction object image sample. On the coding layer, through a specific neural network structure (such as a convolutional layer, a fully connected layer, etc.), the layer of the Alpha channel in the prediction object image sample is encoded into a group of coding features. This group of coding features is used to represent the details such as the boundary of the object in the prediction object image sample and the distribution of the transparent area, which is the key to reconstructing the transparency information of the image. Similarly, on the coding layer, the layer of the RGB channel of the prediction object image sample is also encoded into a group of coding features. This group of coding features is used to represent the visual attributes such as the color distribution, texture, brightness, etc. of the prediction object image sample, which is the key to reconstructing the color information of the image.

[0136] In this step, two different sets of coded features are obtained by encoding the layers of the Alpha channel and the RGB channel respectively. The coded features of the Alpha channel focus on the transparency and boundary information of the image, while the coded features of the RGB channel focus on color and details. These two sets of coded features together constitute the coded features of the object image sample, which will be used in the subsequent image reconstruction process to generate an image that is highly similar to the original predicted object image sample.

[0137] As an optional implementation, the determination method of the model also includes: determining the premultiplied color image corresponding to the layer of the color information channel; calling the variational autoencoder to encode the premultiplied color image to obtain latent features; reconstructing the predicted object image sample based on the encoded features to obtain a reconstruction result, including: obtaining the feature sum between the latent features and the encoded features; calling the decoder corresponding to the variational autoencoder to decode the feature sum to obtain a reconstructed premultiplied color image of the premultiplied color image; at the decoding layer, decoding the reconstructed premultiplied color image and the feature sum to obtain a reconstruction result.

[0138] In this embodiment, the premultiplied color image (or premultiplied Alpha image) is obtained by multiplying the color information (RGB channel data) of the prediction object image sample with its transparency information (Alpha channel data). In the prediction object image sample, the Alpha channel is used to represent the opacity of each pixel, and the value range is usually between 0 and 1, where 0 represents that the pixel is completely transparent and 1 represents that the pixel is completely opaque. The premultiplication process is to multiply the RGB color value with the value of the Alpha channel, so that in the transparent area of ​​the image, the RGB color value will also be reduced accordingly, while the color value of the completely opaque part remains unchanged. The generation of the premultiplied color image helps to handle transparency more accurately during image synthesis, and avoid edge flickering or unnatural colors when images are superimposed.

[0139] Optionally, after determining the pre-multiplied color image corresponding to the layer of the color information channel, a variational autoencoder (VAE) can be called to encode the pre-multiplied color image to obtain latent features. Since VAE is a generative model, it can learn the potential distribution of data and compress high-dimensional image data into low-dimensional latent feature vectors. In this process, the encoder part of the VAE receives the pre-multiplied color image as input, and converts the image into latent features through a series of convolution operations or fully connected layers, etc. These latent features capture key information such as the structure, color, and transparency of the image.

[0140] Optionally, after encoding the premultiplied color image to obtain latent features, the latent features are added to the encoded features obtained by encoding the transparency channel to obtain the feature sum between the two. The purpose of this operation is to integrate the color and transparency information of the image to ensure that in the subsequent reconstruction process, the model can simultaneously restore the color and transparency details of the image. The feature sum after the addition operation contains more comprehensive image information and is the key to reconstructing high-quality images.

[0141] Optionally, after obtaining the feature sum between the latent features and the encoded features, the decoder corresponding to the variational autoencoder (VAE) can be called to decode the feature sum to obtain a reconstructed premultiplied color image of the premultiplied color image, wherein the decoder restores the low-dimensional features to high-dimensional image data by gradually amplifying and refining the information stored in the feature sum.

[0142] Optionally, at the decoding layer of the VAE, the reconstruction result of the predicted object image sample is recovered from the reconstructed premultiplied color image and the feature sum through a decoding operation. This process not only reconstructs the premultiplied color image, but also decodes the feature sum into a complete image including color and transparency. The decoder will generate a reconstructed image that matches the original predicted object image sample as much as possible in color and transparency based on the information carried in the feature sum.

[0143] In this step, through a series of encoding, feature integration and decoding steps, the image processing model can learn how to efficiently process and restore images containing transparency information, which is particularly critical for generating high-quality transparent background images and achieving natural image implantation. The entire process not only improves the quality of image reconstruction, but also ensures the accurate restoration of image transparency, allowing the generated image to blend naturally with various backgrounds, improving visual effects and user experience.

[0144] As an optional implementation, the method for determining the model also includes: obtaining mask image samples, and replacing the initial object image samples with target object image samples, wherein the mask image samples are used to represent the image area samples to be displayed in the initial scene image samples, and the viewing angle corresponding to the display object samples in the initial object image samples is different from the viewing angle corresponding to the display object samples in the target object image samples; using the reconstruction results and the labeled result samples, training the diffusion model to obtain the image processing model, including: using the mask image samples, the background image samples corresponding to the mask image samples, the target object image samples, the predicted object image samples, the reconstruction results and the labeled result samples with the initial noise information samples added, to train the diffusion model to obtain the image processing model, wherein the image content of the background image samples includes the background corresponding to the display object samples in the initial scene image samples, and the image processing model is also used to output the target noise information samples corresponding to the mask image samples and the labeled result samples, and the loss between the target noise information samples and the initial noise information samples is less than the loss threshold.

[0145] In this embodiment, the mask image is a binary image used to specify the specific area / position where the target object should be displayed in the scene image. During the training process, the mask image sample is used to prompt the diffusion model to display the position of the object sample in the initial scene image sample, that is, the mask image sample is generally used to provide a positioning reference for the diffusion model.

[0146] Optionally, the target object image sample can be a single white background image at an arbitrary perspective. The object perspective in the initial object image sample is different from that in the target object image sample, which means that during the model training process, the diffusion model needs to learn how to observe the same object from different perspectives and adaptively adjust the object perspective in the final output to meet the requirements of the scene.

[0147] Optionally, the diffusion model is a generative model, which is commonly used for image generation and image editing. During the model training process, the diffusion model receives mask image samples, background image samples corresponding to the mask image samples, target object image samples, predicted object image samples (i.e., the prediction results of the model), and reconstruction results and labeled result samples with the addition of initial noise information samples, and then trains the diffusion model based on these samples to obtain an image processing model, wherein the background image samples are used to provide background information of the scene image, which helps the diffusion model learn to generate object images that are harmonious with the background. At the same time, the predicted object image samples are combined with certain initial noise information samples to simulate the uncertainty in the image generation process.

[0148] Optionally, during the training process, the diffusion model will learn how to gradually remove noise and gradually convert the samples with initial noise information into clear target object images. The training goal is to reduce the difference (noise loss) between the image output by the model (i.e., the image corresponding to the mask and labeling result) and the labeling result sample to less than a preset loss threshold. The target noise information sample here actually refers to the noise information that needs to be removed during the generation process of the diffusion model in order to achieve a result close to the labeling result.

[0149] As an optional implementation, a diffusion model is trained using mask image samples, background image samples corresponding to the mask image samples, target object image samples, predicted object image samples, reconstruction results of samples with initial noise information added, and labeled result samples to obtain an image processing model, including: adjusting the channel structure of the diffusion model according to the features of the mask image samples, the features of the background image samples, and the features of the predicted object image samples, wherein the adjusted channel structure of the image processing model matches the features of the mask image samples, the features of the background image samples, and the features of the predicted object image samples; training the adjusted diffusion model using mask image samples, background image samples, target object image samples, predicted object image samples, reconstruction results of samples with initial noise information added, and labeled result samples; and training the trained diffusion model using a low-order adaptive large language model to obtain an image processing model.

[0150] In this embodiment, since the diffusion model is usually used to process single-channel or three-channel image data, based on this, if the diffusion model needs to process mask image samples, background image samples, and prediction object image samples, these samples may have different feature representations. In order to enable the diffusion model to effectively process and fuse these multi-type features, it is necessary to adjust the channel structure of the model to match the characteristics of the input samples.

[0151] Alternatively, if the features of the predicted object image samples occupy 4 channels, the features of the background image samples occupy 4 channels, and the features of the mask image samples occupy one channel, the diffusion model needs to be adjusted to a 9-channel structure.

[0152] Optionally, after adjusting the channel structure, the diffusion model can be trained. The training process includes using mask image samples, background image samples, target object image samples, predicted object image samples, and reconstruction results and labeled result samples with added initial noise information samples as training data to guide the diffusion model to learn how to recover the image of the target object from the noise and naturally merge it into a specified position of the background image (defined by the mask image samples).

[0153] Optionally, the Low-Order Adaptive Large Language Model (Lora) is a method for efficiently fine-tuning large pre-trained models. It adjusts model parameters by learning low-rank matrices, thereby reducing the computational resources and time required for fine-tuning. Based on this, the Lora model is used here to fine-tune the trained diffusion model, which can not only reduce the computational cost, but also further optimize the performance of the diffusion model, thereby improving the quality and accuracy of the images generated by the diffusion model in specific types of products and user scenarios.

[0154] In the above steps, by training the diffusion model and using the Lora model to fine-tune the trained diffusion model, the final image processing model can be equipped with the ability to process multi-channel image features and generate target object images that naturally blend into the background. This image processing model can be applied to scenes such as virtual product display and augmented reality (AR) object implantation, effectively generating high-quality images of users viewing product matching effects in their own homes, while the lightweight model ensures fast response and efficient operation on different devices.

[0155] Under the above operating environment, this application provides Figure 4 The method for generating the image shown. Figure 4 FIG. 1 is a flow chart of another method for generating an image according to an embodiment of the present application. Figure 4 As shown, the method may include the following steps:

[0156] Step S401, identifying an initial product image to be processed from an e-commerce platform.

[0157] In the technical solution provided in the above step S401 of the present application, the image content of the initial product image includes at least one product to be displayed on a solid color background.

[0158] In this embodiment, the user can select the product to be implanted with the initial scene image on the e-commerce platform. In response to the user's selection operation of a product on the e-commerce platform, the product can be determined as the product to be displayed, and an image of the product to be displayed located in a solid color background can be identified from the e-commerce platform, and the identified image can be determined as the initial product image.

[0159] Step S402: using the image processing model to take the initial product image as visual reference information of the mask image, and predicting a predicted product image corresponding to the mask image.

[0160] In the technical solution provided in the above step S402 of the present application, the image content of the predicted product image includes the object to be displayed on the transparent background, and the product to be displayed on the transparent background is associated with the image area represented by the mask image. Among them, the visual encoder can be a high-resolution visual encoder, and the diffusion model can be a Unet structure. The image processing model can be used to use the initial product image as the visual reference information of the mask image to predict the predicted product image corresponding to the mask image. The specific prediction process can refer to the introduction of the above step S202, which will not be repeated here.

[0161] Step S403: superimpose the predicted product image onto the initial scene image to obtain a target scene image.

[0162] In the technical solution provided in the above step S403 of the present application, the initial scene image can be a scene photo of the user's home, or any other scene photo. Superimposing the predicted product image on the initial scene image is to fuse the predicted product image with the initial scene image to obtain the target scene image, so as to realize the natural display of the product in the user's scene. The specific superposition process can be referred to the introduction of the above step S203, which will not be repeated here.

[0163] Step S404: returning the target scene image to the e-commerce platform.

[0164] In the technical solution provided in the above step S404 of the present application, after the target scene image is generated through the above step S404, the target scene image can be returned to the e-commerce platform, so that the user can preview the actual matching effect of the product in his own home on the e-commerce platform.

[0165] In this step, by allowing users to preview the effects of the product in the actual environment before purchase, users can be provided with more accurate expectations before purchase, reducing uncertainty and dissatisfaction with the actual effects of the product after purchase. The e-commerce platform can reduce the return rate, improve user satisfaction, and increase sales conversion rate.

[0166] In the above steps S401 to S404 of the present application, the initial product image is used as the visual reference information of the mask image through the image processing model to predict the predicted product image corresponding to the mask image, which can improve the accuracy of the predicted product image, and then the predicted product image is superimposed on the initial scene image to obtain the target scene image, which can realize the integration of the product to be displayed into the initial scene image selected by the user, provide the user with the matching effect of the product in the actual environment, and improve user satisfaction.

[0167] Under the above operating environment, this application provides Figure 5 The method for generating the image shown. Figure 5 FIG. 1 is a flowchart of another method for generating an image according to an embodiment of the present application. Figure 5 As shown, the method may include the following steps:

[0168] Step S501 : In response to an input instruction on the operation interface, an initial object image and a mask image to be processed are displayed on the operation interface.

[0169] In the technical solution provided in the above step S501 of the present application, the operation interface refers to the interface for the user to interact with the application, and the input instruction is used to indicate the input operation of the user on the operation interface. For example, the user uploads the initial object image and the mask image to be processed through the operation interface. Based on this, when responding to the input instruction acting on the operation interface, the initial object image and the mask image to be processed can be displayed on the operation interface. The image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to indicate the image area to be displayed in the initial scene image where the object to be displayed is located.

[0170] Step S502 , in response to an image generation instruction applied to the operation interface, a prediction target image corresponding to the mask image is displayed on the operation interface.

[0171] In the technical solution provided in the above step S502 of the present application, the predicted object image is obtained by using an image processing model to use the initial object image as visual reference information of the mask image, and the visual reference information is predicted. The image content of the predicted object image includes the object to be displayed located on the transparent background. The object to be displayed located on the transparent background is associated with the image area represented by the mask image. The image processing model is obtained by training a diffusion model using object image samples and a visual encoder. The image content of the object image samples includes at least one display object sample.

[0172] In this embodiment, after the initial object image and mask image to be processed are displayed on the operation interface, the user can issue an image generation instruction through the operation interface so that the image processing model generates a predicted object image based on the initial object image and mask image to be processed.

[0173] Step S503: displaying the target scene image corresponding to the predicted object image on the operation interface.

[0174] In the technical solution provided in the above step S503 of the present application, the target scene image is obtained by superimposing the predicted object image onto the initial scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image area.

[0175] In this embodiment, after the target scene image is obtained, the target scene image may be displayed on the operation interface for the user to view.

[0176] In the above-mentioned steps S501 to S503 of the present application, the initial object image and mask image to be processed are displayed in real time through the operation interface, and the user can intuitively see the object being processed and its expected position in the scene, which enhances the user's sense of participation and control. The user can adjust the position and viewing angle of the product through simple interface operations, and even make multiple rounds of modifications until they are satisfied. This interactivity greatly improves the user experience. And after generating the target scene image, the target scene image can be displayed on the operation interface, so that the user can preview the actual effect of the product in his home before purchasing the product. This visual feedback helps users make more accurate purchasing decisions, reduces hesitation and returns due to uncertainty about the actual effect of the product, thereby improving purchasing efficiency and satisfaction.

[0177] Under the above operating environment, this application provides Figure 6 The system for generating the image shown. Figure 6 FIG. 1 is a flow chart of a system for generating an image according to an embodiment of the present application. Figure 6 As shown, the image generation system 600 includes: a client 601 and a server 602 .

[0178] The client 601 is used to upload an initial object image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background.

[0179] Server 602 is used to use an image processing model to use the initial object image as visual reference information of the mask image to predict a predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, the mask image is used to represent the image area to be displayed in the initial scene image where the object to be displayed is located, and the image content of the predicted object image includes the object to be displayed located on a transparent background, and the object to be displayed located on the transparent background is associated with the image area represented by the mask image; the predicted object image is superimposed on the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area; and the target scene image is returned to the client for display.

[0180] In the image generation system, the user can upload the initial object image to be processed through the client. After receiving the initial object image uploaded by the client, the server can use the image processing model to use the initial object image as the visual reference information of the mask image, predict the predicted object image corresponding to the mask image, and then superimpose the predicted object image on the initial scene image to obtain the target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area; the target scene image is returned to the client for display, that is, through the interactive operation between the client and the server, the user can instantly see the fusion effect of the product and the scene on the client, thereby improving the user experience.

[0181] The technical solution of the embodiment of the present application is further introduced below with examples in combination with the preferred implementation manner.

[0182] In an augmented display or virtual reality scenario, users can take a photo of their home and use an AI algorithm based on a generative model to naturally embed products into the user scenario, allowing users to view the effects of product matching in their own homes. In order to achieve the superposition of products with different matching effects, it is necessary to make the white background image products transparent.

[0183] In the related art, a cutout solution is directly used to make a transparent layer of a white-background product, and then the transparent layer is pasted into the background scene. However, due to the lack of shadow extraction capability due to direct cutout, the effect of implanting the product into the scene is poor.

[0184] Figure 7 is a schematic diagram of a cutout image according to an embodiment of the present application, such as Figure 7As shown in the figure, the transparent layering of the white background product image is processed by directly cutting out the image, which lacks the ability to extract shadows and causes the problem of missing the main body of the cutout. Transparent layering generation technology is usually associated with image decomposition, layer extraction, palette processing, and image cutout. Current excellent layering schemes include: PP-matting, arbitrary matting (Matting Anything), and transparent layer generation based on vision transformer (ViT Matte). These technologies are closely related to image semantic segmentation. The main technical process is summarized as follows: (1) Extract the foreground target in the image through the semantic segmentation model; (2) Refine the training of the main branch and detail branch based on the extracted foreground target; (3) Add a fusion module to add an alpha channel to fuse the details and the main body to synthesize a transparent foreground layer.

[0185] In related technologies, with the help of the development of diffusion technology, a technology for end-to-end transparent layer generation has also emerged. By adding a transparent image generation module to the existing SD model, the SD model supports transparent image generation capabilities. Based on the transparent image generation process, joint learning is constructed to simultaneously generate background and foreground to support multi-layer generation. This method controls the generation of targets through text.

[0186] Figure 8 is a schematic diagram of an end-to-end generation of a transparent layer according to an embodiment of the present application, such as Figure 8 As shown in Figure 1, the transparent image can be decomposed into a transparent layer and an RGB layer. Then, the transparent layer and the RGB layer are processed using ε to obtain the potential transparency and the premultiplied image. sd * The potential transparency and the premultiplied image are processed to obtain a parameter x, and the potential transparency is adjusted based on the parameter x to obtain the adjusted potential transparency. After obtaining the adjusted potential transparency, the parameter D is obtained based on the adjusted potential transparency. sd * , based on the parameter D sd * The premultiplied image is decoded to obtain parameter D, the transparent layer and the RGB layer are decoded based on the parameter D, and then the transparent image is reconstructed according to the decoded transparent layer and RGB layer.

[0187] Fig. 9 is a schematic diagram of a model training process in an end-to-end generation of a transparent layer according to an embodiment of the present application, such as Fig. 9 As shown, the training process of the basic model and the training process of the layer model are demonstrated.

[0188] The disadvantages of this end-to-end transparent layer generation solution are that it lacks the ability to generate transparent layers based on reference images, and does not have the ability to specify the location and area where products are placed in the scene, resulting in random locations of products implanted in the scene, poor effect of generating transparent layers with shadows, and poor effect of adaptively adjusting the foreground perspective according to the background.

[0189] However, this application proposes a high-definition transparent layer generation method for indoor scenes, which focuses on solving the problem of naturally implanting products into photos provided by users in indoor scenes to view the matching effect of products and home environments. Based on the diffusion model, the innovative transparent layer VAE encoder-decoder module generates a complete, delicate and natural shadowed product transparent layer to ensure seamless integration of the product and the environment. At the same time, the high-definition white background image feature fusion module retains the high-resolution details of the product by performing quintuple preprocessing on the input white background image. Combined with the pre-trained DINO-V2 model, the generated layer has higher fidelity and better visual effects. In addition, the diffusion framework design of control position and adaptive viewing angle allows users to accurately control the placement of products and automatically adjust the product viewing angle to match the scene, greatly improving user experience and application flexibility. Overall, this technical solution provides the ability of position control and viewing angle adaptation while generating high-quality product transparent layers, bringing revolutionary interactive experience improvements to scenes such as home decoration and virtual shopping.

[0190] Fig.10 is a schematic diagram of a furniture implantation scenario according to an embodiment of the present application, such as Fig.10 As shown, the left side shows a white background product image, the middle side shows the effect of inserting furniture into the user scene based on the cutout method, and the right side shows the effect of inserting furniture into the user scene based on the high-definition transparent layer generation technology for indoor scenes in this application. Fig.10 It can be seen that compared with the method of directly cutting out and overlaying images to implant furniture into the user scene, the high-definition transparent layer generation technology used in this application to implant products into the user scene is more harmonious and natural, the light and shadow are more natural, and the main body of the product is complete. At the same time, the perspective of the implanted furniture can also be adaptively changed.

[0191] Next, the network structure of the embodiment of the present application is further introduced.

[0192] Fig.11 is a schematic diagram of a network structure of a diffusion model for generating a high-definition transparent layer according to an embodiment of the present application, such as Fig.11As shown in the figure, first, the original foreground layer (i.e., the product image) is encoded through the transparent layer VAE encoding layer. This process converts the image into a low-dimensional latent space representation. Subsequently, random noise is added to the encoded foreground layer to form a noisy foreground layer. At the same time, the mask image (used to indicate the position of the product in the scene) is encoded to obtain the encoded representation of the mask image. The encoded mask image and the noisy foreground layer are input into the diffusion model together. At the same time, the high-definition white background image of the product is used as additional visual reference information to guide the model generation to maintain the details and realism of the product.

[0193] The diffusion model plays a key role in this process. It predicts a noise distribution based on the input noisy foreground layer, the encoded mask image, and the visual reference of the product white background image. Then the model iterates and continuously subtracts the predicted noise from the random noise to gradually restore the predicted foreground layer.

[0194] Optionally, after multiple iterations and denoising, the model is able to generate a predicted foreground layer (i.e., transparent layer) that matches the position of the input mask image, adapts to the background environment in perspective, and is rich in details and naturally integrated.

[0195] Optionally, the network structure of the diffusion model for generating a high-definition transparent layer may include three modules: a transparent layer VAE encoder / decoder supporting transparent layer generation, a high-definition white background feature fusion module, and a diffusion framework for controlling position and adaptive viewing angle.

[0196] Next, these three modules will be further introduced.

[0197] VAE encoder / decoder for transparent graphs. Fig.12 is a schematic diagram of a transparent graph VAE encoder / decoder according to an embodiment of the present application, such as Fig.12 As shown in the figure, the transparent layer VAE encoder / decoder is an extension of the variational autoencoder (VAE) model included in the original SD. An additional encoding layer is added to encode the Alpha layer and the processed RGB layer. The encoded features are added to the latent features obtained after the pre-multiplied RGB layer passes through the VAE encoder. The added features are decoded by the original VAE decoder to obtain the corresponding reconstructed product image. At the same time, an additional decoding layer is added to decode the product image and the added features to reconstruct the corresponding transparent layer. Training this model only requires a transparent layer dataset. After training, the weights will be used in the diffusion model for direct reasoning, and the weights will no longer be updated.

[0198] For the high-definition white background image feature fusion module. When a white background reference image is required to control the generation, most solutions directly scale the white background product reference image to a fixed size, which will result in the loss of reference product details when the resolution of the white background product reference image is high, resulting in low fidelity of the generated product redrawing. Based on this, the application proposes a high-resolution visual encoder that expands the white background image into 5 object groups (e.g., the white background image after resizing, and the cropped white background image in four directions). Fig.13 is a schematic diagram of an object group obtained after a white background image is expanded according to an embodiment of the present application, such as Fig.13 As shown, by scaling the original high-definition white background image and cropping it from four different directions, namely, upper left, upper right, lower left, and lower right, five object groups corresponding to the original high-definition white background image can be obtained, and the five object groups can retain the detailed detail information of the original white background image.

[0199] Optionally, after obtaining the five object groups corresponding to the original high-definition white background image, the five object groups can be input into the diffusion model as reference images. Compared with the direct size scaling method, this expansion method can retain the detail information of the white background image.

[0200] Optionally, after the white background image is fed into the diffusion model, a pre-trained DINO-V2 model can be used as an image encoder. Each reference image will be encoded by the image encoder, producing object features of size 325x1024. These object features are stacked and then mapped to a 325*768 tag through an MLP layer. This tag contains information from multiple reference white background images and is subsequently injected into the diffusion process through a cross-attention layer within the U-Net.

[0201] A diffusion framework for controlling position and adaptive perspective. For position control, a mask image is used to provide position information. For perspective issues, the white background image is randomly replaced with other white background images of the same product with different perspectives by modifying the organizational structure of the dataset. At the same time, the model structure is improved. The joint features of the mask image and the background image are extracted based on the VAE encoder, and are spliced ​​with the original foreground features and background features into 9-channel feature outputs. The Unet model is also adjusted to a 9-channel structure. During training, a single white background image, foreground image, background image, and mask position map of random perspective are input, and the model generates a layer that is consistent with the real correct perspective foreground layer. For testing, a single white background image, background image, and mask position map of random perspective are input, and the model generates a harmonious perspective product foreground layer that matches the background and position.

[0202] In the method for generating a high-definition commodity transparent layer for an indoor scene provided in an embodiment of the present application, a high-resolution visual encoder, when using a white background image as a reference image as the input of the model, can retain the detail information of the white background image compared to the direct size scaling method, and determines a model for generating a transparent layer that controls the position and adaptively adjusts the viewing angle. Compared to the current cutout solution, there will be no problems of cutout defects and missing shadows after cutout. At the same time, the generated layer has a natural effect after being mapped with the background, and the layering solution is also convenient for manual adjustment of the position and matching and superposition.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0204] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0205] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0206] According to an embodiment of the present application, a device for generating an image for implementing the above-mentioned method for generating an image is also provided, such as Fig.14 As shown, the image generating device 1400 includes: a determination module 1401 , a first prediction module 1402 and a first superposition module 1403 .

[0207] The determination module 1401 is used to determine the initial object image and mask image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to indicate the image area where the object to be displayed is to be displayed in the initial scene image.

[0208] A first prediction module 1402 is used to use an image processing model to take the initial object image as visual reference information of the mask image, and predict a predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, and the image content of the predicted object image includes an object to be displayed on a transparent background, and the object to be displayed on the transparent background is associated with an image region represented by the mask image;

[0209] The first superposition module 1403 is used to superimpose the predicted object image onto the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area.

[0210] It should be noted that the above-mentioned determination module 1401, the first prediction module 1402 and the first superposition module 1403 correspond to steps S201 to S203 in the above-mentioned embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned embodiment. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of a device and may run in the server 10 provided in the above-mentioned embodiment.

[0211] According to an embodiment of the present application, a model determination device for implementing the above-mentioned model determination method is also provided, such as Fig.15 As shown, the model determination device 1500 includes: an acquisition module 1501 and a training module 1502.

[0212] An acquisition module 1501 is used to acquire an object image sample;

[0213] The training module 1502 is used to train the diffusion model using the object image samples and the visual encoder to obtain the image processing model, wherein the image processing model is used to use the initial object image to be processed as the visual reference information of the mask image, and predict the predicted object image corresponding to the mask image, the image content of the initial object image includes at least one object to be displayed on a solid color background, the mask image is used to represent the image area to be displayed in the initial scene image where the object to be displayed is located, the image content of the predicted object image includes the object to be displayed on a transparent background, the object to be displayed on the transparent background is associated with the image area represented by the mask image, the predicted object image is used to be superimposed on the initial scene image to obtain the target scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image area.

[0214] It should be noted that the acquisition module 1501 and the training module 1502 correspond to steps S301 to S302 in the above embodiment, and the examples and application scenarios implemented by the two modules and the corresponding steps are the same, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0215] According to an embodiment of the present application, a device for generating an image for implementing the above-mentioned method for generating an image is also provided, such as Fig.16 As shown, the image generating device 1600 includes: a recognition module 1601 , a second prediction module 1602 , a second superposition module 1603 and a return module 1604 .

[0216] An identification module 1601 is used to identify an initial product image to be processed from an e-commerce platform, wherein the image content of the initial product image includes at least one product to be displayed on a solid color background;

[0217] The second prediction module 1602 is used to use the image processing model to take the initial product image as visual reference information of the mask image, and predict the predicted product image corresponding to the mask image, wherein the image processing model is obtained by training the diffusion model using product image samples and a visual encoder, the mask image is used to represent the image area to be displayed in the initial scene image where the product to be displayed is located, and the image content of the predicted product image includes the product to be displayed located on a transparent background, and the product to be displayed located on the transparent background is associated with the image area represented by the mask image;

[0218] The second superposition module 1603 is used to superimpose the predicted product image onto the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the product to be displayed in the image area;

[0219] The returning module 1604 is used to return the target scene image to the e-commerce platform.

[0220] It should be noted that the identification module 1601, the second prediction module 1602, the second superposition module 1603 and the return module 1604 correspond to steps S401 to S404 in the above embodiment, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be part of the device and can be run in the server 10 provided in the above embodiment.

[0221] According to an embodiment of the present application, a device for generating an image for implementing the above-mentioned method for generating an image is also provided, such as Fig.17 As shown, the image generating device 1700 includes: a first display module 1701 , a second display module 1702 and a third display module 1703 .

[0222] The first display module 1701 is used to respond to the input instruction on the operation interface and display the initial object image and the mask image to be processed on the operation interface, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to indicate the image area to be displayed in the initial scene image of the object to be displayed;

[0223] The second display module 1702 is used to respond to the image generation instruction acting on the operation interface and display the predicted object image corresponding to the mask image on the operation interface, wherein the predicted object image is obtained by using the image processing model to use the initial object image as the visual reference information of the mask image, and the visual reference information is predicted, the image content of the predicted object image includes the object to be displayed on the transparent background, and the object to be displayed on the transparent background is associated with the image area represented by the mask image, and the image processing model is obtained by training the diffusion model using the object image sample and the visual encoder;

[0224] The third display module 1703 is used to display the target scene image corresponding to the predicted object image on the operation interface, wherein the target scene image is obtained by superimposing the predicted object image onto the initial scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image area.

[0225] It should be noted that the first display module 1701, the second display module 1702 and the third display module 1703 correspond to steps S501 to S503 in the above embodiment, and the three modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the server 10 provided in the above embodiment as part of the device.

[0226] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in the above embodiments, as well as the application scenario and implementation process, but is not limited to the scheme provided in the above embodiments.

[0227] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0228] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0229] In this embodiment, the computer terminal can execute the program code in the method.

[0230] Optionally, Fig.18 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.18 As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 1802, a memory 1804, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0231] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the electronic device A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0232] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: determine the initial object image and mask image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to represent the image area to be displayed of the object to be displayed in the initial scene image; use the image processing model to use the initial object image as visual reference information of the mask image to predict the predicted object image corresponding to the mask image, wherein the image processing model is obtained by training the diffusion model using the object image samples and the visual encoder, the image content of the predicted object image includes the object to be displayed on a transparent background, and the object to be displayed on the transparent background is associated with the image area represented by the mask image; superimpose the predicted object image on the initial scene image to obtain the target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area.

[0233] Optionally, the processor may also execute the program code of the following steps: inputting initial noise information into the image processing model; using the image processing model to use the initial object image as visual reference information to predict the initial noise information to obtain predicted noise information; and generating a predicted object image based on the predicted noise information and the initial noise information.

[0234] Optionally, the processor may also execute program codes of the following steps: obtaining difference noise information between the predicted noise information and the initial noise information; and converting the difference noise information into a predicted object image.

[0235] Optionally, the processor may also execute program code of the following steps: transforming the initial object image to obtain a transformed initial object image; using an image processing model to use the transformed initial object image as visual reference information to predict initial noise information to obtain predicted noise information.

[0236] Optionally, the processor may also execute program code of the following steps: scaling the size of the initial object image, and cropping the initial object image from different cropping directions; and determining the scaled initial object image and the cropped initial object image as the transformed initial object image.

[0237] Optionally, the processor may also execute the program code of the following steps: encoding the transformed initial object image to obtain object features of the initial object image; marking the object features to obtain marking results, wherein the marking results include detail information of the transformed initial object image; using the image processing model to use the marking results as visual reference information to predict the initial noise information to obtain predicted noise information.

[0238] Optionally, the processor may also execute program code for the following steps: determining a joint feature between a mask image and a background image of an object to be displayed, wherein the image content of the background image includes the background corresponding to the object to be displayed in the initial scene image; inputting the joint feature into an image processing model; wherein, using the image processing model to use the initial object image as visual reference information of the mask image to predict a predicted object image corresponding to the mask image, including: using the image processing model to use the initial object image as visual reference information of the joint feature to predict a predicted object image.

[0239] The embodiment of the present application provides a method for generating an image. The image processing model is used to effectively capture the detail information in the initial object image, and the mask image is used to specify the position and area where the object to be displayed is placed in the initial scene, so that the generated image effect is more harmonious and natural, thereby solving the technical problem of poor image effect generated by using the cutout and overlay method in the related art.

[0240] It will be understood by those skilled in the art that the structure shown in the figure is for illustration only, and the electronic device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, and a mobile Internet device (Mobile Internet Devices, referred to as MID), PAD, etc. The figure does not limit the structure of the above-mentioned electronic device. For example, the electronic device A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a configuration different from that shown in the figure.

[0241] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0242] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0243] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0244] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for executing the following steps: determining an initial object image and a mask image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to represent the image region to be displayed in the initial scene image where the object to be displayed is located; using an image processing model to use the initial object image as visual reference information of the mask image, and predicting a predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, and the image content of the predicted object image includes the object to be displayed on a transparent background, and the object to be displayed on the transparent background is associated with the image region represented by the mask image; and superimposing the predicted object image onto the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image region.

[0245] Optionally, the computer readable storage medium is further configured to store program codes for executing the following steps:

[0246] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0247] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0248] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0249] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0250] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0251] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0252] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0253] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0254] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating an image, characterized in that: include: Determining an initial object image and a mask image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to indicate an image region in the initial scene image where the object to be displayed is to be displayed; Using an image processing model to take the initial object image as visual reference information of the mask image, predicting a predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, and the image content of the predicted object image includes the object to be displayed on a transparent background, and the object to be displayed on the transparent background is associated with the image area represented by the mask image; The predicted object image is superimposed on the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area.

2. The method according to claim 1, characterized in that: The initial object image is used as visual reference information of the mask image by using an image processing model to predict a predicted object image corresponding to the mask image, including: Inputting initial noise information into the image processing model; Using the image processing model to take the initial object image as the visual reference information, the initial noise information is predicted to obtain predicted noise information; The prediction target image is generated based on the predicted noise information and the initial noise information.

3. The method according to claim 2, characterized in that Generating the predicted object image based on the predicted noise information and the initial noise information includes: Acquire difference noise information between the predicted noise information and the initial noise information; The difference noise information is converted into the prediction target image.

4. The method according to claim 2, characterized in that: The initial object image is used as the visual reference information by using the image processing model to predict the initial noise information to obtain predicted noise information, including: Transforming the initial object image to obtain a transformed initial object image; The image processing model is used to take the transformed initial object image as the visual reference information to predict the initial noise information to obtain the predicted noise information.

5. The method according to claim 4, characterized in that Transforming the initial object image to obtain the transformed initial object image includes: Performing a scaling operation on the size of the initial object image, and performing a cropping operation on the initial object image from different cropping directions respectively; The scaled initial object image and the cropped initial object image are determined as the transformed initial object image.

6. The method according to claim 4, characterized in that The method comprises: using the image processing model to use the transformed initial object image as the visual reference information to predict the initial noise information to obtain the predicted noise information, including: Encoding the transformed initial object image to obtain object features of the initial object image; Marking the object features to obtain a marking result, wherein the marking result includes detailed information of the transformed initial object image; The image processing model is used to take the labeling result as the visual reference information to predict the initial noise information to obtain the predicted noise information.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Determining a joint feature between the mask image and a background image of the object to be displayed, wherein the image content of the background image includes the background corresponding to the object to be displayed in the initial scene image; inputting the joint features into the image processing model; Among them, using the image processing model to take the initial object image as the visual reference information of the mask image to predict the predicted object image corresponding to the mask image includes: using the image processing model to take the initial object image as the visual reference information of the joint feature to predict the predicted object image.

8. A method for determining a model, characterized in that: include: Obtaining an object image sample; The diffusion model is trained using the object image samples and the visual encoder to obtain an image processing model, wherein the image processing model is used to use the initial object image to be processed as the visual reference information of the mask image, and predict the predicted object image corresponding to the mask image, the image content of the initial object image includes at least one object to be displayed on a solid color background, the mask image is used to represent the image area to be displayed in the initial scene image of the object to be displayed, the image content of the predicted object image includes the object to be displayed on a transparent background, the object to be displayed on the transparent background is associated with the image area represented by the mask image, the predicted object image is used to be superimposed on the initial scene image to obtain a target scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image area.

9. The method according to claim 8, characterized in that The object image samples include initial object image samples at different viewing angles, and the image content of the initial object image samples includes at least one display object sample located on a pure color background sample. The diffusion model is trained using the object image samples and a visual encoder to obtain an image processing model, including: Transforming the initial object image sample to obtain the transformed initial object image sample; The diffusion model is trained using the transformed object image samples and the visual encoder to obtain the image processing model.

10. The method according to claim 9, characterized in that Transforming the initial object image sample to obtain the transformed initial object image sample includes: Performing a scaling operation on the size of the initial object image sample, and performing a cropping operation on the initial object image sample from different cropping directions respectively; The scaled initial object image samples and the cropped initial object image samples are determined as the transformed initial object image samples.

11. The method according to claim 9, characterized in that The diffusion model is trained using the transformed object image sample and the visual encoder to obtain the image processing model, including: Using the visual encoder, encoding the transformed initial object image sample to obtain an object feature sample of the initial object image sample; Calling a multilayer perceptron to label the object feature samples to obtain labeling result samples, wherein the labeling result samples include detailed information of the transformed initial object image samples; The diffusion model is trained using the labeled result samples to obtain the image processing model.

12. The method according to claim 11, characterized in that The object image sample includes a predicted object image sample, and the image content of the predicted object image sample includes at least one display object sample located on a transparent background sample. The method further includes: Encoding layers of different color channels in the prediction object image sample to obtain encoding features, and reconstructing the prediction object image sample based on the encoding features to obtain a reconstruction result; Using the labeled result samples to train the diffusion model to obtain the image processing model includes: using the reconstruction results and the labeled result samples to train the diffusion model to obtain the image processing model.

13. The method according to claim 12, characterized in that Encoding the layers of different color channels in the prediction object image sample to obtain encoding features, including: On the coding layer, a layer of a transparency channel in the prediction object image sample is encoded, and a layer of a color information channel in the prediction object image sample is encoded to obtain the coding feature.

14. The method according to claim 13, characterized in that The method further comprises: Determine a premultiplied color image corresponding to the layer of the color information channel; Calling a variational autoencoder to encode the premultiplied color image to obtain latent features; Reconstructing the predicted object image sample based on the encoded feature to obtain a reconstruction result, including: obtaining the feature sum between the latent feature and the encoded feature; calling the decoder corresponding to the variational autoencoder to decode the feature sum to obtain a reconstructed premultiplied color image of the premultiplied color image; on the decoding layer, decoding the reconstructed premultiplied color image and the feature sum to obtain the reconstruction result.

15. The method according to claim 12, characterized in that The method further comprises: Obtaining a mask image sample, and replacing the initial object image sample with a target object image sample, wherein the mask image sample is used to represent an image region sample to be displayed in the initial scene image sample, and the viewing angle corresponding to the display object sample in the initial object image sample is different from the viewing angle corresponding to the display object sample in the target object image sample; The diffusion model is trained using the reconstruction results and the labeled result samples to obtain the image processing model, including: using the mask image samples, the background image samples corresponding to the mask image samples, the target object image samples, the predicted object image samples, the reconstruction results with initial noise information samples added, and the labeled result samples to train the diffusion model to obtain the image processing model, wherein the image content of the background image samples includes the background corresponding to the display object samples in the initial scene image samples, and the image processing model is also used to output target noise information samples corresponding to the mask image samples and the labeled result samples, and the loss between the target noise information samples and the initial noise information samples is less than a loss threshold.

16. The method according to claim 15, characterized in that The diffusion model is trained using the mask image samples, the background image samples corresponding to the mask image samples, the target object image samples, the predicted object image samples, the reconstruction results with the initial noise information samples added, and the labeled result samples to obtain the image processing model, including: adjusting the channel structure of the diffusion model according to the features of the mask image samples, the features of the background image samples, and the features of the prediction object image samples, wherein the adjusted channel structure of the image processing model matches the features of the mask image samples, the features of the background image samples, and the features of the prediction object image samples; Training the adjusted diffusion model using the mask image samples, the background image samples, the target object image samples, the predicted object image samples, the reconstruction results to which the initial noise information samples are added, and the labeled result samples; The trained diffusion model is trained using a low-order adaptive large language model to obtain the image processing model.

17. A method for generating an image, characterized in that: include: Identifying an initial product image to be processed from an e-commerce platform, wherein the image content of the initial product image includes at least one product to be displayed on a solid color background; The initial product image is used as visual reference information of the mask image by using an image processing model to predict a predicted product image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using product image samples and a visual encoder, the mask image is used to represent an image region to be displayed in the initial scene image where the product to be displayed is located, and the image content of the predicted product image includes the product to be displayed located on a transparent background, and the product to be displayed located on the transparent background is associated with the image region represented by the mask image; superimposing the predicted product image onto the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the product to be displayed in the image area; The target scene image is returned to the e-commerce platform.

18. A method for generating an image, characterized in that: include: In response to an input instruction on the operation interface, an initial object image and a mask image to be processed are displayed on the operation interface, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background, and the mask image is used to indicate an image region in the initial scene image where the object to be displayed is to be displayed; In response to an image generation instruction acting on the operation interface, a predicted object image corresponding to the mask image is displayed on the operation interface, wherein the predicted object image is obtained by using an image processing model to use the initial object image as visual reference information of the mask image and predicting the visual reference information, the image content of the predicted object image includes the object to be displayed located on a transparent background, the object to be displayed located on the transparent background is associated with the image area represented by the mask image, and the image processing model is obtained by training a diffusion model using object image samples and a visual encoder; On the operation interface, a target scene image corresponding to the predicted object image is displayed, wherein the target scene image is obtained by superimposing the predicted object image onto the initial scene image, and the target scene image is used to represent the result of displaying the object to be displayed in the image area.

19. A system for generating an image, characterized in that: include: A client, used for uploading an initial object image to be processed, wherein the image content of the initial object image includes at least one object to be displayed on a solid color background; The server is used to use the image processing model to take the initial object image as visual reference information of the mask image, and predict the predicted object image corresponding to the mask image, wherein the image processing model is obtained by training a diffusion model using object image samples and a visual encoder, the mask image is used to represent the image area to be displayed in the initial scene image of the object to be displayed, and the image content of the predicted object image includes the object to be displayed located on a transparent background, and the object to be displayed located on the transparent background is associated with the image area represented by the mask image; the predicted object image is superimposed on the initial scene image to obtain a target scene image, wherein the target scene image is used to represent the result of displaying the object to be displayed in the image area; and the target scene image is returned to the client for display.

20. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 18 when running.

21. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 18.

22. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 18.

Images