Image generation method and system, electronic equipment and computer program product

Abstract

The invention discloses an image generation method and system, electronic equipment and a computer program product, and relates to the field of large model technology and image processing. The method can comprise the steps that an initial scene image to be processed is determined, and the image content of the initial scene image comprises at least one display object; determining an auxiliary object image to be drawn at a target position in the initial scene image; based on the auxiliary object image, the target position and the initial scene image, a posture image and an object mask image of the auxiliary display object are determined, the posture image is used for representing the posture of the auxiliary display object, the object mask image is used for representing an image area where the auxiliary display object is located in the initial scene image, and the image area comprises the target position; and analyzing the attitude image, the object mask image and the initial scene image by using the redrawing model to obtain a target scene image. The technical problem of low image generation accuracy is solved.

Description

Technical Field

[0001] The present application relates to large model technology and image processing fields, and more specifically, to an image generation method, system, electronic device and computer program product. Background Art

[0002] At present, with the continuous development and application of artificial intelligence technology, artificial intelligence (AI) models (i.e., auxiliary display objects) are becoming more and more popular among users. Users can select corresponding postures, attributes, etc. according to their needs, such as clothing, movements, gestures, etc., to generate AI models that meet personalized requirements.

[0003] In the related art, for the case where the AI ​​model is a human, when the human posture skeleton is used to control the model's posture, limb abnormalities often occur, which seriously affects the aesthetic quality of the generated image (that is, the target scene image), for example, the AI ​​model has three legs in the image. Specifically, there is still a lack of unified standards for the evaluation of limb distortion in AI-generated images, and it is often necessary to rely on human intuitive feedback to evaluate the naturalness of the limbs. The above-mentioned method of relying on manual intervention is not only inefficient, but also not conducive to improving the AI ​​model generation effect and the visual appeal of the target scene image, and there is a situation where the image generation effect is poor. Therefore, there is still a technical problem of low image generation accuracy.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, system, electronic device and computer program product for generating an image, so as to at least solve the technical problem of low accuracy in generating an image.

[0006] According to one aspect of an embodiment of the present application, a method for generating an image is provided. The method may include: determining an initial scene image to be processed, wherein the image content of the initial scene image includes at least one display object; determining an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the above display object; based on the auxiliary object image, the target position and the initial scene image, determining a posture image and an object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position; using a redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain a target scene image, wherein the redrawing model is obtained by reinforcement learning of the diffusion model.

[0007] According to another aspect of the embodiments of the present application, a method for determining a model is provided. The method may include: obtaining a preference data set, wherein the preference data set includes a pose image sample, an object mask image sample, and a target scene image sample pair generated based on the auxiliary display object sample, the pose image sample is used to represent the pose of the auxiliary display object sample, the object mask image sample is used to assist in the image region where the display object sample is located in the initial scene image sample, and the target scene image sample pair includes a target scene image sample whose evaluation result is higher than the evaluation result threshold, and a target scene image sample whose evaluation result is lower than the evaluation result threshold; using the preference data set, performing reinforcement learning on the diffusion model to obtain a redrawing model, wherein the redrawing model is used to The posture image of at least one auxiliary display object, the object mask image of the auxiliary display object and the initial scene image to be processed are analyzed to obtain a target scene image, the auxiliary display object is used to assist in displaying at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

[0008] According to another aspect of the embodiment of the present application, another method for generating an image is provided. The method may include: identifying an initial scene image to be processed from an e-commerce platform, wherein the image content of the initial scene image includes at least one display product; determining an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the above-mentioned display product; based on the auxiliary object image, the target position and the initial scene image, determining a posture image and an object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position; using a redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain a target scene image, wherein the redrawing model is obtained by reinforcement learning of the diffusion model; returning the target scene image to the e-commerce platform.

[0009] According to another aspect of the embodiment of the present application, another method for generating an image is provided. The method may include: in response to an input instruction acting on an operation interface, displaying an initial scene image to be processed on the operation interface, wherein the image content of the initial scene image includes at least one display object; in response to an image generation instruction acting on the operation interface, displaying a target scene image corresponding to the initial scene image on the operation interface, wherein the target scene image is obtained by analyzing a posture image of at least one auxiliary display object, an object mask image of the auxiliary display object, and the initial scene image using a redrawing model, the redrawing model is obtained by performing reinforcement learning on a diffusion model, the auxiliary display object is used to assist in displaying at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position, and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

[0010] According to another aspect of this embodiment of the present application, a system for generating an image is provided. The system may include: a client, for uploading an initial scene image to be processed, wherein the image content of the initial scene image includes at least one display object; a server, for determining an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the above display object; based on the auxiliary object image, the target position and the initial scene image, determining a posture image and an object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position; using a redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain a target scene image, wherein the redrawing model is obtained by reinforcement learning of a diffusion model; wherein the client is used to display the target scene image.

[0011] According to another aspect of an embodiment of the present application, an electronic device is also provided. The electronic device may include a memory and a processor: the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions. When the above-mentioned computer-executable instructions are executed by the processor, any one of the above-mentioned methods is implemented.

[0012] According to another aspect of an embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein any one of the above methods is executed when the program is running.

[0013] According to another aspect of an embodiment of the present application, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any of the above methods.

[0014] According to another aspect of the embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, and when the computer program is executed by a processor, the method of the embodiment of the present application is implemented.

[0015] In an embodiment of the present application, if it is necessary to better display the display object in the image, the interaction between the display object and the auxiliary display object can be reflected by adding an auxiliary display object to assist in displaying the display object, for example, using the display object. An initial scene image including the display object can be obtained. According to the needs of the user, the target position where the auxiliary object image needs to be drawn can be determined from the initial scene image, and the auxiliary object image that meets the user's needs for the posture, clothing, etc. of the auxiliary display object can be screened from a large number of auxiliary object images. The posture of the auxiliary display object can be determined based on the auxiliary object image, the target position and the initial scene image, and the posture image and the object mask image of the auxiliary display object are obtained. The hidden variables corresponding to the posture image, the object mask image and the initial scene image are input into the corresponding hidden variables into the redrawing model to integrate the target scene image. In the target scene image, the interaction between the auxiliary display object and the display object that meets the user's needs can be displayed, thereby improving the display effect of the display object.

[0016] In this embodiment, a redrawing model is set up to integrate human preferences into the AI ​​model generation process. The generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, and the visual effect and attractiveness of image generation are improved. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, so that image generation is more in line with user expectations and needs. In this way, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0017] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a schematic diagram of an application scenario of an image generation method according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for generating an image according to an embodiment of the present application;

[0021] Figure 3 is a flow chart of a method for determining a model according to an embodiment of the present application;

[0022] Figure 4 is a flow chart of another method for generating an image according to an embodiment of the present application;

[0023] Figure 5 is a flow chart of another method for generating an image according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of an image generation system according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of an interface for generating a scene promotion picture with a model according to an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of generating abnormal conditions based on a scene promotion diagram with a model;

[0027] Fig. 9 is a schematic diagram of a process of generating a home scene virtual model image based on reinforcement learning enhancement according to an embodiment of the present application;

[0028] Fig.10 is a schematic diagram of a skeleton graph transformation according to an embodiment of the present invention;

[0029] Fig.11 is a schematic diagram of a posture estimation network according to an embodiment of the present invention;

[0030] Fig.12 is a schematic diagram of a mask expansion effect according to an embodiment of the present invention;

[0031] Fig.13 is a schematic diagram of a mask dilation effect of a virtual model image according to an embodiment of the present invention;

[0032] Fig.14 is a schematic diagram of a posture control network according to an embodiment of the present invention;

[0033] Fig.15 is a schematic diagram of a virtual model image before and after color space loss training according to an embodiment of the present invention;

[0034] Fig.16 is a schematic diagram of a training framework of a redrawing model according to an embodiment of the present invention;

[0035] Fig.17 is a schematic diagram of a reasoning framework of a redrawing model according to an embodiment of the present invention;

[0036] Fig.18 is a schematic diagram of a model with a designated posture according to an embodiment of the present invention;

[0037] Fig.19 is a schematic diagram of stacking a model in a specified posture back to the original scene according to an embodiment of the present invention;

[0038] Fig. 20 is a schematic diagram of image generation by a computer device according to an embodiment of the present invention;

[0039] Fig.21 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0040] Fig. 22 is a schematic diagram of a device for determining a model according to an embodiment of the present application;

[0041] Fig.23 is a schematic diagram of another image generating device according to an embodiment of the present application;

[0042] Fig.24 is a schematic diagram of another image generating device according to an embodiment of the present application;

[0043] Fig.25 is a structural block diagram of a computer terminal according to an embodiment of the present application;

[0044] Fig.26 is a block diagram of an electronic device for a method for generating an image according to an embodiment of the present application;

[0045] Fig. 27 It is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method according to an embodiment of the present application;

[0046] Fig.28 It is a structural block diagram of a computing environment of an image generation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0048] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion model parameters. The large model can also be called a foundation model / foundation model. The large model is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (Language Model for Information Retrieval, referred to as LLM), a multi-modal pre-training model, etc.

[0049] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc., and can also be widely used in text-based sentiment classification, text summary generation, machine translation and other natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, the target language processing model obtained by training the generative large model in the dialogue scenario is used as an example to explain the application program generation.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0052] The diffusion model is a continuous-time Markov process. Its main idea is to learn the data distribution by gradually adding random noise to the clear image. Subsequently, through the process of inverse reconstruction of the model, an image with high quality and high resolution is generated. In the above process, the diffusion model can control the generation process of each step, so as to fine-tune the details and quality of the image. Through the diffusion model, the noise in the image can be effectively removed, the quality of the image can be enhanced, and the details and clarity of the image can be restored. The above method can improve the quality and visualization of the image;

[0053] Variational Autoencoder (VAE) is a generative model that learns low-dimensional representations of data and reconstructs data, thereby efficiently generating new similar samples. VAE combines the ideas of autoencoder and variational inference, and can be used to generate new samples with the characteristics of the underlying data distribution while retaining the key features of the original data. In VAE, the encoder network maps the input data to a low-dimensional representation in the latent space, which is the encoding process; the decoder network maps this low-dimensional representation back to the original data space, which is the decoding process. By minimizing the reconstruction error and the regularization term of the latent space, VAE can learn the distribution characteristics of the data and sample the data in the latent space to generate new similar samples. By minimizing the reconstruction error and the regularization term of the latent space, VAE can learn the distribution characteristics of the data and generate new samples with the characteristics of the underlying data distribution. In general, VAE is a generative model that learns low-dimensional representations of data and reconstructs data to efficiently generate new similar samples. It combines the ideas of autoencoders and variational inference and has a wide range of applications, such as image generation, data compression, and feature learning;

[0054] Luminance, a channel, b channel (Luminance*a*b, referred to as LAB) color space is a color space that describes colors. LAB color space is more in line with human visual perception and can describe colors more accurately. LAB color space consists of three channels: L, a, and b. Luminance (Luminance, referred to as L) channel reflects the brightness and darkness information of colors. L channel represents the brightness or lightness of a color. The value range is usually 0 to 100. The higher the value, the brighter the color, and the lower the value, the darker the color. The red-green axis position (a) channel represents the position of the color on the red-green axis. Positive values ​​represent red, negative values ​​represent green, and the larger the absolute value of the value, the more saturated the color. The value range of a channel is usually -128 to 127, where 0 represents neutral gray. The blue-yellow axis position (b) channel represents the position of a color on the blue-yellow axis. Positive values ​​represent yellow, negative values ​​represent blue, and the larger the absolute value of the value, the more saturated the color. The value range of b channel is also usually -128 to 127, where 0 represents neutral gray. The advantage of LAB color space is that it is more consistent with human visual perception and can more accurately describe the differences and relationships between colors. In fields such as image processing and color correction, LAB color space is often used to represent and process colors, especially when color correction, color balance, and color matching are required;

[0055] Reinforcement learning is a machine learning method whose goal is to allow the model to learn and optimize by interacting with the environment, selecting actions and receiving feedback scores (rewards or penalties). In reinforcement learning, the model explores the environment by trying different actions and adjusts its strategy based on the feedback from the environment in order to accumulate the maximum reward as much as possible during the training process. Through continuous attempts and feedback, the reinforcement learning model can gradually optimize its own strategy so that it can obtain the maximum reward in a specific task. For example, in reinforcement learning based on the diffusion model, the above action can be the result of the next step of denoising of the diffusion model, and the above strategy can be the conditional distribution of denoising step by step given the current state. Through reinforcement learning, the model can learn and optimize its own behavior in a complex and uncertain environment, and realize automated solutions to various complex tasks.

[0056] According to an embodiment of the present application, a method for generating an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] Considering the huge number of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned image generation method provided in the embodiment of the present application can be applied to the following Figure 1 The application scenarios shown are not limited to these. Figure 1 In the application scenario shown, the large model is deployed in a server 30, which can be a cloud. The server 30 can be connected to one or more terminal devices 10 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks, wherein the terminal device 10 can be a client device, and the client device here can include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc., and the client devices together constitute a client relative to the server. An operation interface for obtaining product images can be deployed on a graphical user interface on the client device, and the operation interface can be an operation interface of an e-commerce platform. The terminal device 10 can interact with the user through a graphical user interface to implement the call of the large model, and then implement the image generation method provided in the embodiment of the present application. Information exchange can be performed between the server 30 and the terminal device 10 through the network 20. A large model can be deployed in the database 40 in the server, and in the embodiment of the present application, the database 40 stores a redrawing model obtained by reinforcement learning training of the diffusion model through a preference data set. It can be used to call through the server 30 to generate a target scene image with both the auxiliary display object and the display object, so as to achieve the effect of using the auxiliary display object to assist the display object.

[0058] In the embodiment of the present application, the system composed of the terminal device, the network and the server can perform the following steps: if the user needs to use a certain auxiliary display object to assist the current display object, the user can perform input or selection operations on the operation interface of the corresponding terminal device 10 to upload the initial scene image to be generated to the terminal device 10. It is also possible to select information that meets the user's needs from a large amount of auxiliary display object related information displayed on the operation interface, such as posture, clothing, attributes, etc., to generate the corresponding auxiliary display object. The terminal device 10 can send the initial scene image and the auxiliary object image to the server 30 through the network 20. After the server 30 receives the initial scene image and the auxiliary object image, the following steps may be performed: Step S102, determining the initial scene image to be processed; Step S104, determining the auxiliary object image to be drawn at the target position in the initial scene image; Step S106, determining the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image; Step S108, using the redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain the target scene image.

[0059] The target scene image generated by the redrawing model can be output to the terminal device 10 through the network 20 to display the display object in combination with the auxiliary display object for the user to view.

[0060] In this embodiment, a redrawing model is set up to integrate human preferences into the AI ​​model generation process. The generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, and the visual effect and attractiveness of image generation are improved. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, so that image generation is more in line with user expectations and needs. In this way, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0061] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0062] Under the above operating environment, this application provides Figure 2 It should be noted that the method for generating the image of this embodiment can be Figure 1 The illustrated embodiment is executed by a mobile terminal. Figure 2 is a flow chart of a method for generating an image according to an embodiment of the present application, such as Figure 2 As shown, the method may include the following steps:

[0063] Step S202: determining an initial scene image to be processed.

[0064] In the technical solution provided in the above step S202 of the present application, the initial scene image may also be referred to as a user scene graph, a scene graph or an original scene graph. The image content of the initial scene image may include at least one display object and a related background for displaying the corresponding display object. For example, if the display object is a sofa, the related background may be a room, which may also include related furnishings commonly found in the room (e.g., carpets, coffee tables, bookshelves, and vases, etc.). The initial scene image does not include a model used to assist in displaying the display object. If the application scenario targeted by the image generation method is an e-commerce scenario, the display object is the commodity (also referred to as a product) to be promoted or sold in the scenario, and the initial scene image containing the commodity may be a promotion scene graph, which may be used to better display the effect of the commodity to be promoted.

[0065] Optionally, in an e-commerce scenario, if the target is interior design, the initial scene image can be an indoor space, and the display objects can be indoor furniture, lamps, decorations and other contents to show the interior decoration effects of different styles and design concepts. If the target is car advertising, the initial scene image can be a road or parking lot, and the display objects can be cars of different brands and models to show the appearance, interior and performance of the car. If the target is clothing matching, the initial scene image can be a street or shopping mall, and the display objects can be clothes of different styles and colors to show the matching effect and style of clothing. If the target is catering, the initial scene image can be a restaurant or coffee shop, and the display objects can be different kinds of food and drinks to show the appearance and taste of different foods.

[0066] It should be noted that the application scenarios of the above-mentioned image generation method, the different situations targeted in different application scenarios, and the different initial scene images and display objects targeted in different situations are only for illustration and are not specifically limited here. In the embodiments of the present application, in different application scenarios, the types of initial scene images and display objects can be selected according to specific needs and purposes to display different products, services or concepts. As long as the initial scene images and display objects are reasonably selected and designed to better display their characteristics and effects, improve the application scenarios of attractiveness and conveying information, and the initial scene images therein, they are all within the protection scope of the embodiments of the present application.

[0067] In this embodiment, by determining the initial scene image to be processed, a clear reference and basis can be provided for the subsequent image generation process, ensuring that the generated image can fully display the characteristics and effects of the display object, while also improving the user's cognition and experience of the display object.

[0068] Optionally, the initial scene image can be uploaded by the user according to the user's needs, or the initial scene image that meets the user's needs can be automatically generated according to the display object to be displayed.

[0069] For example, if a user needs to display a certain display object, the user can select a suitable display scene as the initial scene image according to the characteristics and positioning of the display object. For example, if the display object is a sofa, the user can select a modern-style living room, including furniture and decorations such as sofas, coffee tables, and carpets. The user can take or select a photo of an indoor scene that meets the requirements as the initial scene image. Through the above process, the generated initial scene image can show the effect of the display object in a real environment, helping users to better understand the appearance and matching of the product.

[0070] For another example, after determining the initial scene image, if image generation is required, the corresponding operation can be performed on the operation interface of the terminal device, the corresponding initial scene image can be selected, and uploaded. After uploading the initial scene image, the initial scene image can be transmitted to the server through the network. The server can identify and analyze the received initial scene image, determine the display object therein, and use it for subsequent image processing.

[0071] For example, if a user needs to display a certain display object, the corresponding initial scene image can be automatically generated according to the user's own needs and the display object to be displayed. For example, if the display object is a new car product. The user can use the tools provided by the server to select parameters such as the brand, model, color of the car, and the background requirements for displaying the car, and then the server can automatically generate an image that meets the above parameters as the initial scene image. Through the above process of generating the initial scene image, the effect of the display object in the display environment can be generated quickly and efficiently.

[0072] For another example, the need for the initial scene image to be generated can be input on the operation interface of the tool that can automatically generate the initial scene image on the terminal device. The above-mentioned demand can be transmitted to the server through the network. The server can identify and analyze the received demand, automatically generate an initial scene image that meets the above-mentioned demand, and return the above-mentioned initial scene image to the operation interface through the network for display, so that the user can confirm whether it needs to be adjusted. If no adjustment is required, the server can retain the initial scene image for subsequent image generation.

[0073] It should be noted that the above process and method for generating the initial scene image are only for illustration and are not specifically limited here. As long as it is possible to select a suitable initial scene image according to the user's own needs and the characteristics of the display object in order to better display and convey the characteristics and effects of the display object, it is within the protection scope of the embodiments of the present application.

[0074] Step S204: determining the auxiliary object image to be drawn at the target position in the initial scene image.

[0075] In the technical solution provided in the above step S204 of the present application, the target position can be the position where the auxiliary object image is set in the initial scene image, which can also be called a position parameter. The auxiliary object image can also be called a model posture diagram. The image content of the auxiliary object image can include at least one auxiliary display object. The auxiliary display object can be used to assist in the display of the display object, and can be an AI model, for example, a person or an animal. The number and type of auxiliary display objects can be determined according to the display object. For example, if the display object is a mahjong table, the auxiliary display object can be four people. If the display object is a cat teaser, the auxiliary display object can be a cat.

[0076] It should be noted that the number and type of the auxiliary display objects mentioned above are only examples and are not specifically limited here and can be determined according to actual user needs and display objects.

[0077] In this embodiment, after the initial scene image to be processed is determined, the auxiliary object image to be drawn at the target position in the initial scene image may be determined.

[0078] Optionally, according to various needs of users, auxiliary display objects that meet the needs are set, and corresponding auxiliary object images are formed.

[0079] In an embodiment of the present application, a corresponding application can be set up to generate corresponding auxiliary display objects based on user needs and generate a target scene image. In the application, a large amount of model attributes, postures, clothing and other information that can be selected by users can be set up. Model attributes can include male / female, adult / child, and skin color, etc., postures can include lying, standing, lying, walking, and sitting, etc., and clothing can include shirts, dresses, etc. Users can select more suitable information to assist the current display object from the large amount of model attributes, postures, and clothing obtained above according to their own needs on the operation interface of the application on the terminal device, thereby combining the information selected by the user to form a corresponding auxiliary display object.

[0080] It should be noted that the content and type of relevant information about auxiliary display objects for users to select in the above-mentioned application are only for example purposes and are not specifically limited here. As long as the needs of users can be met and corresponding auxiliary display objects can be generated to assist in the display of display objects, the information is within the protection scope of the embodiments of the present application.

[0081] Optionally, in the above application, corresponding controls or operations can be set to adjust the generated auxiliary object image, such as moving, resizing, and rotating, etc. Therefore, after the corresponding auxiliary object image is generated based on the information selected by the user, the user can perform corresponding operations to adjust the auxiliary object image to a satisfactory size and angle according to his own needs, and can move it to the target position.

[0082] Step S206: determining a posture image and an object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image.

[0083] In the technical solution provided in the above step S206 of the present application, the posture image can be used to represent the posture of the auxiliary display object, which can also be called a posture graph, for example, it can be a posture estimation skeleton graph obtained by performing posture estimation on the auxiliary object image, referred to as a skeleton graph. The object mask image can be used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area can include the target position. The object mask image can also be referred to as a model mask or a mask graph of the model.

[0084] In this embodiment, after the auxiliary object image to be drawn at the target position in the initial scene image is determined, a posture image and an object mask image of the auxiliary display object may be determined based on the auxiliary object image, the target position and the initial scene image.

[0085] Optionally, the auxiliary object image is analyzed to obtain the posture of the auxiliary display object, thereby determining the posture skeleton information of the auxiliary display object. Based on the posture skeleton information, a posture image corresponding to the posture skeleton of the auxiliary display object can be obtained. By performing mask generation on the auxiliary object image, a corresponding object mask image is obtained.

[0086] In an embodiment of the present application, a posture image and an object mask image of an auxiliary display object are determined, and the above images play an important role in the image generation process. The posture image is used to represent the posture of the auxiliary display object, that is, the posture and action of the auxiliary display object in the image. Through the posture image, the posture skeleton information (for example, posture key point information) of the auxiliary display object can be captured more accurately, and accurate posture skeleton information is provided for the subsequent generation of the target scene image, so that the auxiliary display object in the generated target scene image can maintain the posture indicated by the above posture skeleton information.

[0087] Optionally, the object mask image can be used to represent the image area where the auxiliary display object is located in the initial scene image, that is, to mark the position and area occupied by the display object. Through the object mask image, the auxiliary display object can be accurately segmented and positioned, providing key reference information for subsequent image synthesis and processing. By determining the posture image and the object mask image, the auxiliary display object can be made more vivid and natural in the generated image, improving the realism and attractiveness of the image. Based on the posture skeleton information and mask generation technology, the auxiliary display object can be accurately positioned and segmented, so that it can be better integrated with the initial scene image, enhancing the visual effect and expressiveness of the image. By analyzing and processing the auxiliary object image, the posture and position of the auxiliary display object can be better controlled, and the display effect can be finely controlled and optimized.

[0088] In summary, the above method can effectively improve the performance and realism of auxiliary objects in the image generation process, provide important technical support for generating more vivid and attractive images, and also help improve the quality and visual appeal of image generation, and meet users' needs for display effects.

[0089] Step S208: using the redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain a target scene image.

[0090] In the technical solution provided in the above step S208 of the present application, the redrawing model can be obtained by reinforcement learning of the diffusion model. The auxiliary display object satisfies the preference result in the target scene image. The target scene image includes the display object and the auxiliary display object, which can also be called the generation result or output result. The target scene image can be a scene model picture, or a model scene promotion picture, such as a high-definition model image. If the scene of the display object is a home scene, the target scene image can be a home scene virtual model image.

[0091] In this embodiment, after determining the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image, the posture image, the object mask image and the initial scene image can be analyzed using the redrawing model to obtain the corresponding target scene image.

[0092] Optionally, by analyzing the data input to the redrawing model, such as pose images, object mask images, and initial scene images, we can learn how to reasonably integrate the pose and position of the auxiliary display object into the initial scene image to generate a target scene image that meets human preferences. The preference dataset contains human ratings and preference information for generated images, which can help the redrawing model learn how to generate target scene images that are more in line with human preferences. The above data plays a key role in guiding the redrawing model to generate high-quality target scene images.

[0093] In an embodiment of the present application, by using the redrawing model to analyze the posture image, the object mask image and the initial scene image, the posture and position of the auxiliary display object can be reasonably integrated with the initial scene image. This helps to generate a target scene image that is more in line with human aesthetic standards and preferences, and improves the quality and visual appeal of the image. The analysis ability of the redrawing model can help the model better understand and process the input data, and generate the target scene image more accurately. Through the comprehensive analysis of information such as posture, position and mask, accurate redrawing of the scene can be achieved, making the generated image more natural and vivid.

[0094] In summary, by using the redrawing model in the above method to analyze the pose image, object mask image and initial scene image, it is possible to effectively generate a target scene image that meets human preferences. By integrating multiple data such as pose, position and preference information, the quality and realism of the generated image can be improved, the user's needs for visual effects can be met, and the expressiveness and attractiveness of the image generation can be improved.

[0095] Through the above steps S202 to S208 of the present application, if it is necessary to better display the display object in the image, the interaction between the display object and the auxiliary display object can be reflected by adding an auxiliary display object to assist in displaying the display object, for example, using the display object. An initial scene image including the display object can be obtained. According to the needs of the user, the target position where the auxiliary object image needs to be drawn can be determined from the initial scene image, and the auxiliary object image that meets the user's needs for the posture, clothing, etc. of the auxiliary display object can be screened from a large number of auxiliary object images. The posture of the auxiliary display object can be determined based on the auxiliary object image, the target position and the initial scene image, and the posture image and the object mask image of the auxiliary display object are obtained. The hidden variables corresponding to the posture image, the object mask image and the initial scene image are input into the redrawing model to integrate the target scene image. In the target scene image, the interaction between the auxiliary display object and the display object that meets the user's needs can be displayed, thereby improving the effect of displaying the display object. In this embodiment, a redrawing model is set up to integrate human preferences into the AI ​​model generation process. The generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and conformity of AI model generation. Thereby achieving the technical effect of improving the accuracy of image generation and solving the technical problem of low image generation accuracy.

[0096] The above method of this embodiment is further introduced below.

[0097] As an optional implementation, the redrawing model is obtained by reinforcement learning the diffusion model using a preference data set, the preference data set including pose image samples, object mask image samples, and target scene image sample pairs generated based on the auxiliary display object samples, the pose image samples are used to represent the pose of the auxiliary display object samples, the object mask image samples are used to represent the image area where the auxiliary display object samples are located in the initial scene image samples, and the target scene image sample pairs include target scene image samples whose evaluation results are higher than the evaluation result threshold, and target scene image samples whose evaluation results are lower than the evaluation result threshold.

[0098] In this embodiment, the redrawing model can be obtained by using the preference data set to perform reinforcement learning on the diffusion model, that is, it can be a redrawing model obtained by reinforcement learning training. The diffusion model can be a stable diffusion (StableDiffus ion) model, which can also be called a generation model or a diffusion generation model. The preference data set can include gesture image samples, object mask image samples, and initial scene image samples based on auxiliary display object samples, and the generated target scene image sample pairs can also be called preference pair data sets. The preference data set contains the input data required for diffusion model training, that is, the gesture image, object mask image, and initial scene image input by the user, and can also include preference pairs composed of high-scoring generated images (good images) and low-scoring generated images (bad images) obtained through human scoring. That is, the image can be generated by the pre-trained redrawing model based on the above-mentioned gesture image samples, object mask image samples, and initial scene image samples, and the image can be scored by humans, and then the good and bad images obtained by scoring can be assigned to obtain preference pairs. The preference data set helps to guide the model to learn how to generate target scene images that are more in line with human preferences.

[0099] Optionally, the evaluation result may also be referred to as a scoring result, and the target scene image sample whose evaluation result is higher than the evaluation result threshold may be the above-mentioned high-score generated image, or may be referred to as a high-score image. The target scene image sample whose evaluation result is lower than the evaluation result threshold may be the above-mentioned low-score generated image, or may be referred to as a low-score image. The scoring result threshold may be a pre-set score, or may be a score set by itself according to the actual image generation situation.

[0100] It should be noted that the above-mentioned setting method of the scoring result threshold is only for illustration and is not specifically limited here. As long as the method and process can be used to measure whether the target scene image sample meets the user's preferences, it is within the protection scope of the embodiments of the present application.

[0101] As an optional implementation, step S208, using the redrawing model to analyze the posture image, the object mask image and the initial scene image to obtain the target scene image, includes: using the model parameters of the redrawing model to analyze the posture image, the object mask image and the initial scene image to obtain the target scene image, wherein the model parameters are obtained by alternately performing reinforcement learning and supervised training on the diffusion model using the preference data set.

[0102] In this embodiment, in the process of analyzing the pose image, the object mask image and the initial scene image using the redrawing model, the model parameters of the redrawing model can be used to analyze the pose image, the object mask image and the initial scene image to obtain the final target scene image, wherein the model parameters can be obtained by alternating reinforcement learning and supervised training of the diffusion model using the preference data set.

[0103] Optionally, the preference pair dataset consists of a pair of paired data (preference pairs), each pair contains two images: a good image and a bad image. A good image refers to an image that is rated as high quality by humans, while a bad image is an image of relatively poor quality. These two images, as part of the training data, can help the diffusion model learn to generate target scene images that are more in line with human preferences. During the training process, the diffusion model uses the preference pair dataset for supervised fine-tuning to improve the quality and conformity of the generated images. By fine-tuning the good images, the diffusion model can learn the visual features and quality standards recognized by humans, thereby generating images that are more in line with human preferences. The purpose of the preference pair dataset is to guide the model to learn to generate images that are more in line with human aesthetic standards through human subjective scoring. By allowing the model to learn the features and quality of good images, the visual appeal and realism of the generated images can be improved, so that the generated images are more in line with the user's expectations and preferences. Introducing out-of-distribution data can. The preference pair dataset can also prevent the degradation of the model's generation ability by injecting out-of-distribution knowledge. By enabling the model to learn more diverse features and information, the generalization and generation capabilities of the model are improved.

[0104] In summary, preference pair datasets play an important role in the training process. By guiding the model to learn the characteristics and quality of good images, the quality and conformity of generated images can be improved, making the generated images more in line with human aesthetic standards and preferences. By rationally constructing and utilizing preference pair datasets, the performance and effect of the generated model can be effectively improved, and the visual appeal and realism of the generated target scene images can be improved.

[0105] Optionally, model parameters are the core component of the redrawing model, which contains all weights and parameters learned by the model during training. These parameters are obtained through alternating reinforcement learning and supervised training, which can help the model better understand and process input data, thereby generating target scene images that meet human preferences. Using the model parameters of the redrawing model, the pose image, object mask image, and initial scene image are analyzed. Guided by the model parameters, the model can accurately understand the pose information, object position, and scene features, so as to reasonably integrate the auxiliary display objects into the initial scene image and generate the final target scene image.

[0106] In the embodiment of the present application, the model parameters of the redrawing model are used for analysis, which can effectively improve the generation quality and accuracy of the target scene image. The learning of the model parameters has undergone multiple alternating reinforcement learning and supervised training, which can enable the model to better learn human preferences and standards and generate images that are more in line with human aesthetics. Through the comprehensive analysis of posture images, object mask images, and initial scene images, the model can more accurately understand the scene content and elements, improve the realism and visual effects of image generation, and make the generated target scene images more vivid and attractive.

[0107] In summary, by analyzing the pose image, object mask image, and initial scene image using the model parameters of the redrawing model, target scene images that meet human preferences can be effectively generated. The learning and application of model parameters can improve the quality and expressiveness of generated images, meet the user's needs for visual effects, and make the generated images more in line with user expectations and preferences. The above method helps to improve the quality and attractiveness of image generation and provides effective technical support for the generation of target scene images.

[0108] As an optional implementation, the model parameters of the redrawing model are used to analyze the posture image, the object mask image and the initial scene image to obtain the target scene image, including: extracting posture features from the posture image; using the posture features to control the redrawing model under the model parameters to analyze the posture image, the object mask image and the initial scene image to obtain the target scene image, wherein the auxiliary display object in the target scene image matches the posture features.

[0109] In this embodiment, in the process of analyzing the posture image, the object mask image and the initial scene image using the model parameters of the redrawing model, the posture features can be extracted from the posture image, and the posture features are used to control the redrawing model under the model parameters to analyze the posture image, the object mask image and the initial scene image to obtain the target scene image, wherein the posture features can also be referred to as posture control information, or simply control information. The auxiliary display object matches the posture features in the target scene image, and the auxiliary display object can be a designated posture model controlled by the posture control network.

[0110] Optionally, posture features are extracted from the posture image, which contain posture and motion information of the auxiliary display object, and are key factors in controlling the redrawing model to generate the target scene image. The posture features are used to control the redrawing model to analyze input images such as the posture image, the object mask image, and the initial scene image under the model parameters. By learning and understanding the posture features, the redrawing model can accurately analyze and redraw each of the above input images based on the above information to generate the final target scene image.

[0111] Optionally, the redrawing model is controlled by the posture feature to analyze and obtain the target scene image. In the process of generating the target scene image, the redrawing model adjusts and optimizes the posture and position of the auxiliary display object in the target scene image according to the posture feature and model parameters to ensure the matching degree between the auxiliary display object and the scene.

[0112] In this embodiment of the present application, by using the posture features to control the redrawing model, accurate generation and adjustment of the target scene image can be achieved, so that the generated image is more in line with expectations and expectations. Using the posture features for control can enable the redrawing model to better understand and process the posture information, thereby improving the quality and realism of the target scene image, making the fusion of the auxiliary display object and the scene more natural and realistic.

[0113] In summary, by utilizing the model parameters and posture features of the redrawing model, combined with the analysis of the posture image, object mask image, and initial scene image, it is possible to generate a target scene image that meets human preferences. The above method can effectively improve the quality and visual effect of the generated image, making the generated image more vivid, natural, and attractive. Through the precise control of the posture features, the target scene image can be finely adjusted and optimized, and the expressiveness and attractiveness of the generated image can be improved.

[0114] As an optional implementation, the posture feature is used to control the redrawing model to analyze the posture image, the object mask image and the initial scene image under the model parameters to obtain the target scene image, including: using the attention mechanism to convert the posture feature into control information of the redrawing model; using the control information to control the redrawing model to analyze the posture image, the object mask image and the initial scene image under the model parameters to obtain the target scene image.

[0115] In this embodiment, in the process of using the posture feature to control the redrawing model to analyze the posture image, the object mask image and the initial scene image under the model parameters, the attention mechanism can be used to convert the posture feature into the control information of the redrawing model. Using the control information, the redrawing model is controlled to analyze the posture image, the object mask image and the initial scene image under the model parameters to obtain the target scene image, wherein the posture feature can be the information output by the posture control network, and the information can be injected into the generation process of the redrawing model through the cross-attention mechanism.

[0116] Optionally, in the process of analyzing the posture image, the object mask image and the initial scene image under the model parameters by controlling the redrawing model using the posture features, an attention mechanism is introduced to convert the posture features into control information of the redrawing model. The attention mechanism can make the redrawing model pay more attention to the important parts when processing the input data, thereby improving the performance and effect of the model. In this embodiment, the attention mechanism is used to convert the posture features into control information suitable for the redrawing model. Through the attention mechanism, the key information in the posture features can be effectively extracted and utilized to guide the model to generate the target scene image.

[0117] Optionally, the attention mechanism is used to convert the posture features into control information of the redrawing model. The control information may include important information about posture, position and features. Through the processing and conversion of the attention mechanism, the posture features can better guide the generation process of the redrawing model. Using the converted control information, the redrawing model is controlled to analyze the posture image, the object mask image and the initial scene image under the model parameters to obtain the final target scene image. The above process enables the redrawing model to understand and process the input data more accurately and generate a target image that meets human preferences.

[0118] In an embodiment of the present application, by introducing an attention mechanism and control information conversion, the model can focus more on the important information in the posture features, and improve the model's ability to analyze posture images, object mask images, and initial scene images. The injection of control information can make the redrawing model more accurate and detailed when generating target scene images, thereby improving the quality and realism of the generated images, and making the target scene images more in line with human aesthetic standards and expectations. In summary, by utilizing the attention mechanism and control information conversion, the posture features can be effectively converted into control information of the redrawing model, thereby guiding the model to generate target scene images that meet human preferences. The above method helps to improve the quality and accuracy of the generated images, making the generated target scene images more vivid, natural, and attractive. The introduction of attention mechanisms and control information adds more flexibility and precision to the generation process of the model, and enhances the expressiveness and effect of image generation.

[0119] As an optional implementation, control information is used to control the redrawing model to analyze the pose image, the object mask image and the initial scene image under the model parameters to obtain the target scene image, including: combining the object mask image and the initial scene image to obtain the scene mask image, wherein the scene mask image includes a mask corresponding to the object mask image, and the size of the scene mask image is smaller than the size of the initial scene image; using an encoder to generate initial latent variables in the latent variable space from the scene mask image, the object mask image and the initial scene image; using control information to control the redrawing model to analyze the initial latent variables under the model parameters to obtain an intermediate scene image, wherein the size of the intermediate scene image is smaller than the size of the initial scene image; and superimposing the intermediate scene image on the initial scene image to obtain the target scene image.

[0120] In this embodiment, in the process of using control information to control the redrawing model to analyze the posture image, the object mask image and the initial scene image under the model parameters, the object mask image and the initial scene image can be combined to obtain the scene mask image. The encoder can generate the initial latent variables in the latent variable space through the scene mask image, the object mask image and the initial scene image. Using the control information, the redrawing model is controlled to analyze the initial latent variables under the model parameters to obtain the intermediate scene image. The intermediate scene image is superimposed on the initial scene image to obtain the final target scene image. Among them, the scene mask image is obtained by combining the model mask with the scene graph, and can also be called a scene graph with a mask or a scene graph mask. The scene mask image may include a mask corresponding to the object mask image, and the size of the scene mask image is smaller than the size of the initial scene image. The encoder can be a variational auto encoder (VAE for short), and the encoder is used to encode the input image data into a representation in a latent space (e.g., a latent variable space). The intermediate scene image may be a cropped scene model image, for example, a red-green-blue (RGB) image. The size of the intermediate scene image is smaller than the size of the initial scene image.

[0121] Optionally, the model mask is combined with the scene graph to generate a scene graph with the model mask. Through the above method, the pose information of the model can be integrated into the scene graph to provide accurate pose information for subsequent image generation.

[0122] Optionally, the model mask image is a segmentation of the model image to obtain the model's contour information, which is usually a binary image in which the model area is white and the background is black. The scene image contains the background image that needs to be integrated into the model. The model mask image is superimposed or synthesized with the scene image to obtain a scene graph with a model mask. For example, the white area (model area) in the model mask image can be superimposed on the scene image, retaining the background information of the corresponding position in the scene image, while displaying the model area. By combining the model mask image with the scene image, a scene graph with a model mask is generated, thereby integrating the model's posture information into the scene graph, providing accurate posture information for subsequent image generation.

[0123] Through the above method, the model mask can be combined with the scene graph to generate a scene graph with the model mask. The processed scene graph will contain the model's posture information, providing more accurate input data for subsequent image generation tasks, and helping to generate images with more realistic and complete posture information.

[0124] Optionally, the scene graph with the model mask, the model mask and the scene graph are input into the VAE to generate latent variables (ie, initial latent variables). The above method can reduce the dimension of the image to the latent variable space to reduce the computing time and video memory space required for subsequent image generation.

[0125] Optionally, the scene graph with the model mask, the model mask, and the scene graph are input into the data of the VAE variational encoder. The scene graph with the model mask contains the pose information of the model and the scene background, the model mask is the outline information of the model, and the scene graph is the background image that needs to be integrated into the model. VAE consists of an encoder and a decoder. In the encoder part, the scene graph with the model mask, the model mask, and the scene graph are input into it. The role of the encoder is to encode these input image data into potential representations in the latent variable space. The encoder will learn to map the input data to the mean vector and variance vector in the latent variable space. The above mean and variance vectors will be used to generate random latent vectors, thereby encoding the input data. In VAE, random latent vectors are obtained by sampling from the mean and variance learned by the encoder. The above sampling process can be regarded as a re-parameterization of the input data in the latent variable space for subsequent decoding operations.

[0126] Through the above method, the VAE variational encoder can be used to convert the scene graph with model mask, model mask and scene graph into latent variables for subsequent image generation and processing tasks to achieve more accurate and expressive target scene image generation.

[0127] As an optional implementation, control information is used to control the redrawing model to analyze the initial latent variables under the model parameters to obtain an intermediate scene image, including: using control information to control the redrawing model to analyze the initial latent variables under the model parameters to obtain target latent variables in the latent variable space; using a decoder to decode the target latent variables to obtain an intermediate scene image.

[0128] In this embodiment, in the process of using the control information to control the redrawing model to analyze the initial latent variables under the model parameters, the control information can be used to control the redrawing model to analyze the initial latent variables under the model parameters to obtain the corresponding target latent variables. The target latent variables can be decoded by the decoder to obtain the corresponding intermediate scene image, wherein the decoder can be a decoder of the VAE variational encoder.

[0129] Optionally, the target latent variable is the result obtained after analyzing the initial latent variable under the action of the control information. It is a representation in the latent variable space obtained after model analysis and processing, and usually contains the key features and information of the input image. The target latent variable can be considered as a more abstract and advanced representation of the input image, which captures the potential structure and characteristics of the image. The target latent variable may contain more information and more meaningful representations than the initial latent variable. By analyzing the initial latent variable through the control information, the target latent variable can better capture the important features of the image. The target latent variable may be a higher-level and more abstract representation of the initial latent variable. Through model analysis and processing, the target latent variable can better express the essential characteristics of the image, which is helpful for subsequent tasks and applications. The target latent variable may be a better representation of the input image, which can more accurately capture the intrinsic structure and characteristics of the image. This helps to improve the performance of the model and generate more expressive results.

[0130] Optionally, the posture control information output by the posture control network can be input into the redrawing model, and the posture control information can help the redrawing model to better generate a model image that conforms to a specified posture. Using the posture control information, the redrawing model is controlled to determine the target latent variable.

[0131] Optionally, the target latent variable is used as input and a decoding operation is performed through the decoder part. The decoder is a part of the VAE model that maps the latent variable back to the original data space to generate an image similar to the input data. The decoder maps the target latent variable back to the image space through the learned parameters to generate an intermediate scene image. The above process involves operations such as deconvolution operations or fully connected layers to convert the latent variables into pixel values ​​of the image. Through the operation of the decoder, the target latent variable is decoded into an intermediate scene image. This image can be regarded as one of the samples generated by the redrawing model based on the learned data distribution. The intermediate scene image is the result generated based on the target latent variable, reflecting the important features and information of the input data.

[0132] Through the above method, the decoder can be used to decode the target latent variable to generate an intermediate scene image. This intermediate scene image is the result generated based on the target latent variable, reflecting the important features and information of the input data, that is, the intermediate scene image at this time has auxiliary display objects, and the image can reflect the posture and other information of the auxiliary display objects. The above operation provides a basis for subsequent image generation tasks, and can also be used for tasks such as image processing and analysis.

[0133] Optionally, the function of the redrawing model trained based on reinforcement learning is to generate a model with a specified posture controlled by a posture control network in the mask part of the scene graph. The network structure of the model refers to the stable diffusion model and expands it to adapt to existing input requirements. The main function of the redrawing model is to generate an AI model with a specified posture in the mask part of a given scene graph. Through the control information provided by the posture control network, the model can generate a model image that meets the requirements in the specified area. The network structure of the stable diffusion model (that is, the original network structure) refers to the stable diffusion model, and the redrawing model is generally used to process image generation tasks. The stable diffusion model is a probabilistic generation model for generating images, which has good generation effect and stability.

[0134] Optionally, in order to adapt to the existing input requirements, the input layer of the original network structure is expanded from four channels to nine channels. It can better adapt to the current input data, including three latent variables encoded by the VAE encoder, namely the model mask, scene graph, and masked scene graph. These three inputs are used to guide the model to generate the model image of the mask part, maintain the invariance of the original scene graph, and generate a model that conforms to the posture control information.

[0135] Through the above method, the redrawing model trained based on reinforcement learning can use posture control information to generate an AI model with a specified posture in the mask part of a given scene graph. By expanding the input layer of the network structure and adapting to multi-input requirements, the model can better achieve image generation tasks and improve the quality and accuracy of generated images. This processing method helps to generate model images that better meet user needs and improve the generation effect of the model.

[0136] As an optional implementation, the method also includes: cropping the object mask image, wherein the size of the cropped object mask image is smaller than the size of the initial scene image; cropping the initial scene image and the posture image respectively according to the cropping parameters corresponding to the cropped object mask image; combining the object mask image and the initial scene image to obtain a scene mask image, including: combining the cropped object mask image and the cropped initial scene image to obtain a scene mask image.

[0137] In this embodiment, the object mask image may also be cropped, and the initial scene image and the posture image may be cropped respectively according to the cropping parameters corresponding to the cropped object mask image. The cropped object mask image and the cropped initial scene image may be combined to obtain a scene mask image. The size of the cropped object mask image is smaller than the size of the initial scene image. The cropping parameters of the cropping process may include cropping position parameters, etc., which are only used as examples and are not specifically limited. The cropping position parameters may be used to determine the cropping position area.

[0138] Optionally, in the process of cropping the mask part (i.e., the object mask image) to obtain a cropped mask map centered on the mask (i.e., the cropped object mask map), the scene graph and the posture graph are cropped accordingly according to the cropping position, so that the mask part can occupy a larger proportion of pixels in the subsequent generation process, thereby facilitating the generation of a higher-definition model image.

[0139] Optionally, the mask part is cropped to obtain a cropped mask image centered on the mask. The cropped mask image will be concentrated around the model part, so as to better highlight and locate the model position. According to the position information of the cropped mask image, the scene graph and the pose graph are cropped accordingly. This ensures that the contents of the scene graph and the pose graph are aligned with the cropped mask image, making the model image generated subsequently more accurate and complete.

[0140] In an embodiment of the present application, by cropping the mask, scene graph, and posture graph, the mask part can occupy a larger proportion of pixels in the subsequent generation process, so that the generated model image can be made clearer and higher-definition, because the model part occupies more pixels, which is conducive to the detail display and clarity improvement of the model image. The cropping process helps to ensure that the model part occupies a larger proportion in the generated image, making the generated model image more prominent and high-definition. This can improve the quality and realism of the generated image and provide a better data basis for subsequent processing and application. Through the above process, the purpose of cropping the mask graph, scene graph, and posture graph is to optimize the generation process of the model image, so that the model part can be better presented in the generated image, improve the clarity and quality of the model image, and help generate a higher quality and more realistic model image.

[0141] Optionally, cropping the mask image refers to cropping the object mask image according to specific position information to obtain a new mask image of the region of interest. The purpose of cropping the object mask image is to limit the image to a specific area, remove unnecessary parts or obtain the region of interest.

[0142] Optionally, in the process of cropping the object mask image, the position of the region to be cropped can be determined according to the requirements and the specific position information (target position). According to the determined position information, the mask image is cropped to retain the region of interest and remove the unnecessary part. After the cropping operation, a new mask image is obtained, which only contains the region of interest.

[0143] As an optional implementation, the method further includes: dilating the edge of the object mask image; and cropping the object mask image, including: cropping the dilated object mask image.

[0144] In this embodiment, before the object mask image is cropped, the edge of the object mask image may be expanded.

[0145] Optionally, edge dilation processing can be used to expand the edge pixels of the model mask so that the redrawing area generated by the model is wider. The above processing can ensure that the generated model image is more complete visually and contains more features and details of the model. Through edge dilation processing, not only can the integration of the model and the background be enhanced, but also the shadow effect can be improved, and the quality of the generated image can be improved. Specifically, through edge dilation processing, the edge of the model will become smoother and more natural, making the transition between the model and the background more harmonious. This can enhance the integration of the model and the background, making the generated image look more realistic. Edge dilation processing can make the redrawing area generated by the model wider, thereby effectively generating more natural shadows around the model. This processing can improve the visual effect of the overall picture and make the interaction between the model and the surrounding environment more harmonious. By expanding the redrawing area generated by the model, improving the integration of the model and the background and the shadow effect, edge dilation processing helps to generate higher quality and more realistic model images. This can improve the visual effect and perceptual quality of the generated image.

[0146] In general, performing edge dilation on the object mask image helps to make the generated image more realistic, natural, and high-quality.

[0147] Optionally, edge dilation is an image processing technique for expanding the edge portion of an object in an image so that the redrawing area of ​​the object is wider. During the edge dilation process, an object mask image can be obtained, that is, a binary image containing a model, wherein the pixel value of the model portion is foreground (can be white or have a value of 1), and the pixel value of the background portion is background (can be black or have a value of 0). Edge dilation usually uses a dilation operation in mathematical morphology. The dilation operation processes the image based on the structure element to expand the edge portion of the object. In the edge dilation operation, the structure element can slide in the image, and for each pixel position covered by the structure element, if the structure element overlaps with any foreground pixel in the image, the center position of the structure element is updated to a foreground pixel, so that the edge of the foreground object can be expanded. Specifically, the edge dilation operation causes the edge portion of the model to expand outward and merge adjacent foreground pixels together. Through multiple iterations of the dilation operation, the edge portion of the model will gradually expand to form a smoother and continuous edge. The effect of the edge dilation operation depends on the shape and size of the structure element and the number of dilation iterations. By adjusting the shape and size of the structure element, the degree of dilation can be controlled. Larger structure elements and more iterations will result in greater dilation. Through the above method, the edge of the object can be enlarged, the transition effect between the object and the background can be improved, and the visual effect and realism of the image can be enhanced.

[0148] Optionally, after obtaining the processed object mask image through the above edge dilation processing, the object mask image may be cropped accordingly to obtain a cropped mask image centered on the object mask.

[0149] As an optional implementation, superimposing the intermediate scene image onto the initial scene image to obtain the target scene image includes: superimposing the intermediate scene image onto the initial scene image according to a cropping parameter to obtain the target scene image.

[0150] In this embodiment, in the process of superimposing the intermediate scene image on the initial scene image, the intermediate scene image may be superimposed on the initial scene image according to the cropping parameters, thereby obtaining the target scene image.

[0151] Optionally, the object mask image and the scene graph are cropped so that the model part can occupy a larger pixel ratio during the generation process, thereby generating a high-definition model image. Therefore, the generated image output by the redrawing model is a cropped scene model image, which needs to be superimposed back to the original-size scene graph to restore the complete scene image.

[0152] Optionally, the generated image output by the redrawing model is a cropped scene model image, in which the model part occupies a larger pixel ratio to improve the clarity and quality of the model image. This cropping process is to highlight the model part and make it more prominent and clear. According to the position parameters for cropping the object mask image, the generated cropped scene model image is superimposed back to the original size scene image. Through the above process, the generated image can be placed at the corresponding position of the original scene image according to the cropped position information to restore the integrity of the original scene. The generated overlapping image is used as the final output result. The output result is a high-definition model image in which the model part occupies a larger pixel ratio, which is overlapped with the original scene image, so that the model and the background scene are fully presented together. By superimposing the cropped generated image back to the original size scene image, the high definition of the model image can be maintained and the complete scene image can be restored. In this way, an effective combination of the model image and the scene image can be achieved to generate a more visually attractive and realistic final result.

[0153] In the embodiment of the present application, the above method can realize the superposition of the generated cropped scene model image back to the original size scene image, thereby obtaining a final output result with a high-definition model image and a complete scene. The above process is an important step in the image generation process, which can improve the image quality and visual effect.

[0154] As an optional implementation, posture features are extracted from posture images, including: using a posture control model to extract posture features of different sizes from the posture image, wherein the posture control model is obtained by training a control network structure using the posture of an auxiliary display object sample.

[0155] In this embodiment, in the process of extracting posture features from the posture image, the posture control model can be used to extract posture features of different sizes from the posture image, wherein the posture control model can be obtained by training the control network structure using the posture of the auxiliary display object sample. The posture control model can be a posture control network (Control Net). Different sizes can also be called different dimensions, for example, they can be 512*512, 256*256, 128*128, etc. This is only for example and is not specifically limited.

[0156] Optionally, the posture image is input into a posture control network, and the posture features in different size dimensions are extracted therefrom using the posture control network.

[0157] Optionally, the posture control network is used to extract posture features in different dimensions from the input posture estimation skeleton graph. These posture features may include the posture information of the model, the position of key points, etc., and are used to control the posture performance of the generative model in the subsequent image generation process. The posture control network injects the extracted posture features into the diffusion model of subsequent image generation through cross-attention, thereby providing posture control information for the generative model, so that the generated image can be adjusted and optimized according to the posture features.

[0158] Optionally, the ControlNet posture control method is used to implement the posture control network. This method combines deep learning technology to extract and process posture features. During the training process, the outputs of each layer of the posture control network are injected into the decoding part of the diffusion generation model in a cross-attention manner. Therefore, in the process of generating images, the image can be adjusted and generated in combination with the posture information, achieving more accurate and expected posture performance.

[0159] Through the above method, the posture control network extracts posture features and injects them into the generation model, which can realize the control and adjustment of the posture during the image generation process, so that the generated image is more in line with the expected posture requirements. This processing method helps to improve the quality and visual effects of the generated image, making the generated model image more realistic and expressive.

[0160] Optionally, in the process of training the posture control network, the posture control method of ControlNet is used, by copying a diffusion model downsampling network and taking the posture skeleton map corresponding to the model as input. During training, the diffusion generation model is kept frozen, and the outputs of each layer of the posture control network are injected into the decoding part of the diffusion generation model in a cross-attention manner to provide posture control information for the generation model.

[0161] Optionally, a posture control network constructed by the ControlNet posture control method is used to process the posture skeleton image data corresponding to the model. The purpose of the network is to extract and learn posture information so that the subsequent generative model can generate images with specified postures according to the posture control information. The posture skeleton image corresponding to the model is used as input data for training the posture control network. These data contain the posture information of the model, such as joint positions, angles, etc. During the training of the posture control network, the diffusion generation model is kept frozen, that is, it is not updated. This ensures that the generation model will not be affected during the training process, and focuses on learning the posture control information. The outputs of each layer of the posture control network are injected into the decoding part of the diffusion generation model by cross attention. In this way, the generation model can take the posture control information into account when generating images, so as to better control the posture and features of the generated images. While receiving the posture control information, the generation model generates corresponding image outputs according to the injected posture control information. By providing the posture control information, the generation model can generate image results with specified postures, thereby improving the accuracy and quality of the generated images.

[0162] Through the above method, the posture control network extracts posture information through the ControlNet method and injects it into the diffusion generation model to provide posture control information. Such a method can effectively control the posture and features of the image generated by the generation model, so that the generated image is more in line with the expected posture requirements, thereby improving the quality and realism of the generated image.

[0163] Optionally, the posture control information refers to relevant information used to control and adjust the posture or action of the human body, and is used to guide the model to generate an image with a specific posture or action. The posture control information may include key point positions, joint angles, and action sequences. The posture control information may include position information of key points of the human body, such as the coordinates of key points such as the head, shoulders, elbows, and knees. By adjusting the positions of the key points, the posture of the human body in the generated image can be controlled. The posture control information may also be the angle information of the joints, that is, the bending angles of different joints. By adjusting the angles of the joints, the generated human body image can present different postures and actions. The posture control information may also include an action sequence, which describes the action flow or action sequence of the human body. The model can generate corresponding action images according to different action sequences.

[0164] It should be noted that the content included in the above-mentioned posture control information is only for illustration and no specific limitation is made here.

[0165] For example, if you want to generate an image of a person running, the posture control information can include the position information and action sequence of the key points of the human body, guiding the model to generate an image of the human body with running movements. By adjusting the posture control information, the model can generate images with different postures and movements, and realize the control and adjustment of the human body posture. The posture control information can help generate more realistic and accurate human body images.

[0166] As an optional implementation, step S106, based on the auxiliary object image, the target position and the initial scene image, determines the posture image and the object mask image of the auxiliary display object, including: using the posture estimation model to analyze the auxiliary object image to obtain the initial posture image corresponding to the posture image, and using the transparency channel of the auxiliary object image to generate the initial object mask image corresponding to the object mask image, wherein the posture estimation model is obtained by training the transformation network structure using the self-attention mechanism; the initial posture image and the initial object mask image are moved to the target position of the initial scene image to obtain the posture image and the object mask image.

[0167] In this embodiment, in the process of determining the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image, the posture estimation model can be used to analyze the auxiliary object push to obtain the initial posture image corresponding to the posture image, and the transparency channel of the auxiliary object image can be used to generate the initial object mask image corresponding to the object mask image. By moving the initial posture image and the initial object mask image to the target position, the posture image and the object mask image can be obtained. Among them, the posture estimation model can also be called a posture estimation network, for example, it can be a visual transformer posture estimation (Vis ion Transformer Pose Est imat ion, referred to as ViTPose) network. ViTPose can be a posture estimation network based on a transformer (Transformer) structure, which is a deep learning model based on a self-attention mechanism for estimating and predicting the posture of the auxiliary display object. The initial posture image can be a key point posture skeleton image obtained by analyzing the auxiliary object image through a posture skeleton model. It can be used to reflect the relevant information between the key points of the auxiliary display object.

[0168] Optionally, the auxiliary object image can be a four-channel model posture image, three of which can be RGB channels, and the additional fourth channel can be a transparency (alpha, referred to as A) channel, that is, the auxiliary object image at this time can be an RGBA image. The A channel is used to store the transparency value of each pixel in the auxiliary object image. Through the above-mentioned additional transparency channel, the contour features of the model can be more comprehensively described, providing richer data for subsequent processing and operations.

[0169] Optionally, the alpha channel is usually used to indicate the degree of opacity of each pixel in the image, and the value range is usually between 0 and 1, where 0 means completely transparent and 1 means completely opaque. When processing a model pose graph, the transparency of each pixel in the model image can be controlled through the alpha transparency channel, thereby adjusting the transparency of the model image. This is very useful for compositing or superimposing a model with a background, which can achieve a more natural and smooth transition effect. Through the alpha transparency channel, the model can be separated from the background to generate a mask map of the model. By performing threshold processing or segmentation operations on the alpha channel, the outline of the model can be extracted for subsequent processing and operation. In the process of pose estimation and control, the alpha transparency channel is used in combination with the pose skeleton map to more accurately locate the pose information of the model. Transparency information can help calibrate the correspondence between the pose skeleton map and the model image, thereby more accurately extracting and processing the pose information of the model.

[0170] In summary, the alpha transparency channel plays the role of controlling transparency, extracting masks, and combining with posture skeleton graphs in the above method. It can help to accurately process and analyze the model posture graph and provide necessary data and information support for the subsequent image generation process.

[0171] Optionally, by processing the four-channel model posture map selected by the user, the mask map and posture skeleton map of the model are obtained for the subsequent image generation process. According to the four-channel model posture map selected by the user, the mask information of the model in the image can be extracted using the alpha transparency channel to obtain the mask map of the model. Through the above-mentioned mask map of the model, the pixel area occupied by the model can be accurately identified for subsequent processing and operation. The posture skeleton map of the model can be obtained by performing posture estimation on the model map using the posture estimation network. The posture skeleton map describes the position information of each key point or joint point of the model's body. The above information can accurately describe the posture of the model and provide important data for subsequent control and generation.

[0172] Optionally, in the process of determining the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image can be analyzed by the posture estimation model to obtain the initial posture image corresponding to the posture image. The posture estimation model is a deep learning model that can be obtained by training the transformation network structure using the self-attention mechanism. The function of the model is to perform posture estimation and analysis on the input object image to obtain the posture information of the object. The transparency channel of the auxiliary object image is used to generate the initial object mask image corresponding to the object mask image. The transparency channel is usually used to represent the transparency level of each pixel in the image, which can help generate the mask image of the object to identify the position and boundary of the object. The initial posture image and the initial object mask image are moved to the target position of the initial scene image to obtain the final posture image and object mask image. The above process can ensure that the posture image and the object mask image are aligned with the scene image and accurately positioned at the target position.

[0173] Through the above method, the conversion and generation process from auxiliary object images to posture images and object mask images can be realized. With the help of the posture estimation model, the posture information of the object can be accurately obtained and applied to subsequent image processing and generation to achieve more accurate and posture-controlled image results.

[0174] As an optional implementation, the auxiliary object image is analyzed using a posture estimation model to obtain an initial posture image, including: using the posture estimation model to slice the auxiliary object image to obtain multiple sub-auxiliary object images, and determining a sub-pose image of the sub-auxiliary object image at a target position in the initial scene image, wherein the multiple sub-auxiliary object images have an association relationship; using the posture estimation model to fuse image information in multiple sub-pose images corresponding to the multiple sub-auxiliary object images according to the association relationship to obtain the initial posture image.

[0175] In this embodiment, in the process of analyzing the auxiliary object image using the posture estimation model, the posture estimation model can be used to cut the auxiliary object image into blocks to obtain multiple sub-auxiliary object images. The sub-pose image at the target position corresponding to each sub-auxiliary object image in the initial scene image can be determined. The posture estimation model is used to merge the corresponding multiple sub-pose images according to the association relationship between the sub-auxiliary object images, and after merging, the initial pose image is obtained. Among them, the sub-auxiliary object images obtained by cutting the auxiliary object image into blocks can also be called blocks, blocks or slices, and can also be called posture estimation data sets. There is a certain association relationship between each block, that is, each block can establish a connection with other blocks.

[0176] Optionally, the relationship between each slice in the auxiliary object image can be established by using the self-attention mechanism. Each slice can not only process its internal features individually, but also interact with other slices through the attention mechanism to learn mutual dependencies. Through the above method, the posture estimation model can not only focus on the features of the image area of ​​a single slice, but also focus on the position and other relationships between each slice, which is conducive to the fusion of overall information. Finally, the local information corresponding to each slice (image information in the sub-state image) can be fused to obtain the initial posture image.

[0177] In the embodiment of the present application, ViTPose uses a posture estimation network based on the Transformer structure to replace the traditional convolutional neural network. The Transformer structure better captures the relationship between different key points through the self-attention mechanism, achieving efficient feature learning. Compared with the related convolution operation, the self-attention mechanism can simultaneously consider the information of all positions in the sequence, which helps to better model long-distance dependencies.

[0178] Optionally, in the ViTPose model, multiple decoders can be used to process multiple pose estimation datasets while sharing a backbone encoder. That is, the ViTPose model can simultaneously process multiple different datasets by integrating multiple decoders, thereby achieving estimation of different poses. The backbone encoder is the core part of the entire ViTPose model, which is used to extract feature representations of the input image. By sharing the backbone encoder, it can be ensured that different decoders can share the same feature representation, thereby improving the learning efficiency and generalization ability of the model. The ViTPose model can integrate multiple decoders, each decoder for processing different pose estimation datasets. This design enables the model to process multiple datasets at the same time to achieve estimation of different poses, increasing the flexibility and applicability of the model. By using multiple decoders to process multiple datasets, more comprehensive and diverse training data can be provided for the ViTPose model, thereby improving the generalization ability and performance of the model. Different datasets may contain data samples of different types or different poses. By processing these datasets simultaneously, the model can learn more different types of information. Different decoders can be optimized specifically for different Tasks to achieve more accurate and efficient pose estimation.

[0179] In summary, by using multiple decoders and sharing the backbone encoder, the ViTPose model can process multiple different pose estimation data sets, improve the applicability and performance of the model, and increase the model's adaptability to diverse application scenarios. The design of the above ViTPose model can effectively improve the accuracy and efficiency of the model and bring better performance to the pose estimation task.

[0180] Optionally, the Transformer structure has strong parallel processing capabilities and can process information at different locations in parallel, thereby improving the efficiency of feature extraction. This means that ViTPose can learn the relationship and feature representation between key points faster, speeding up the process of pose estimation. The Transformer structure can fully consider global contextual information, which makes ViTPose perform better when dealing with complex scenes. Through the self-attention mechanism, the model can simultaneously focus on the relationship between different regions, thereby better understanding the semantic information of the overall image. When processing an image, ViTPose divides it into multiple blocks and uses the self-attention mechanism to obtain information connections between different blocks in parallel. This can improve the ability to extract global information, allowing the model to better capture the relationship between different regions, thereby improving the accuracy of pose estimation. ViTPose allows users to set the number of local regions to be divided, so that it can be adjusted according to the needs of specific tasks and scenarios, making the model more flexible and adaptable to different situations.

[0181] Through the above method, the Transformer-based posture estimation network ViTPose has strong feature learning and global information extraction capabilities, and can process image information in parallel through the self-attention mechanism to better capture the relationship between different regions, thereby improving the efficiency and accuracy of posture estimation and performing better in complex scenes. The advantages of this structure enable ViTPose to have better performance and performance when handling posture estimation tasks.

[0182] Optionally, the above-mentioned global context information refers to considering the relevant information and relationships within the overall range when processing the auxiliary object image, rather than just focusing on the information in the local area, which can provide broader and deeper background knowledge, help the posture estimation model to better understand and interpret the input data, thereby improving the performance and effect of the posture estimation model. The global context information includes all the information in the entire auxiliary object image, not just the information in the local area. This includes the background, environment, overall structure, etc., which can provide cognition and understanding of the overall situation. The association relationship and mutual influence between different parts (sub-auxiliary object images) are also considered. Through the global context information, the posture estimation model can better capture the complex relationship between different parts and help the posture estimation model understand more accurately. The global context information can also maintain the global consistency of the auxiliary object image, ensuring that the posture estimation model takes into account the overall consistency and relevance when processing the above image, and avoids the one-sidedness and errors of local information. By considering the global context information, the posture estimation model can better optimize and make decisions, thereby improving the performance and generalization ability of the posture estimation model. The integration and consideration of global information can help the posture estimation model better capture the essential characteristics and laws of the above image.

[0183] As an optional implementation, the initial pose image and the initial object mask image are moved to the target position of the initial scene image to obtain the pose image and the object mask image, including: scaling the initial pose image and the initial object mask image according to the size of the initial scene image, respectively; moving the scaled initial pose image to the target position of the initial scene image to obtain the pose image, and moving the scaled initial object mask image to the target position of the initial scene image to obtain the object mask image.

[0184] In this embodiment, in the process of moving the initial pose image and the initial object mask image to the target position of the initial scene image, the initial pose image and the initial object mask image can be scaled according to the size of the initial scene image. The scaled initial pose image can be moved to the target position of the initial scene image to obtain the pose image, and the scaled initial object mask image can be moved to the target position of the initial scene image to obtain the object mask image.

[0185] Optionally, according to the position parameters specified by the user, the size of the initial object mask image and the initial pose image can be scaled and moved to the position corresponding to the position parameters. That is, in order to accurately control the generation of the model at the corresponding position, according to the position parameters specified by the user, the model mask and the pose estimation skeleton graph are scaled according to the size of the scene graph and moved to the corresponding position. Through the above method, it can be ensured that the model can be accurately placed at the position specified by the user when generating the image, thereby meeting the user's needs.

[0186] In the embodiment of the present application, the above method can accurately obtain the mask image and posture skeleton image of the model, and adjust them according to the position parameters specified by the user to ensure that the position and posture of the model can be accurately controlled when the image is subsequently generated, thereby generating a model scene image that meets the user's needs. These processing steps provide the necessary data and information basis for the smooth progress of the entire process.

[0187] The embodiment of the present application also provides a method for determining a model from the perspective of model training. Figure 3 is a flow chart of a method for determining a model according to an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps:

[0188] Step S302, obtaining a preference data set, wherein the preference data set includes posture image samples, object mask image samples, and target scene image sample pairs generated based on the auxiliary display object samples, the posture image samples are used to represent the posture of the auxiliary display object samples, the object mask image samples are used to represent the image area where the auxiliary display object samples are located in the initial scene image samples, and the target scene image sample pairs include target scene image samples whose evaluation results are higher than the evaluation result threshold, and target scene image samples whose evaluation results are lower than the evaluation result threshold.

[0189] In the technical solution provided in the above step S302 of the present application, if the diffusion model needs to be trained, the preference data set can be obtained first.

[0190] Optionally, the preference pair dataset is a dataset containing good images and bad images scored by humans. The above preference pair data is used to train the model, and during the training process the model will learn how to generate images that are more in line with human preferences. Specifically, each preference pair contains a set of input conditions (e.g., pose images, model mask images, and initial scene images), and the above input conditions are conditions for generating good images and bad images. Before training, the model can generate corresponding images through the above input conditions, and let humans score the generated images, thereby obtaining the preference dataset. The good images in the preference pair dataset are images that are considered to have high quality after being scored by humans. These good images are the target images that the model needs to learn to generate, and the generation results of the model should be as close to these good images as possible. The bad images in the preference pair dataset are images that are considered to have low quality after being scored by humans. These bad images are used to compare good images, and the images that the model needs to learn to generate should be as far away from these bad images as possible. Each preference pair also contains human scoring information on good images and bad images. These scoring information can be used as a supervisory signal to help the model learn to generate images that are more in line with human preferences.

[0191] Optionally, the input conditions of the pre-trained redrawing model can be a pose image, a mask image, and an initial scene image. Based on the above input conditions, multiple target scene images can be generated multiple times. And the generated target scene images can be scored by humans. Pairing the high-score images and low-score images generated under the same set of input conditions can obtain a preferred pair.

[0192] By training the preference pair dataset, the model can learn how to generate images that are more in line with human preferences and avoid generating low-quality images during the generation process. This training method can effectively improve the generation ability and generation effect of the generation model.

[0193] Step S304, using the preference data set, reinforcement learning is performed on the diffusion model to obtain a redrawing model, wherein the redrawing model is used to analyze the posture image of at least one auxiliary display object, the object mask image of the auxiliary display object and the initial scene image to be processed to obtain a target scene image, the auxiliary display object is used to assist in displaying at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

[0194] In the technical solution provided in the above step S304 of the present application, after determining the preference data set, the diffusion model can be reinforced for learning, so as to obtain a redrawing model that has been finally trained, wherein the redrawing model can be used to analyze the posture image of the auxiliary display object, the object mask image of the auxiliary display object and the initial scene image to be processed to obtain the target scene image. The auxiliary display object can be used to assist in displaying at least one display object in the initial scene image. The posture image can be used to represent the posture of the auxiliary display object. The object mask image can be used to represent the image area where the auxiliary display object is located in the initial scene image. The posture image and the object mask image can be determined by the auxiliary object image, the target position and the initial scene image. The auxiliary object image can be an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image can include the auxiliary display object. The target position can be contained in the image area.

[0195] Optionally, the limb variation phenomenon that may occur in the actual process of generating AI models and how to use reinforcement learning to fine-tune the redrawing model based on the preference information of human feedback so that the model output is more in line with the direction of human preferences.

[0196] Alternatively, due to the uncertainty of the diffusion model, even if a posture control network is designed to extract posture skeleton information and perform posture control, the generated AI model may still have limb variation, that is, the generated model's limb posture is significantly different from the expectation. This may be caused by insufficient training data, high complexity, or insufficient model design. In order to evaluate the naturalness of the limbs generated by the AI ​​model, preference information based on human feedback becomes an important evaluation indicator. Humans can provide preference information by observing and comparing the limb postures of different AI model generation results to express which results are more in line with their aesthetics and preferences.

[0197] In an embodiment of the present application, in order to solve the limb variation phenomenon, reinforcement learning can be used to fine-tune the redrawing model from the preference information of human feedback. The above method can make the redrawing model more inclined to the direction of human preferences when generating images, thereby improving the quality and naturalness of the generated results. A cutting-edge solution based on human feedback learning is the direct preference learning (Direct Preference Opt imizat ion, referred to as DPO) algorithm. The algorithm maximizes the probability of human preference answers and minimizes the probability of non-preference answers by giving preference information of different results generated by the same input, that is, humans choose which result is more in line with their preferences, so that the learning and generation process of the redrawing model can be guided by human feedback information to improve the satisfaction of the generated results. By fine-tuning the redrawing model using reinforcement learning and human feedback preference information, the redrawing model can better generate AI model images that meet human aesthetics and preferences, reduce limb variation phenomena, and improve the naturalness and quality of the generated results. The above method combines human subjective evaluation and the learning ability of the redrawing model, which can effectively improve the performance and performance of the redrawing model.

[0198] Optionally, the method based on human feedback learning can achieve personalized learning, improve the naturalness of the generated results and the timeliness of feedback. By collecting the preference information of human feedback, personalized learning of the output results of the redrawing model can be achieved, so that the generated AI model is more in line with human aesthetic standards. By maximizing the probability of human preferred answers, the body posture of the generated AI model can be made more natural, the occurrence of body variation can be reduced, and the generation quality of the redrawing model can be improved. The method based on human feedback learning can adjust the model parameters in time, so that the output results of the redrawing model are more in line with human needs, and improve the real-time and flexibility of the redrawing model.

[0199] Optionally, the DPO algorithm also has certain limitations, including problems that may occur when processing limb variation indicators and possible distortion of the model image generated by the model after training. Specifically, the DPO algorithm needs to implicitly learn preference information, which may lead to the erroneous learning of some "color preferences" in the early stage of training. This means that the model may over-focus on color information during training and ignore other important features, resulting in distortion of the generated model image in terms of light, shadow, color, etc. The DPO algorithm may have a preference for specific colors during training, which may cause the images generated by the model to be too prominent or unnatural in color, affecting the authenticity and quality of the generated results. The DPO algorithm will give a greater penalty to the generated results that do not meet human preferences, which may cause the model to be more inclined to avoid results that do not meet human preferences during training, while weakening the response to the generated results that meet human preferences. This may cause the model to be unable to maximize the use of the generated results that meet human preferences for gradient updates, affecting the learning effect of the model. Due to the possible color preference problems and the weakening of gradient updates in the DPO algorithm, the model image generated by the redrawing model after training may be distorted in terms of light, shadow, color, etc. This distortion may cause the generated results to be inconsistent with the actual situation, reducing the generation quality and authenticity of the model.

[0200] In summary, the DPO algorithm has some limitations, including color preference problems, gradient update weakening, and generated distortion. These problems may affect the training effect of the model and the quality of the generated results. The algorithm and training strategy can be further optimized to solve the above problems and improve the performance and effect of the redrawing model. For the color preference problem, it is possible to consider introducing a color regularization function to adjust the original color space of the model image generated by the redrawing model. For the gradient update weakening problem, the alternating training method of reinforcement learning and supervised fine-tuning can be used to increase the utilization of good images.

[0201] It should be noted that the above methods and processes for solving the limitations of the DPO algorithm are only for illustration and are not specifically limited here.

[0202] In an embodiment of the present application, a preference data set is used to represent a preference result of the posture of an auxiliary display object sample in a scene image sample. The auxiliary display object sample can be used to assist in displaying at least one display object sample. The auxiliary display object sample may be a specific object, person or other element, which exists in the scene image sample. The position and state of the posture of the auxiliary display object sample in the scene image will affect the preference result. The scene image sample is an image content containing the display object sample. The scene image sample is an image used to display and analyze the posture of the auxiliary display object sample, which contains the auxiliary display object sample. By analyzing these scene image samples and auxiliary display object samples, the preference result of the auxiliary display object sample in the scene image can be obtained. Using the preference data set, the diffusion model can be reinforced to learn, thereby obtaining a redrawing model. This redrawing model is used to analyze the posture image, object mask image and the initial scene image to be processed of at least one auxiliary display object, and finally generate a target scene image. By training and analyzing the redrawing model, combined with the information in the preference data set, it is possible to generate a target scene image that meets human preferences, which contains the posture and position of the auxiliary display object sample that meets the requirements. The above method can improve the precision and accuracy of image generation, so that the generated target scene image is more in line with the needs and preferences of the user.

[0203] In this embodiment, the redrawing model trained by the above method integrates human preferences into the AI ​​model generation process, and the reinforcement learning mechanism automatically learns and optimizes the generation process. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, and the visual effect and attractiveness of image generation are improved. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, so that image generation is more in line with user expectations and needs. In this way, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0204] The above method of this embodiment is further introduced below.

[0205] As an optional implementation, step S304, using the preference data set, performing reinforcement learning on the diffusion model to obtain the redrawing model, includes: using the preference data set, alternately performing reinforcement learning and supervised training on the diffusion model to obtain the redrawing model.

[0206] In this embodiment, in the process of performing reinforcement learning on the diffusion model using the preference data set, the preference data set can be used to alternately perform reinforcement learning and supervised training on the diffusion model to obtain a redrawing model.

[0207] Optionally, the diffusion model is trained by the proposed alternating training method of reinforcement learning and supervised fine-tuning to obtain a redrawing model that is finally trained. The redrawing model can be used to analyze the posture image of the auxiliary display object, the object mask image of the auxiliary display object, and the initial scene image to be processed, so as to obtain the target scene image.

[0208] In the embodiment of the present application, through the alternating training method of reinforcement learning and supervised fine-tuning, the redrawing model can learn the characteristics and rules of generating images that are more in line with human preferences. The above training method can improve the quality of the target scene image generated by the redrawing model, making the generated target scene image more realistic and more in line with human visual perception.

[0209] As an optional implementation, the preference data set is used to alternately perform reinforcement learning and supervised training on the diffusion model to obtain a redrawing model, including: in the reinforcement learning strategy and the supervised training strategy, a randomly selected reinforcement learning strategy or a supervised training strategy is determined as a target training strategy, wherein the reinforcement learning strategy is used to represent the rules for reinforcement learning of the diffusion model, and the supervised training strategy is used to represent the rules for supervised training of the diffusion model; according to the target training strategy, the preference data set or the alternating training data set is used to train the diffusion model to obtain the redrawing model.

[0210] In this embodiment, in the process of alternately performing reinforcement learning and supervised training on the diffusion model using the preference data set or the alternating training data set, the randomly selected reinforcement learning strategy or supervised training strategy is determined as the target training strategy in the reinforcement learning strategy and the supervised training strategy. The diffusion model can be trained using the corresponding data set according to the target training strategy to obtain a redrawing model, wherein the reinforcement learning strategy is used to represent the rules for performing reinforcement learning on the diffusion model, and can also be referred to as the training method of reinforcement learning. The supervised training strategy can be used to represent the rules for performing supervised training on the diffusion model, and can also be referred to as the training method of supervised fine-tuning.

[0211] In an embodiment of the present application, the limitations of the DPO algorithm can be overcome by using a training method of reinforcement learning and supervised fine-tuning. During the training process, the regularization loss of the LAB color space is introduced to constrain the color performance of the model when generating images. The LAB color space is a color space that divides color information into three channels: brightness (L), red-green hue (A), and blue-yellow hue (B). By introducing this regularization loss, the image generated by the model can be made more accurate in color. Before each training step begins, the training method of supervised fine-tuning or reinforcement learning is randomly selected with a certain probability. The above-mentioned alternating training method can balance between supervised fine-tuning and reinforcement learning, while ensuring that the model can be effectively trained under different training methods.

[0212] As an optional implementation, according to a target training strategy, a diffusion model is trained using a preference data set or an alternating training data set to obtain a redrawing model, including: when the target training strategy is a supervised training strategy, a first preference data set is determined from the preference data set, and external knowledge data is obtained, wherein the first preference data set includes target scene image samples whose evaluation results are higher than an evaluation result threshold, and the external knowledge data is used to represent knowledge allowed to be injected into the diffusion model; according to a supervised training strategy, a first drawing model is trained using the first preference data set and the external knowledge data to obtain a second drawing model, wherein the first drawing model is trained by a reinforcement learning strategy, and the redrawing model includes the second drawing model.

[0213] In this embodiment, in the process of training the diffusion model using the preference data set according to the target training strategy, if the target training strategy is a supervised training strategy, the first preference data set can be determined from the preference data set and the external knowledge data can be obtained. The first drawing model can be trained using the first preference data set and the external knowledge data according to the supervised training strategy to obtain the second drawing model, wherein the first preference data set can include target scene image samples with evaluation results higher than the evaluation result threshold, that is, the first preference data set can also be called a good image, that is, it can be a good image in the preference data set. The external knowledge data can also be called an out-of-distribution data set. The evaluation result threshold can be pre-set data or data set according to the actual training situation, which can be used to determine the degree to which the preference data in the preference data set meets the user's preference. It should be noted that the above-mentioned setting method of the evaluation result threshold is only for example and is not specifically limited here. The first drawing model can be obtained by training with a reinforcement learning strategy. The redrawing model can include a second drawing model. The first preference data set and the out-of-distribution data set can be collectively referred to as an alternating training data set.

[0214] Optionally, an out-of-distribution dataset refers to a dataset with a different distribution from the training data. In the field of deep learning, a dataset is usually divided into a training set, a validation set, and a test set, which are from the same data distribution and are used to train and evaluate the performance of the model. An out-of-distribution dataset refers to a dataset from different data distributions. The role of the out-of-distribution dataset is to provide more diversity and generalization for the redrawing model. By introducing an out-of-distribution dataset, the redrawing model can be trained on a wider data distribution, thereby improving its generalization ability and avoiding overfitting the training dataset. The introduction of an out-of-distribution dataset can help the model better adapt to different data distributions and improve the robustness and generalization ability of the redrawing model. In the proposed training framework, the out-of-distribution dataset is used to inject out-of-distribution knowledge into the model, which helps to prevent the degradation of the redrawing model's generation ability. By introducing an out-of-distribution dataset during the training process, the redrawing model's adaptability to different data distributions can be improved, thereby improving the diversity and quality of the generated results.

[0215] Optionally, by introducing out-of-distribution datasets and preference pair datasets during the training process, the redrawing model can learn more different types of information and knowledge. This helps to improve the generalization ability of the model, so that the redrawing model can better adapt to various situations when generating images of different scenes. The alternating training method can balance the training process of reinforcement learning and supervised fine-tuning, and avoid the redrawing model from falling into a local solution that satisfies the conditions or overfitting. This helps to improve the stability of the redrawing model, making the redrawing model more reliable and stable when generating images. The redrawing model obtained through training can analyze the posture image and object mask image of the auxiliary display object, so as to more accurately determine the position and state of the auxiliary display object in the target scene image. This helps to generate target scene images that are more in line with human preferences.

[0216] In summary, by training the diffusion model through the alternating training method of reinforcement learning and supervised fine-tuning proposed in the embodiment of the present application, a redrawing model with significant effect can be obtained, which improves the quality, generalization ability and stability of the generated image, and at the same time enhances the model's ability to analyze posture and object position, thereby achieving better generation of target scene images that meet human preferences.

[0217] Optionally, in the supervised fine-tuning training part, the redrawing model is trained using the alternating training dataset and the supervised fine-tuning loss is calculated. The alternating training dataset includes the out-of-distribution dataset and the good images in the preference pair dataset. The out-of-distribution dataset is used to inject out-of-distribution knowledge into the model to prevent the model generation ability from degrading; while the good images in the preference pair dataset are fine-tuned through supervised fine-tuning to make up for the problem of few good image gradient flows and low utilization rate in reinforcement learning.

[0218] As an optional implementation, according to the target training strategy, the diffusion model is trained using the preference data set to obtain the redrawing model, including: when the target training strategy is a reinforcement learning strategy, determining a first preference data set and a second preference data set from the preference data set, wherein the first preference data set includes target scene image samples whose evaluation results are higher than the evaluation result threshold, and the second preference data set includes target scene image samples whose evaluation results are lower than the evaluation result threshold; using a second drawing model with frozen model weights, performing noise prediction on the first preference data set and the second preference data set to obtain a first noise prediction result, and decoding a first decoded image from the first noise prediction result, wherein the second drawing model is trained by a supervised training strategy; using a second drawing model with unfrozen model weights, performing noise prediction on the first preference data set and the second preference data set to obtain a second noise prediction result, and decoding a second decoded image from the second noise prediction result; training a first drawing model based on the first noise prediction result, the first decoded image, the second noise prediction result, and the second decoded image, wherein the redrawing model includes the first drawing model.

[0219] In this embodiment, in the process of training the diffusion model using the preference data set according to the target training strategy, when the target training strategy is a reinforcement learning strategy, a first preference data set and a second preference data set can be determined from the preference data set. The first preference data set and the second preference data set are predicted by using the second rendering model with frozen model weights to obtain a first noise prediction result. A first decoded image can be decoded from the first noise prediction noise result. The first preference data set and the second preference data set can be predicted by using the second rendering model with unfrozen model weights to obtain a second noise prediction result. A second decoded image can be decoded from the second noise prediction result. The first rendering model can be trained based on the first noise prediction result, the second decoded image, the second noise prediction result, and the second decoded image, wherein the second preference data set can be a set of target scene image samples below the evaluation result threshold, which can also be called a bad image. The first noise prediction result can be the noise predicted by the redrawing model. The second redrawing model can be obtained by training through a supervised training strategy. The redrawing model can include the first rendering model.

[0220] Optionally, in the process of decoding the first decoded image from the first noise prediction result, a deterministic deep denoising strategy (Denois ing Deep Image Prior, referred to as DDIM) can be used to restore the first noise prediction result to a latent variable at time 0. The latent variable can be integrated into the VAE decoder to decode the first decoded image.

[0221] Optionally, freezing model weights means setting some or all model parameters to non-trainable during the training and redrawing of the model, that is, not updating the values ​​of these parameters during the training process. By freezing the model weights, these parameters can be kept fixed during the training process and will not be affected by gradient updates.

[0222] Optionally, in the reinforcement learning training part, the training data comes from the pre-collected preference pairs. Each preference pair contains the input required to generate the image and two pictures scored by humans: good and bad. During the training process, two redrawing models are initialized, one of which freezes the model weights and does not update. The DPO direct preference loss is calculated using the noise predictions of these two models, and the original image is decoded using the predicted noise, and the LAB spatial loss is calculated.

[0223] In the embodiment of the present application, the above-mentioned training framework and method of reinforcement learning training and supervised fine-tuning training can better solve the limitations of the DPO algorithm and improve the accuracy and quality of the model when generating images. This alternating training method combining reinforcement learning and supervised fine-tuning can effectively balance the advantages of different training methods and improve the training effect and generation results of the model.

[0224] As an optional implementation, based on the first noise prediction result, the first decoded image, the second noise prediction result and the second decoded image, a first drawing model is trained, including: determining a reinforcement learning loss based on the first noise prediction result and the second noise prediction result; determining a color space loss based on the first decoded image and the second decoded image; and using the reinforcement learning loss and the color space loss to train the first drawing model.

[0225] In this embodiment, in the process of training the first rendering model based on the first noise prediction result, the first decoded image, the second noise prediction result and the second decoded image, a reinforcement learning loss can be determined based on the first noise prediction result and the second noise prediction result. A color space loss is determined based on the first decoded image and the second decoded image. The first rendering model is trained using the reinforcement learning loss and the color space loss, wherein the reinforcement learning loss can be a DPO reinforcement learning loss, which can also be called a DPO direct preference loss. The color space loss can be a LAB color space regularization loss, which can also be called a LAB space loss.

[0226] Optionally, in reinforcement learning training, the training data is a preference pair collected in advance, one of which includes the input required for generation, and good and bad images scored by humans. These good and bad images are used to train the model for reinforcement learning. By comparing good and bad images, the model can learn to generate images that are more in line with human preferences. During the training process, two redrawing models (a second drawing model corresponding to the frozen model weights and a second drawing model corresponding to the unfrozen model weights) are initialized, and their weights are the same. The weight of one of the redrawing models will be frozen (that is, the frozen model weights), and the weights will not be updated. Through the above settings, a stable model can be maintained during the training process, while the other model is used to learn to update the weights. By using the noise prediction of the above two redrawing models, a direct preference loss can be calculated. This loss function can help the model learn to generate images that are more in line with human preferences, thereby improving the quality of the generated images. In addition, the predicted noise can be used to directly predict and decode the original image, and calculate the LAB space loss. This loss function can help the model learn to reconstruct the original image more accurately, thereby improving the accuracy of the generated image.

[0227] In summary, by using the pre-collected preferences to conduct reinforcement learning training on the dataset, combined with two initialized redrawing models and different loss function calculation methods, the model can learn to generate images that are more in line with human preferences and improve the quality and accuracy of generated images. The above training method can effectively improve the performance of the model, making the generated target scene images more in line with user needs and expectations.

[0228] Optionally, the DPO reinforcement learning loss can be determined by the following formula:

[0229]

[0230] Among them, L1 can be used to represent the DPO reinforcement learning loss; ∈ can be used to represent the noise predicted by the second redrawing model, that is, the first noise prediction result; ω can be used to represent the good images in the preference data set, that is, the first preference data set; l can be used to represent the bad images in the preference data set; ref can be used to represent the output result of the frozen redrawing model, that is, the first decoded image, so ε ref It can be used to represent the second noise prediction result; σ can be used to represent the sigmoid function; β can be used to represent the weight used to balance the direct preference loss and the regularization loss. By adjusting the value of β, the importance of the above two losses in the training process can be controlled; T can be used to represent the noise adding time step, which can be preset to 1000. This is only an example and is not limited to this; λ t Can be used to represent the signal-to-noise ratio.

[0231] Alternatively, the LAB color space loss may be determined by the following formula:

[0232]

[0233] Among them, L2 can be used to represent the LAB color space loss; L, a, and b can be used to represent the LAB color space pixels in the decoded image respectively.

[0234] As an optional implementation, the method also includes: expanding the initial number channels of the diffusion model to obtain the target number channels; using the preference data set to perform reinforcement learning on the diffusion model to obtain a redrawing model, including: using the preference data set to perform reinforcement learning on the diffusion model of the target number channels to obtain a redrawing model.

[0235] In this embodiment, the initial number of channels of the diffusion model can be expanded to obtain the target number of channels. Therefore, in the process of using the preference data set to perform reinforcement learning on the diffusion model, the preference data set can be used to perform reinforcement learning on the diffusion model of the target number of channels to obtain a redrawing model, wherein the initial number of channels can be four channels. The target number of channels can be nine channels.

[0236] Optionally, the function of the redrawing model is to generate a model in a specified pose controlled by a pose control network in the mask part of the scene graph. This means that the model needs to generate a corresponding model image in a given scene graph according to the specified pose information provided by the pose control network. The original stable diffusion model has only one four-channel latent variable feature as input, but now needs to adapt to three input data: model mask, scene graph, and scene graph with mask. To achieve this goal, the input layer of the original network structure needs to be expanded to nine channels to process these three inputs. The input data of the redrawing model includes model mask, scene graph, and scene graph with mask. The model mask is used to specify the position of the model, the scene graph provides background information, and the scene graph with mask combines the information of the model mask and the scene graph.

[0237] Optionally, by expanding the input layer of the original network structure to nine channels, the redrawing model can simultaneously receive and process inputs from the model mask, scene graph, and masked scene graph, so as to better utilize the above information and generate a model image that meets the requirements. The redrawing model needs to keep the original scene graph unchanged while generating an AI model in the masked part. By expanding the input layer and adapting to the new input data, the redrawing model can better achieve this generation requirement and improve the quality and accuracy of the model image. By adjusting the network structure and processing the input data, the redrawing model can better handle the model generation task, ensure that the generated model image meets the specified requirements of the posture control network, and maintain the integrity of the original scene graph. The above adjustments and adaptability can improve the performance of the model and the quality of the generation results, so that it can better meet the needs of practical applications.

[0238] Optionally, reinforcement learning training is an offline strategy, which means that the data used in the training process is a previously collected preference pair dataset, rather than interacting with the outside world in a real-time environment to collect data. Therefore, during the reasoning process, only the redrawing model trained by reinforcement learning needs to be used for reasoning, without the need for real-time feedback and training, which makes the reasoning framework more concise and efficient.

[0239] Optionally, since reinforcement learning training is an offline strategy, the redrawing model has learned and optimized the data set through the pre-collected preferences during the training process, and obtained a trained redrawing model. Therefore, during the reasoning process, there is no need to interact with the outside world, nor is there any need to collect data and conduct training in real time. It is only necessary to use the trained model for reasoning. Since only the trained redrawing model needs to be used in the reasoning process, and no additional real-time data collection and training process is required, the reasoning framework will be more concise. The reasoning framework only needs to load the trained model, input the data to be processed, and then perform reasoning and generate results. The whole process is relatively simple and efficient. The offline strategy reinforcement learning training method can improve the efficiency of the reasoning process. Since the model has learned the user's preference information during the training process and does not require additional real-time interaction, the reasoning process can be carried out faster, saving computing resources and time costs.

[0240] In summary, the embodiment of the present application adopts an offline strategy reinforcement learning training method to make the reasoning framework more concise and efficient. By performing reasoning with a pre-trained redrawing model, the efficiency and speed of the reasoning process can be improved while ensuring the accuracy of the generated results. The above settings can make the system easier to deploy and apply, while providing fast and reliable generation results.

[0241] The embodiment of the present application also provides a method for generating an image from an e-commerce application scenario. Figure 4is a flow chart of a method for generating an image according to an embodiment of the present application, such as Figure 4 As shown, the method may include the following steps:

[0242] Step S402: identifying an initial scene image to be processed from an e-commerce platform, wherein the image content of the initial scene image includes at least one displayed product.

[0243] In the technical solution provided in the above step S402 of the present application, an initial scene image to be processed can be identified from the e-commerce platform, wherein the image content of the initial scene image can include displayed products.

[0244] Optionally, if the user has generated a target scene image with an AI model, the user can upload an initial scene image corresponding to the corresponding display product on the operation interface corresponding to the e-commerce platform of the terminal device. The successfully uploaded initial scene image can be transmitted to the server via the network. In the server, the initial scene image can be identified and analyzed.

[0245] Optionally, an initial scene image is obtained from an e-commerce platform, and the initial scene image is analyzed and recognized through image recognition and processing technology. The initial scene image may be a picture of goods or products displayed on the e-commerce platform, including a scene of displaying products. The content of the initial scene image includes at least one displayed product, that is, a displayed product to be sold. In addition to the product itself, the image may also include relevant backgrounds for displaying the product, such as rooms, home environments, etc., to provide a more realistic and vivid display effect. Related furnishings such as carpets, coffee tables, bookshelves, and vases can enrich the scene background and enhance the display effect of the product.

[0246] Optionally, if the application scenario is e-commerce promotion, the displayed product is the commodity or product to be promoted. The initial scene image containing the commodity is also called the promotion scene image, which is used to show the effect of the product and attract consumers. By identifying the initial scene image, the system can obtain the promotion scene image to better display the product and promote sales. The initial scene image does not include models that assist in displaying the displayed product.

[0247] Step S404, determining an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the display product.

[0248] In the technical solution provided in the above step S404 of the present application, after the initial scene image is identified, the auxiliary object image to be drawn can be determined at the target position in the initial scene image.

[0249] Optionally, according to various needs of users, auxiliary display objects that meet the needs are set, and corresponding auxiliary object images are formed.

[0250] Optionally, an image generation application associated with the e-commerce platform is set up, and the application can be used to generate corresponding auxiliary display objects based on user needs and generate target scene images. In this application, a large amount of model attributes, postures, clothing and other information that can be selected by users can be set. Model attributes can include male / female, adult / child, and skin color, etc., postures can include lying, standing, lying, walking and sitting, etc., and clothing can include shirts, dresses, etc. Users can select more suitable information to assist the current display product from the large amount of model attributes, postures and clothing obtained above according to their own needs on the operation interface of the application on the terminal device, so as to form corresponding auxiliary display objects in combination with the information selected by the user. Among them, if the scene for which the image generation is targeted is a home scene, the above application can be a home platform, which is only used as an example here and is not specifically limited.

[0251] Optionally, in the above application, corresponding controls or operations can be set to adjust the generated auxiliary object image, such as moving, resizing and rotating, etc. Therefore, after the corresponding auxiliary object image is generated based on the information selected by the user, the user can perform corresponding operations to adjust the auxiliary object image to a satisfactory size and angle according to his own needs, and can move it to the target position.

[0252] Step S406, based on the auxiliary object image, the target position and the initial scene image, determine the posture image and the object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position.

[0253] In the technical solution provided in the above step S406 of the present application, after determining the auxiliary object image to be drawn at the target position in the initial scene image, the posture image and object mask image of the auxiliary display object can be determined based on the auxiliary object image, the target position and the initial scene image.

[0254] Optionally, the auxiliary object image is analyzed to obtain the posture of the auxiliary display object, thereby determining the posture skeleton information of the auxiliary display object. Based on the posture skeleton information, a posture image corresponding to the posture skeleton of the auxiliary display object can be obtained. By performing mask generation on the auxiliary object image, a corresponding object mask image is obtained.

[0255] Optionally, the posture image is used to represent the posture of the auxiliary display object, that is, the posture and action of the auxiliary display object in the image. Through the posture image, the dynamic characteristics of the auxiliary display object can be more vividly displayed, making it more lifelike in the scene. The object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, that is, to mark the position and area occupied by the display product. Through the object mask image, the accurate segmentation and positioning of the auxiliary display object can be achieved, providing key reference information for subsequent image synthesis and processing. By determining the posture image and the object mask image, the auxiliary display object can be made more vivid and natural in the generated image, improving the realism and attractiveness of the image. Based on the posture skeleton information and mask generation technology, the auxiliary display object can be accurately located and segmented, so that it can be better integrated with the initial scene image, enhancing the visual effect and expressiveness of the image. Through the analysis and processing of the auxiliary object image, the posture and position of the auxiliary display object can be better controlled, and the fine regulation and optimization of the display effect can be achieved.

[0256] Step S408, using the redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain the target scene image, wherein the redrawing model is obtained by performing reinforcement learning on the diffusion model.

[0257] In the technical solution provided in the above step S408 of the present application, after determining the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image, the redrawing model can be used to analyze the posture image, the object mask image and the initial scene image to obtain the corresponding target scene image.

[0258] Optionally, by analyzing the data input to the redrawing model, such as pose images, object mask images, and initial scene images, we can learn how to reasonably integrate the pose and position of the auxiliary display object into the initial scene image to generate a target scene image that meets human preferences. The preference dataset contains human ratings and preference information for generated images, which can help the redrawing model learn how to generate target scene images that are more in line with human preferences. The above data plays a key role in guiding the redrawing model to generate high-quality target scene images.

[0259] Optionally, by analyzing the pose image, the object mask image, and the initial scene image using the redrawing model, the pose and position of the auxiliary display object can be reasonably integrated with the initial scene image. This helps to generate a target scene image that is more in line with human aesthetic standards and preferences, and improves the quality and visual appeal of the image. The analysis capability of the redrawing model can help the model better understand and process the input data and generate the target scene image more accurately. Through the comprehensive analysis of information such as pose, position, and mask, the scene can be accurately redrawn, making the generated image more natural and vivid.

[0260] In the embodiment of the present application, by analyzing the posture image, the object mask image and the initial scene image using the redrawing model in the above method, a target scene image that meets human preferences can be effectively generated. By integrating various data such as posture, position and preference information, the quality and realism of the generated image can be improved, the user's demand for visual effects can be met, and the expressiveness and attractiveness of the image generation can be improved.

[0261] Step S410, returning the target scene image to the e-commerce platform.

[0262] In the technical solution provided in the above step S410 of the present application, the generated target scene image with the AI ​​model can be returned to the e-commerce platform for users to retain and use.

[0263] Optionally, after the server generates the target scene image, the target scene image can be transmitted to the operating interface of the user's terminal device through the network, and the user can choose whether to use the target scene image according to his or her own needs. If you choose to use it, you can save the target scene image and upload it to the product details page in the user's store on the e-commerce platform to display the effect of the corresponding product. You can also directly display the target scene image to the corresponding location through the relevant controls in the application that generates the target scene image. For example, the application can include an "e-commerce delivery" control. By clicking on the control, you can deliver advertisements to the e-commerce platform based on the target scene image. It should be noted that the above-mentioned usage and operation process of the target scene image are only for example purposes and are not specifically limited here.

[0264] In an embodiment of the present application, an initial scene image to be processed is identified from an e-commerce platform; an auxiliary object image to be drawn at a target position in the initial scene image is determined; based on the auxiliary object image, the target position and the initial scene image, a posture image and an object mask image of the auxiliary display object are determined; using a redrawing model, the posture image, the object mask image and the initial scene image are analyzed to obtain a target scene image; and the target scene image is returned to the e-commerce platform.

[0265] In this embodiment, a redrawing model is set up to integrate human preferences into the AI ​​model generation process. The generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, and the visual effect and attractiveness of image generation are improved. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, so that image generation is more in line with user expectations and needs. In this way, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0266] The embodiment of the present application also provides a method for generating an image from the human-computer interaction side. Figure 5 is a flow chart of a method for generating an image according to an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps:

[0267] Step S502 , in response to an input instruction on the operation interface, an initial scene image to be processed is displayed on the operation interface, wherein the image content of the initial scene image includes at least one display object.

[0268] In the technical solution provided in the above step S502 of the present application, if an input instruction on the operation interface is detected, the initial scene image to be processed can be displayed on the operation interface, wherein the image content of the initial scene image can include a corresponding display object.

[0269] Optionally, if the user needs to generate a corresponding target scene image with an AI model to better display the display object, the initial scene image with the display object can be uploaded by performing a corresponding input operation on the operation interface. If the operation interface detects the above input operation, it can generate a corresponding input instruction and display it on the operation interface.

[0270] Step S504, in response to the image generation instruction acting on the operation interface, a target scene image corresponding to the initial scene image is displayed on the operation interface, wherein the target scene image is obtained by analyzing the posture image of at least one auxiliary display object, the object mask image of the auxiliary display object and the initial scene image using a redrawing model, the redrawing model is obtained by reinforcement learning of the diffusion model, the auxiliary display object is used to assist in displaying at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

[0271] In the technical solution provided in the above step S504 of the present application, when an image generation instruction is detected on the operation interface, the target scene image corresponding to the initial scene image can be displayed on the operation interface.

[0272] Optionally, after the initial scene image to be processed is determined, the auxiliary object image to be drawn at the target position in the initial scene image may be determined.

[0273] Optionally, according to various needs of users, auxiliary display objects that meet the needs are set, and corresponding auxiliary object images are formed.

[0274] Optionally, after determining the auxiliary object image to be drawn at the target position in the initial scene image, a posture image and an object mask image of the auxiliary display object may be determined based on the auxiliary object image, the target position and the initial scene image.

[0275] Optionally, the auxiliary object image is analyzed to obtain the posture of the auxiliary display object, thereby determining the posture skeleton information of the auxiliary display object. Based on the posture skeleton information, a posture image corresponding to the posture skeleton of the auxiliary display object can be obtained. By performing mask generation on the auxiliary object image, a corresponding object mask image is obtained.

[0276] Optionally, after determining the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image, the posture image, the object mask image and the initial scene image can be analyzed using the redrawing model to obtain the corresponding target scene image.

[0277] Optionally, by analyzing the data input to the redrawing model, such as pose images, object mask images, and initial scene images, we can learn how to reasonably integrate the pose and position of the auxiliary display object into the initial scene image to generate a target scene image that meets human preferences. The preference dataset contains human ratings and preference information for generated images, which can help the redrawing model learn how to generate target scene images that are more in line with human preferences. The above data plays a key role in guiding the redrawing model to generate high-quality target scene images.

[0278] Optionally, after the target scene image is determined, the target scene image may be transmitted to a terminal device of the user via a network, and may be displayed in an operation interface of the terminal device.

[0279] In an embodiment of the present application, in response to an input instruction acting on the operation interface, an initial scene image to be processed is displayed on the operation interface; in response to an image generation instruction acting on the operation interface, a target scene image corresponding to the initial scene image is displayed on the operation interface. In this embodiment, a redrawing model is set to integrate human preferences into the AI ​​model generation process, and the generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, improving the visual effect and attractiveness of image generation. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, making image generation more in line with user expectations and needs. This achieves the technical effect of improving the accuracy of image generation and solves the technical problem of low image generation accuracy.

[0280] According to an embodiment of the present application, an embodiment of a system for generating an image is also provided. Figure 6 is a schematic diagram of an image generation system according to an embodiment of the present application, such as Figure 6 As shown, the image generation system 600 may include: a client 601 and a server 602 .

[0281] The client 601 is used to upload an initial scene image to be processed, wherein the image content of the initial scene image includes at least one display object. The client 601 is also used to display a target scene image.

[0282] In this embodiment, the client 601 can upload the initial scene image to be processed, and transmit the initial scene image to the server 602 through the network.

[0283] Optionally, if the user needs to use a certain auxiliary display object to assist in displaying the current display object, the user can perform input or selection operations on the operation interface of the corresponding client 601 to upload the initial scene image to be generated to the client 601. It is also possible to select information that meets the user's needs from a large amount of auxiliary display object related information displayed on the operation interface, such as posture, clothing, attributes, etc., to generate a corresponding auxiliary display object. The client 601 can send the initial scene image and the auxiliary object image to the server 602 through the network.

[0284] Optionally, the initial scene image can be uploaded by the user according to the user's needs, or the initial scene image that meets the user's needs can be automatically generated according to the display object to be displayed.

[0285] For example, if a user needs to display a certain display object, the user can select a suitable display scene as the initial scene image according to the characteristics and positioning of the display object. For example, if the display object is a sofa, the user can select a modern-style living room, including furniture and decorations such as sofas, coffee tables, and carpets. The user can take or select a photo of an indoor scene that meets the requirements as the initial scene image. Through the above process, the generated initial scene image can show the effect of the display object in the real environment, helping users to better understand the appearance and matching of the product. After determining the initial scene image, if image generation is required, the corresponding operation can be performed on the operation interface of the terminal device, the corresponding initial scene image can be selected, and uploaded. After uploading the initial scene image, the initial scene image can be transmitted to the server through the network. The server can identify and analyze the received initial scene image, determine the display object therein, and use it for subsequent image processing.

[0286] For another example, if a user needs to display a certain display object, the corresponding initial scene image can be automatically generated according to the user's own needs and the display object to be displayed. For example, if the display object is a new car product. The user can select parameters such as the brand, model, color of the car and the background requirements for displaying the car through the tools provided by the server, and then the server can automatically generate an image that meets the above parameters as the initial scene image. Through the above process of generating the initial scene image, the effect of the display object in the display environment can be generated quickly and efficiently. On the operation interface of the tool that can automatically generate the initial scene image on the terminal device, the requirements for the initial scene image to be generated can be input. The above requirements can be transmitted to the server through the network. The server can identify and analyze the received requirements, automatically generate the initial scene image that meets the above requirements, and can return the above initial scene image to the operation interface through the network for display, so that the user can confirm whether it needs to be adjusted. If no adjustment is required, the server can retain the initial scene image for subsequent image generation.

[0287] Server 602 is used to determine an auxiliary object image to be drawn at a target position in an initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the display object; based on the auxiliary object image, the target position and the initial scene image, determine a posture image and an object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position; use a redrawing model to analyze the posture image, the object mask image and the initial scene image to obtain a target scene image, wherein the redrawing model is obtained by reinforcement learning of a diffusion model.

[0288] In this embodiment, after receiving the initial scene image, the server 602 can determine the auxiliary object image to be drawn at the target position in the initial scene image. The posture image and the object mask image of the auxiliary display object can be determined based on the auxiliary object image, the target position and the initial scene image. Using the redrawing model, the posture image, the object mask image and the initial scene image can be analyzed to obtain the target scene image.

[0289] Optionally, the user can select appropriate information to assist the current display object from the large number of model attributes, postures and clothing obtained above according to his / her needs on the operation interface of the corresponding application on the terminal device, so as to form a corresponding auxiliary display object in combination with the information selected by the user. In the above application, corresponding controls or operations can also be set to adjust the generated auxiliary object image, such as moving, resizing and rotating. The user can perform corresponding operations to adjust the auxiliary object image to a satisfactory size and angle according to his / her needs, and can move it to the target position.

[0290] Optionally, the auxiliary object image is analyzed to obtain the posture of the auxiliary display object, thereby determining the posture skeleton information of the auxiliary display object. Based on the posture skeleton information, a posture image corresponding to the posture skeleton of the auxiliary display object can be obtained. By performing mask generation on the auxiliary object image, a corresponding object mask image is obtained.

[0291] Optionally, by analyzing the data input to the redrawing model, such as pose images, object mask images, and initial scene images, we can learn how to reasonably integrate the pose and position of the auxiliary display object into the initial scene image to generate a target scene image that meets human preferences. The preference dataset contains human ratings and preference information for generated images, which can help the redrawing model learn how to generate target scene images that are more in line with human preferences. The above data plays a key role in guiding the redrawing model to generate high-quality target scene images.

[0292] In an embodiment of the present application, an image generation system 600 is provided. An initial scene image to be processed is uploaded by a client 601; an auxiliary object image to be drawn at a target position in the initial scene image is determined by a server 602; a posture image and an object mask image of an auxiliary display object are determined based on the auxiliary object image, the target position and the initial scene image; the posture image, the object mask image and the initial scene image are analyzed by a redrawing model to obtain a target scene image; wherein the client is used to display the target scene image. In this embodiment, a redrawing model is set to integrate human preferences into the AI ​​model generation process, and the generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and conformity of AI model generation. This can effectively avoid the limb abnormalities and variations that frequently occur in related technologies, thereby optimizing the generation effect of AI models, and the generated AI models are more natural and meet human aesthetic requirements, improving the visual effect and attractiveness of image generation. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, making image generation more in line with user expectations and needs. In turn, the technical effect of improving the accuracy of image generation is achieved, solving the technical problem of low image generation accuracy.

[0293] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, for example, the data for verification, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0294] Below, the embodiments of the present application are further explained in combination with the application of promotion of home furnishings, decorations, etc. in the home furnishing industry.

[0295] At present, in the wave of digital transformation of the home furnishing industry, online display has become a key link in attracting customers and improving the shopping experience. In order to stand out in visual promotion, many home improvement brands and e-commerce platforms have sought innovation to showcase their products, such as furniture and decorations, with highly attractive model pictures. However, the relevant technology relies on actual shooting, which is not only costly, but also difficult to flexibly respond to changing market demands and personalized display needs. In view of this, the application of AI technology in the field of home furnishing is particularly important, especially in the automatic generation of high-quality model pictures, aiming to efficiently create visual content that is both beautiful and in line with product characteristics through intelligent means.

[0296] In a related technology, a redrawing prediction noise method based on a diffusion model is proposed, in which for the area outside the redrawn part, the noise addition result is calculated at each time step, and the final noise is the concatenation of the noise predicted by the redrawn part and the previous noise addition result of the non-redrawn part. By operating only the area that needs to be redrawn and retaining the noise addition result of the non-redrawn part, the noise of the redrawn part can be accurately controlled to avoid unnecessary modifications to the non-redrawn part. This helps to maintain the information integrity and accuracy of the original image. However, since only the noise learning is performed on the redrawn part, unnatural transitions or boundary effects may occur between the redrawn part and the original image. This is because the noise learning of the redrawn part may not completely match the information of the non-redrawn part, resulting in obvious differences in the effect of the generated image between the redrawn part and the non-redrawn part, thereby affecting the naturalness and realism of the generated image. Therefore, there is still a technical problem of low image generation accuracy.

[0297] Further, the present application provides a method for enhancing the image of a virtual model of a home scene based on reinforcement learning, which aims to fundamentally solve the challenges currently faced by the AI ​​model generation technology of the above-mentioned related methods. This embodiment focuses on the model generation system of the home platform, which aims to solve the limb abnormalities or deformations that may occur in the generation process of the prior art, thereby improving the aesthetics and realism of the generated image. The system combines human feedback preference information with reinforcement learning algorithms, and optimizes the AI ​​model generation process and improves the quality of the output image through continuous learning and iteration. Through the above method, attention can be paid to maintaining the integrity and consistency of the original scene graph to ensure that the integration of the AI ​​model does not interfere with the product display information. By seamlessly integrating the AI ​​model, the generated image looks more natural and real, and the fidelity of the overall scene is improved. It can provide users with a high degree of freedom and control, and can flexibly specify the model's posture, position and appearance details according to specific needs to meet different display needs. This highly customized feature can help users better customize and personalize the display content. Through deep learning of human preference information, it is committed to reducing unnatural limbs and ensuring that each frame of the generated image can achieve visual harmony and attractiveness. This helps improve the aesthetics of the image, makes the model image conform to aesthetic standards, and can stably convey the brand style and product features.

[0298] In summary, the embodiment of the present application optimizes the AI ​​model generation process by integrating human feedback preference information with reinforcement learning algorithms, thereby improving the aesthetics and realism of the generated images. At the same time, the system's customization capabilities and optimized visual experience help to improve the user experience and meet diverse display needs, while ensuring that the generated images meet aesthetic standards and can effectively convey brand style and product features. Through continuous learning and iteration, the system can continuously improve the quality of generated images and provide users with better visual effects. Thereby achieving the technical effect of improving the accuracy of image generation and solving the technical problem of low image generation accuracy.

[0299] The above method of this embodiment is further introduced below.

[0300] In the embodiments of the present application, Figure 7 is a schematic diagram of an interface for generating a scene promotion picture with a model according to an embodiment of the present application, such as Figure 7As shown, it is an interface in the application program provided in the embodiment of the present application, on which the promotion scene diagram corresponding to the scene promotion diagram with models required to be generated by the user can be uploaded on the interface. The interface may include a material library, which may include model attributes, model postures, model clothing, and output configurations. The user can select the desired model attributes in the material library according to their own needs, such as female, Asian, and adult, and can also select corresponding model postures, such as sitting or standing. The clothing of the model to be generated can also be selected, such as a dress or a shirt with pants. The number of pictures to be output can also be selected. The interface may include an editing area, and the user can drag the model to be placed in a specified position according to their own needs, and can also be further adjusted by mirroring, size adjustment, and rotation. Based on the model position and style determined by the above-mentioned user, the corresponding scene promotion diagram with models can be generated in the generation result area to increase the visual appeal of the promotion diagram. It is used to filter out the scene promotion diagram with models that meet their own needs from the generation results.

[0301] Alternatively, if Figure 7 As shown, after generating a scene promotion picture with a model, the user can also directly deliver corresponding information, such as advertisements or links, to promote the corresponding products by clicking the e-commerce delivery control.

[0302] In the related art, there are many deficiencies in the process of generating promotional scenes with models. Specifically, when the user selects a specific model pose, appearance attributes and clothing combination on the platform, and specifies the desired scene map and model placement, an AI model image that matches it should be accurately generated. Unfortunately, Figure 8 It is a schematic diagram of generating abnormal conditions based on a scene promotion diagram with a model, such as Figure 8 As shown, the related technology may cause the generated model to have abnormal or deformed limbs in some cases, which seriously damages the overall aesthetic effect of the image, for example, the model may have three legs.

[0303] Fig. 9 is a schematic diagram of a process of generating a home scene virtual model image based on reinforcement learning enhancement according to an embodiment of the present application, such as Fig. 9As shown, the generation process may include position parameters 901, user scene graph 902, model posture graph 903, skeleton graph and model mask 904, posture control network 905, redrawing model 906 trained based on reinforcement learning, and model scene graph 907. By using computer vision technology, posture skeleton information is extracted from a given model graph, that is, model posture graph 903, and abstract information of the posture skeleton is obtained by using the self-trained posture control network 905, and sent to a redrawing model based on a diffusion generation model, that is, the redrawing model 906 trained based on reinforcement learning, and finally a model scene generalization graph (model scene graph 907) of a specified position (position parameter 901) and posture is generated.

[0304] Alternatively, if Fig. 9 As shown in the figure, a user uploads a user scene graph 902 and a selected model posture graph 903, and also records the position parameters 901 where the user drags the model to place it. Based on the above information, the corresponding posture skeleton graph and model mask 904 of the model are extracted. The skeleton graph is then input into the posture control network 905 to abstract the information of the posture skeleton, and is input into the redrawing model 906 based on reinforcement learning training together with the user scene graph 902, the skeleton graph after position transformation, and the model mask 904. The information output by the posture control network will use the cross-attention mechanism to inject control information into the generation process of the redrawing model. Finally, the redrawing model generation graph is overlapped back to the original scene graph using the initial position parameters, and the model scene graph 907 is output.

[0305] In this embodiment, in the process of determining the skeleton graph and the model mask, Fig.10 is a schematic diagram of a skeleton graph transformation according to an embodiment of the present invention, such as Fig.10 As shown, the mask image of the model can be obtained by using the alpha transparency channel according to the 4-channel model pose image selected by the user. At the same time, the pose estimation network is used to perform pose estimation on the model image 1001 to obtain a pose skeleton image 1002. In order to accurately control the generation of the model at the corresponding position, the embodiment of the present application scales the model mask and the pose estimation skeleton image according to the size of the scene image according to the position parameters specified by the user and moves them to the corresponding position, thereby ensuring accurate control of the position when the image is subsequently generated.

[0306] Optionally, the posture estimation method adopted above uses a posture estimation network ViTPose based on a Transformer structure. Fig.11 is a schematic diagram of a posture estimation network according to an embodiment of the present invention, such as Fig.11As shown, the posture estimation network may include an Encoder and a Decoder. The Encoder may include Multi-Head Self-Attention (MHSA), Layer Normalization (LN), Feedforward Neural Network (FFN), Deconvolution (Deconv), Batch Normalization (BN), Residual Connection (ReLU), Predictor, Bilinear, and L Transformer Blocks. The model uses the transformer architecture to replace the convolutional neural network, and better captures the relationship between different key points through the self-attention mechanism to achieve efficient feature learning. Its parallel processing capability improves the efficiency of feature extraction, and can fully consider the global context information, making it perform better in complex scenes.

[0307] Alternatively, if Fig.11 As shown in the figure, the Transformer structure has strong parallel processing capabilities and can process information at different locations in parallel, thereby improving the efficiency of feature extraction. This means that ViTPose can learn the relationship and feature representation between key points more quickly, speeding up the process of pose estimation. The Transformer structure can fully consider global context information, which makes ViTPose perform better when dealing with complex scenes. Through the self-attention mechanism, the model can simultaneously focus on the relationship between different regions, thereby better understanding the semantic information of the overall image. When processing an image, ViTPose divides it into multiple blocks and uses the self-attention mechanism to obtain information connections between different blocks in parallel. This can improve the ability to extract global information, allowing the model to better capture the relationship between different regions, thereby improving the accuracy of pose estimation. ViTPose allows users to set the number of local areas to be divided, so that it can be adjusted according to the needs of specific tasks and scenarios, making the model more flexible and adaptable to different situations.

[0308] Through the above method, the Transformer-based posture estimation network ViTPose has strong feature learning and global information extraction capabilities, and can process image information in parallel through the self-attention mechanism to better capture the relationship between different regions, thereby improving the efficiency and accuracy of posture estimation and performing better in complex scenes. The advantages of this structure enable ViTPose to have better performance and performance when handling posture estimation tasks.

[0309] Fig.12 is a schematic diagram of a mask expansion effect according to an embodiment of the present invention. Fig.12 As shown in the figure, the model mask can also be dilated at the edge. Through the edge dilation process, the redrawing area generated by the model can be expanded, thereby effectively generating more natural shadows around the model. This process not only enhances the integration between the AI ​​model and the scene graph, but also improves the visual effect of the overall picture, making the interaction between the model and the surrounding environment more harmonious and realistic. Fig.13 is a schematic diagram of a mask expansion effect of a virtual model image according to an embodiment of the present invention. Fig.13 As shown, through the above-mentioned edge expansion processing, a more natural shadow is generated when the AI ​​model leans against the edge of the sofa, which makes the integration between the AI ​​model and the sofa background higher and improves the realism of the image.

[0310] In this embodiment, a posture control network is proposed. Fig.14 is a schematic diagram of a posture control network according to an embodiment of the present invention, such as Fig.14 As shown in the figure, for an input posture estimation skeleton image, the posture control network is used to extract posture feature information in different dimensions, for example, 512*512, 256*256, and 128*128. And the posture features can be injected into the diffusion model for subsequent image generation. The right half of the figure is the trained posture control network, which is used to process the posture skeleton image. A method called ControlNet posture control is used to copy a diffusion model downsampling network, input the posture skeleton image corresponding to the model, keep the diffusion generation model frozen during the training process, and inject the output of each layer of the posture control network into the decoding part of the diffusion generation model in a cross-attention manner to provide posture control information for the generation model. Among them, z t It can be used to represent the information at time t after the latent variable information extracted by the encoder of VAE is denoised. Prompt can be text control information input by the user, such as the hair color, clothing, etc. of the AI ​​model input by the user. This is only an example and is not limited to this.

[0311] In this embodiment, a redrawing model based on reinforcement learning training can be set. The function of the redrawing model is to generate a specified posture model controlled by a posture control network in the mask part of the scene graph. The network structure of this module is built with reference to the stable diffusion model Stable Diffusion. However, the original stable diffusion model input has only one 4-channel latent variable feature. To keep the original scene graph unchanged and generate an AI model in the mask part, the input layer of the original network structure is expanded to 9 channels to adapt to the input of latent variables corresponding to the three images of the model mask, scene graph, and masked scene graph.

[0312] However, even if a posture control network is designed to extract posture skeleton information and provide posture control for the redrawing model, due to the uncertainty of the diffusion generation model, limb variation will still occur in the actual generation of AI models. There is currently no unified quantitative standard for limb variation in academia, so preference information based on human feedback has become an important indicator for evaluating limb naturalness. Reinforcement learning can be used to fine-tune the redrawing model based on the preference information of human feedback, so that the model output can be more biased towards human preferences. A cutting-edge solution based on human feedback learning is the direct preference learning DPO algorithm, which maximizes the probability of human preferred answers and minimizes the probability of non-preferred answers by giving preference information of different results generated by the same input.

[0313] Fig.15 is a schematic diagram of a virtual model image before and after color space loss training according to an embodiment of the present invention, such as Fig.15 As shown, the DPO algorithm also has its limitations. The DPO algorithm needs implicit learning preference information for the limb variation indicator. In the early stage of training, it will mistakenly learn "color preference", which will cause the model image generated by the model after DPO algorithm training to be distorted in terms of light, shadow, color, etc. For example, the model on the left side of the picture is too dark and does not match the background, making the image less realistic. At the same time, because the DPO algorithm will give a greater penalty to the generation that does not meet human preferences, it weakens the gradient update of the response that meets human preferences, resulting in the failure to maximize the use of these generated answers that meet human preferences. After training with the LAB color space loss, the color of the model and the background is more compatible, improving the authenticity of the image.

[0314] Optionally, Fig.16 is a schematic diagram of a training framework of a redrawing model according to an embodiment of the present invention, such as Fig.16As shown, the training framework may include randomly selected training methods 1601, alternating training data sets 1602, redrawing models 1603, preference pairs 1604, redrawing models 1605, noise predictions 1606, decoded images 1607, attitude control networks 1608, initial redrawing models 1609, noise predictions 1610, and decoded images 1611. In the embodiment of the present application, an alternating training method of reinforcement learning and supervised fine-tuning is proposed. Before each training step, a training method 1601 is randomly selected with a certain probability. Among them, supervised fine-tuning training uses an alternating training data set 1602 to train the redrawing model 1603, and calculates the supervised fine-tuning loss. The composition sources of the alternating training data set here have two parts, one is an out-of-distribution data set, which is used to inject out-of-distribution knowledge into the model to prevent the model generation ability from degrading, and the other part is derived from the good pictures in the preference pair data set. The role is to supervise and fine-tune these good pictures, which can make up for the problem of few good picture gradient flows and low utilization in reinforcement learning. For the reinforcement learning training part, the training data is the pre-collected preference pairs 1604. A preference pair contains the required input generated by the preference pair and two pictures scored by humans: a good picture and a bad picture. During the training process, two redrawing models 1605 and 1603 are initialized in the same way, one of which freezes the model weights and does not update the weights. The DPO direct preference loss is calculated using the noise prediction of the two redrawing models, and the predicted noise is used to directly predict the decoded images 1607 and 1611, and the LAB space loss is calculated.

[0315] Optionally, the DPO reinforcement learning loss can be determined by the following formula:

[0316]

[0317] Among them, L1 can be used to represent the DPO reinforcement learning loss; ∈ can be used to represent the noise predicted by the second redrawing model, that is, the first noise prediction result; ω can be used to represent the good images in the preference data set, that is, the first preference data set; l can be used to represent the bad images in the preference data set; ref can be used to represent the output result of the frozen redrawing model, that is, the first decoded image, so ε ref It can be used to represent the second noise prediction result; σ can be used to represent the sigmoid function; β can be used to represent the weight used to balance the direct preference loss and the regularization loss. By adjusting the value of β, the importance of the above two losses in the training process can be controlled; T can be used to represent the noise adding time step, which can be preset to 1000. This is only an example and is not limited to this; λ t Can be used to represent the signal-to-noise ratio.

[0318] Alternatively, the LAB color space loss may be determined by the following formula:

[0319]

[0320] Among them, L2 can be used to represent the LAB color space loss; L, a, and b can be used to represent the LAB color space pixels in the decoded image respectively.

[0321] Fig.17 is a schematic diagram of a reasoning framework of a redrawing model according to an embodiment of the present invention, such as Fig.17 As shown, reinforcement learning training is an offline strategy. During the reasoning process, the redrawing model trained by reinforcement learning can be used for reasoning. Compared with the training framework, the reasoning framework is more concise. The reasoning framework may include a model mask 1702, a scene graph 1704, a masked scene graph 1706, a generated latent variable 1708, a redrawing model 1710 based on reinforcement learning training, a posture control network 1712, and a decoded image 1714. The model mask 1702 is combined with the scene graph 1704 to obtain a masked scene graph 1706. Then, these three are used to generate latent variables 1708 using a VAE variational encoder. The above process is to reduce the image dimension to the latent variable space and reduce the time and video memory required for subsequent image generation. At the same time, the posture control information output by the posture control network 1712 is input into the redrawing model 1710 trained based on reinforcement learning. Finally, the latent space variables output by the redrawing model 1710 trained based on reinforcement learning are decoded into an RGB image (decoded image 1714) through the decoder of the VAE model, thereby realizing the function of generating a specified posture model in the mask part of the scene graph.

[0322] Fig.18 is a schematic diagram of a model with a specified posture according to an embodiment of the present invention, such as Fig.18 As shown, through Fig.17 The method in can generate a home scene virtual model image of a female model leaning on a sofa.

[0323] Optionally, Fig.19 FIG. 1 is a schematic diagram of stacking a model in a specified posture back to the original scene according to an embodiment of the present invention. Fig.19 As shown in the figure, the mask and scene graph can be cropped so that the model part can occupy a larger pixel ratio during the generation process, thereby generating a high-definition model image. Therefore, the generated image output by the redrawing model is a cropped scene model image, which needs to be superimposed back to the original size scene graph. According to the cropping position parameters, the generated image is superimposed back to the original scene graph and finally output.

[0324] In the embodiment of the present application, a redrawing model is set up to integrate human preferences into the AI ​​model generation process. The generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, thereby improving the generation effect and making the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, and the visual effect and attractiveness of image generation are improved. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, making image generation more in line with user expectations and needs. Thereby, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0325] Fig. 20 is a schematic diagram of an image generated by a computer device according to an embodiment of the present invention, such as Fig. 20 As shown, a request for generating a target scene image with a model initiated by a user on a client device, that is, an image generation instruction, can be obtained by calling a first interface, and the computer device obtains the initial scene image received on the client device, the corresponding position information, and the auxiliary object image to be generated from the network; based on the above information received on the client device, the model is called and guided to be redrawn to generate a target scene image that meets the above requirements.

[0326] By setting up the redrawing model, human preferences can be integrated into the AI ​​model generation process. The generation process can be automatically learned and optimized through the reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and making the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and conformity of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, and the visual effect and attractiveness of image generation are improved. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, making image generation more in line with user expectations and needs. In this way, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0327] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 2 An image generating device according to the image generating method shown.

[0328] Fig.21 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Fig.21 As shown, the image generating device 2100 may include: a first determining unit 2102 , a second determining unit 2104 , a third determining unit 2106 and a first analyzing unit 2108 .

[0329] The first determining unit 2102 is configured to determine an initial scene image to be processed, wherein the image content of the initial scene image includes at least one display object.

[0330] The second determining unit 2104 is used to determine an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the display object.

[0331] The third determination unit 2106 is used to determine the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position.

[0332] The first analyzing unit 2108 is used to analyze the posture image, the object mask image and the initial scene image by using the redrawing model to obtain the target scene image, wherein the redrawing model is obtained by performing reinforcement learning on the diffusion model.

[0333] Here, the first determination unit 2102, the second determination unit 2104, the third determination unit 2106 and the first analysis unit 2108 correspond to steps S202 to S208 in the embodiment, and the four units are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2504) and processed by one or more processors (e.g., processors 2502a, 2502b..., 2502n), and the above units can also be run as part of the device in the computer terminal A provided in the following embodiment.

[0334] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 3 The model determination method shown is a model determination device.

[0335] Fig. 22 is a schematic diagram of a model determination device according to an embodiment of the present application, such as Fig. 22 As shown, the model determination device 2200 may include: an acquisition unit 2202 and a learning unit 2204.

[0336] The acquisition unit 2202 is used to acquire a preference data set, wherein the preference data set includes posture image samples, object mask image samples, and target scene image sample pairs generated based on the auxiliary display object samples, the posture image samples are used to represent the posture of the auxiliary display object samples, the object mask image samples are used to represent the image area where the auxiliary display object samples are located in the initial scene image samples, and the target scene image sample pairs include target scene image samples whose evaluation results are higher than the evaluation result threshold, and target scene image samples whose evaluation results are lower than the evaluation result threshold.

[0337] The learning unit 2204 is used to use the preference data set to perform reinforcement learning on the diffusion model to obtain a redrawing model, wherein the redrawing model is used to analyze the posture image of at least one auxiliary display object, the object mask image of the auxiliary display object and the initial scene image to be processed to obtain a target scene image, the auxiliary display object is used to assist in displaying at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

[0338] It should be noted that the acquisition unit 2202 and the learning unit 2204 correspond to steps S302 to S304 in the above embodiment, and the two units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above unit can be a hardware component or software component stored in a memory (e.g., memory 2504) and processed by one or more processors (e.g., processors 2502a, 2502b..., 2502n), and the above unit can also be run as part of the device in the computer terminal A provided in the following embodiment.

[0339] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 4 An image generating device according to the image generating method shown.

[0340] Fig.23 is a schematic diagram of another image generating device according to an embodiment of the present application, such as Fig.23 As shown, the image generating device 2300 may include: a recognition unit 2302 , a fourth determination unit 2304 , a fifth determination unit 2306 , a second analysis unit 2308 and a return unit 2310 .

[0341] The identification unit 2302 is used to identify an initial scene image to be processed from the e-commerce platform, wherein the image content of the initial scene image includes at least one displayed product.

[0342] The fourth determining unit 2304 is used to determine the auxiliary object image to be drawn at the target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the display product.

[0343] The fifth determination unit 2306 is used to determine the posture image and object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position.

[0344] The second analysis unit 2308 is used to analyze the posture image, the object mask image and the initial scene image by using the redrawing model to obtain the target scene image, wherein the redrawing model is obtained by performing reinforcement learning on the diffusion model.

[0345] The returning unit 2310 is used to return the target scene image to the e-commerce platform.

[0346] Here, the identification unit 2302, the fourth determination unit 2304, the fifth determination unit 2306, the second analysis unit 2308 and the return unit 2310 correspond to steps S402 to S410 in the above embodiment, and the five units are the same as the examples and application scenarios implemented by the corresponding steps, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2504) and processed by one or more processors (e.g., processors 2502a, 2502b..., 2502n), and the above units can also be run as part of the device in the computer terminal A provided in the following embodiment.

[0347] According to an embodiment of the present application, there is also provided a method for implementing the above Figure 5 An image generating device according to the image generating method shown.

[0348] Fig.24 is a schematic diagram of another image generating device according to an embodiment of the present application, such as Fig.24 As shown, the image generating device 2400 may include: a first display unit 2402 and a second display unit 2404 .

[0349] The first display unit 2402 is used to respond to an input instruction acting on the operation interface and display the initial scene image to be processed on the operation interface, wherein the image content of the initial scene image includes at least one display object.

[0350] The second display unit 2404 is used to respond to the image generation instruction acting on the operation interface, and display the target scene image corresponding to the initial scene image on the operation interface, wherein the target scene image is obtained by analyzing the posture image of at least one auxiliary display object, the object mask image of the auxiliary display object and the initial scene image using the redrawing model, the redrawing model is obtained by reinforcement learning of the diffusion model, the auxiliary display object is used to assist in the display of at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

[0351] Here, the first display unit 2402 and the second display unit 2404 correspond to steps S502 to S504 in the above embodiment, and the two units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above embodiment. It should be noted that the above units can be hardware components or software components stored in a memory (e.g., memory 2504) and processed by one or more processors (e.g., processors 2502a, 2502b..., 2502n), and the above units can also be run as part of the device in the computer terminal A provided in the following embodiment.

[0352] In the image generation device, if it is necessary to better display the display object in the image, the interaction between the display object and the auxiliary display object can be reflected by adding an auxiliary display object to assist in displaying the display object. For example, the initial scene image including the display object can be obtained by using the display object. According to the user's needs, the target position where the auxiliary object image needs to be drawn can be determined from the initial scene image, and the auxiliary object image that meets the user's needs for the posture, clothing, etc. of the auxiliary display object can be screened from a large number of auxiliary object images. The posture of the auxiliary display object can be determined based on the auxiliary object image, the target position and the initial scene image, and the posture image and the object mask image of the auxiliary display object are obtained. The hidden variables corresponding to the posture image, the object mask image and the initial scene image are input into the redrawing model to integrate the target scene image. In the target scene image, the interaction between the auxiliary display object and the display object that meets the user's needs can be displayed, thereby improving the display effect of the display object.

[0353] In this embodiment, a redrawing model is set up to integrate human preferences into the AI ​​model generation process. The generation process is automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. The preference information fed back by humans is used as an important indicator for evaluating the naturalness of limbs, and the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and in line with human aesthetic requirements, and the visual effect and attractiveness of image generation are improved. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, so that image generation is more in line with user expectations and needs. In this way, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0354] The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0355] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0356] In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image generation method: determine the initial scene image to be processed; determine the auxiliary object image to be drawn at the target position in the initial scene image; based on the auxiliary object image, the target position and the initial scene image, determine the posture image and the object mask image of the auxiliary display object; use the redrawing model to analyze the posture image, the object mask image and the initial scene image to obtain the target scene image.

[0357] Optionally, Fig.25 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.25 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 2502 , a memory 2504 and a transmission device 2506 .

[0358] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image generation method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image generation method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0359] Using the embodiment of the present application, a method for generating an image is provided. In the embodiment of the present application, by setting a redrawing model, human preferences can be integrated into the AI ​​model generation process, and the generation process can be automatically learned and optimized through a reinforcement learning mechanism. The above mechanism can adjust the posture and performance of the AI ​​model according to the user's preferences and human aesthetic standards, and can reduce the limb variation phenomenon of the AI ​​model, thereby improving the generation effect of the AI ​​model, and can make the generated AI model more in line with user expectations. Using the preference information fed back by humans as an important indicator for evaluating the naturalness of limbs, the changes in human posture that are difficult to objectively quantify are converted into intuitive user preference indicators. The above evaluation method is closer to human subjective feelings, can more accurately evaluate the naturalness of limbs, and improve the quality and compliance of AI model generation. Thereby, the limb abnormalities and variations that frequently occur in related technologies can be well avoided, thereby optimizing the generation effect of the AI ​​model, and the generated AI model is more natural and meets human aesthetic requirements, and improves the visual effect and attractiveness of image generation. The above method not only improves the quality and effect of AI model generation, but also improves user experience and satisfaction, so that image generation is more in line with user expectations and needs. Thereby, the technical effect of improving the accuracy of image generation is achieved, and the technical problem of low image generation accuracy is solved.

[0360] It can be understood by those skilled in the art that Fig.25 The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, referred to as MID), a PAD, or other terminal devices. Fig.25 It does not limit the structure of the above-mentioned computer terminal A. For example, the computer terminal A may also include Fig.25 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig.25 Different configurations shown.

[0361] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0362] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the image generation method provided in the first embodiment.

[0363] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

[0364] Optionally, in this embodiment, a computer-readable storage medium is configured to store program codes for executing the above steps of the embodiment of the present application.

[0365] An embodiment of the present application may provide an electronic device, which may include a memory and a processor.

[0366] Fig.26 It is a block diagram of an electronic device according to a method for generating an image in an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0367] like Fig.26As shown, the device 2600 includes a computing unit 2601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 2602 or a computer program loaded from a storage unit 2608 into a random access memory (RAM) 2603. In RAM 2603, various programs and data required for the operation of the device 2600 can also be stored. The computing unit 2601, ROM 2602, and RAM 2603 are connected to each other via a bus 2604. An input / output (I / O) interface 2605 is also connected to the bus 2604.

[0368] A number of components in the device 2600 are connected to the I / O interface 2605, including: an input unit 2606, such as a keyboard, a mouse, etc.; an output unit 2604, such as various types of displays, speakers, etc.; a storage unit 2608, such as a disk, an optical disk, etc.; and a communication unit 2609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 2609 allows the device 2600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0369] The computing unit 2601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 2601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 2601 performs the various methods and processes described above, such as the image generation method. For example, in some embodiments, the image generation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 2608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 2600 via the ROM 2602 and / or the communication unit 2609. When the computer program is loaded into the RAM 2603 and executed by the computing unit 2601, one or more steps of the image generation method described above may be performed. Alternatively, in other embodiments, the computing unit 2601 may be configured to perform the image generation method in any other appropriate manner (e.g., by means of firmware).

[0370] The embodiment of the present application further provides a computer program product. Optionally, in the present embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the image generation method of the embodiment of the present application is implemented.

[0371] According to an embodiment of the present application, a method for generating an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0372] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Fig. 27 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image generation method according to an embodiment of the present application, such as Fig. 27 As shown, the computer terminal 270 (or mobile device) may include one or more (shown in the figure as 2702a, 2702b, ..., 2702n) processors 2702 (the processor 2702 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, abbreviated as MCU) or a programmable logic device (Field Programmable Gate Array, abbreviated as FPGA)), a memory 2704 for storing data, and a transmission device 2706 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (Universal Serial Bus, abbreviated as USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Fig. 27 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig. 27 More or fewer components as shown, or with Fig. 27 Different configurations shown.

[0373] Fig. 27 The hardware structure block diagram shown can be used not only as an exemplary block diagram of the above-mentioned computer terminal 270 (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Fig.28 The block diagram shows the use of the above Fig. 27 The computer terminal 270 (or mobile device) is shown as an embodiment of a computing node in the computing environment 2801.

[0374] Fig.28 is a structural block diagram of a computing environment of an image generation method according to an embodiment of the present application, such as Fig.28As shown, computing environment 2801 includes multiple computing nodes (such as servers) running on a distributed network (shown as 2810-1, 2810-2, ...) in the figure. The computing nodes all contain local processing and memory resources, and end users 2802 can remotely run applications or store data in computing environment 2801. Applications can be provided as multiple services 2820-1, 2820-2, 2820-3 and 2820-4 in computing environment 1601, representing services "F", "G", "I" and "H" respectively.

[0375] The end user 2802 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 2802's provision and / or request can be provided to the entry gateway 2830. The entry gateway 2830 may include a corresponding agent to handle the provision and / or request for services (one or more services provided in the computing environment 2801).

[0376] Services are provided or deployed based on various virtualization technologies supported by computing environment 2801. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system so that multiple workloads can run on a single operating system instance.

[0377] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Fig.28 As shown, service 2820-2 can be equipped with one or more Pods 2840-1, 2840-2, ..., 2840-N (collectively referred to as Pods). Pods can include a proxy 2845 and one or more containers 2842-1, 2842-2, ..., 2842-M (collectively referred to as containers). One or more containers in the Pod process requests related to one or more corresponding functions of the service, and the proxy 2845 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with Pods similar to Pods.

[0378] During operation, executing a user request from the end user 2802 may require calling one or more services in the computing environment 2801, and executing one or more functions of a service may require calling one or more functions of another service. Fig.28As shown, service "F" 2820-1 receives a user request from terminal user 2802 from ingress gateway 2830, service "F" 2820-1 may call service "G" 2820-2, and service "G" 2820-2 may request service "I" 2820-3 to perform one or more functions.

[0379] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0380] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that are executable and / or interpreted on a programmable system including at least one programmable processor that may be a special purpose or general purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0381] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0382] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0383] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0384] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory for short), an optical fiber, a portable compact disk read-only memory (CD-ROM for short), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0385] To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD), a monitor for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a path ball), through which the user may provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0386] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

[0387] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0388] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0389] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0390] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0391] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0392] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0393] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories, random access memories, mobile hard disks, magnetic disks or optical disks.

[0394] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating an image, characterized in that: include: Determining an initial scene image to be processed, wherein the image content of the initial scene image includes at least one display object; Determine an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the display object; Based on the auxiliary object image, the target position and the initial scene image, determining a posture image and an object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent an image region where the auxiliary display object is located in the initial scene image, and the image region includes the target position; The posture image, the object mask image and the initial scene image are analyzed by using a redrawing model to obtain a target scene image, wherein the redrawing model is obtained by performing reinforcement learning on a diffusion model.

2. The method according to claim 1, characterized in that The redrawing model is obtained by using a preference data set to perform reinforcement learning on the diffusion model, the preference data set includes posture image samples, object mask image samples, and target scene image sample pairs generated based on auxiliary display object samples, the posture image samples are used to represent the posture of the auxiliary display object samples, the object mask image samples are used to represent the image area where the auxiliary display object samples are located in the initial scene image samples, and the target scene image sample pairs include target scene image samples whose evaluation results are higher than an evaluation result threshold, and target scene image samples whose evaluation results are lower than the evaluation result threshold.

3. The method according to claim 2, characterized in that Analyzing the posture image, the object mask image, and the initial scene image using a redrawing model to obtain a target scene image includes: The pose image, the object mask image and the initial scene image are analyzed using the model parameters of the redrawing model to obtain the target scene image, wherein the model parameters are obtained by alternately performing reinforcement learning and supervised training on the diffusion model using the preference data set.

4. The method according to claim 3, characterized in that Analyzing the posture image, the object mask image, and the initial scene image using the model parameters of the redrawing model to obtain the target scene image includes: extracting posture features from the posture image; The posture feature is used to control the redrawing model to analyze the posture image, the object mask image and the initial scene image under the model parameters to obtain the target scene image, wherein the auxiliary display object matches the posture feature in the target scene image.

5. The method according to claim 4, characterized in that Using the posture feature to control the redrawing model under the model parameters, analyzing the posture image, the object mask image and the initial scene image to obtain the target scene image, including: Using an attention mechanism, converting the posture features into control information of the redrawing model; The control information is used to control the redrawing model to analyze the posture image, the object mask image and the initial scene image under the model parameters to obtain the target scene image.

6. The method according to claim 5, characterized in that Using the control information, controlling the redrawing model to analyze the posture image, the object mask image, and the initial scene image under the model parameters to obtain the target scene image, including: Combining the object mask image and the initial scene image to obtain a scene mask image, wherein the scene mask image includes a mask corresponding to the object mask image, and a size of the scene mask image is smaller than a size of the initial scene image; Using an encoder, the scene mask image, the object mask image and the initial scene image are combined to generate initial latent variables in a latent variable space; Using the control information, controlling the redrawing model to analyze the initial latent variables under the model parameters to obtain an intermediate scene image, wherein the size of the intermediate scene image is smaller than the size of the initial scene image; The intermediate scene image is superimposed on the initial scene image to obtain the target scene image.

7. The method according to claim 6, characterized in that Using the control information, controlling the redrawing model to analyze the initial latent variables under the model parameters to obtain an intermediate scene image, including: Using the control information, controlling the redrawing model to analyze the initial latent variables under the model parameters to obtain target latent variables in the latent variable space; The target latent variable is decoded by using a decoder to obtain the intermediate scene image.

8. The method according to claim 6, characterized in that The method further comprises: Performing a cropping process on the object mask image, wherein a size of the cropped object mask image is smaller than a size of the initial scene image; Cropping the initial scene image and the posture image respectively according to the cropping parameters corresponding to the cropped object mask image; Combining the object mask image with the initial scene image to obtain a scene mask image includes: combining the cropped object mask image with the cropped initial scene image to obtain the scene mask image.

9. The method according to claim 8, characterized in that The method further comprises: Performing dilation processing on the edge of the object mask image; The step of performing a cropping process on the object mask image includes: performing a cropping process on the expanded object mask image.

10. The method according to claim 8, characterized in that The method of superimposing the intermediate scene image onto the initial scene image to obtain the target scene image comprises: According to the cropping parameters, the intermediate scene image is superimposed on the initial scene image to obtain the target scene image.

11. The method according to claim 4, characterized in that Extracting posture features from the posture image includes: The posture features at different sizes are extracted from the posture image using a posture control model, wherein the posture control model is obtained by training a control network structure using the posture of the auxiliary display object sample.

12. The method according to claim 1, characterized in that Determining a posture image and an object mask image of the auxiliary display object based on the auxiliary object image, the target position and the initial scene image includes: Using a posture estimation model, analyzing the auxiliary object image to obtain an initial posture image corresponding to the posture image, and using a transparency channel of the auxiliary object image to generate an initial object mask image corresponding to the object mask image, wherein the posture estimation model is obtained by training a transformation network structure using a self-attention mechanism; The initial posture image and the initial object mask image are moved to the target position of the initial scene image to obtain the posture image and the object mask image.

13. The method according to claim 12, characterized in that The auxiliary object image is analyzed using a posture estimation model to obtain an initial posture image, including: Using the posture estimation model to perform block processing on the auxiliary object image to obtain a plurality of sub-auxiliary object images, and determining a sub-pose image of the sub-auxiliary object image at the target position in the initial scene image, wherein the plurality of sub-auxiliary object images are associated with each other; The posture estimation model is used to fuse image information in a plurality of the sub-posture images corresponding to a plurality of the sub-auxiliary object images according to the association relationship to obtain the initial posture image.

14. The method according to claim 12, characterized in that Moving the initial posture image and the initial object mask image to the target position of the initial scene image to obtain the posture image and the object mask image includes: scaling the initial pose image and the initial object mask image respectively according to the size of the initial scene image; The scaled initial posture image is moved to the target position of the initial scene image to obtain the posture image, and the scaled initial object mask image is moved to the target position of the initial scene image to obtain the object mask image.

15. The method according to claim 12, characterized in that Moving the initial posture image and the initial object mask image to the target position of the initial scene image to obtain the posture image and the object mask image includes: scaling the initial pose image and the initial object mask image respectively according to the size of the initial scene image; The scaled initial posture image is moved to the target position of the initial scene image to obtain the posture image, and the scaled initial object mask image is moved to the target position of the initial scene image to obtain the object mask image.

16. A method for determining a model, characterized in that: include: Acquire a preference data set, wherein the preference data set includes a pose image sample, an object mask image sample, and a target scene image sample pair generated based on the auxiliary display object sample, the pose image sample is used to represent the pose of the auxiliary display object sample, the object mask image sample is used to represent the image region where the auxiliary display object sample is located in the initial scene image sample, and the target scene image sample pair includes a target scene image sample whose evaluation result is higher than an evaluation result threshold, and a target scene image sample whose evaluation result is lower than the evaluation result threshold; The preference data set is used to perform reinforcement learning on the diffusion model to obtain a redrawing model, wherein the redrawing model is used to analyze a posture image of at least one auxiliary display object, an object mask image of the auxiliary display object and an initial scene image to be processed to obtain a target scene image, the auxiliary display object is used to assist in displaying at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

17. The method according to claim 16, characterized in that Using the preference dataset, the diffusion model is reinforced to obtain a redrawing model, including: The preference data set is used to perform reinforcement learning and supervised training alternately on the diffusion model to obtain the redrawing model.

18. The method according to claim 17, characterized in that Using the preference data set, the diffusion model is subjected to alternate reinforcement learning and supervised training to obtain the redrawing model, including: Among the reinforcement learning strategy and the supervised training strategy, a randomly selected reinforcement learning strategy or a supervised training strategy is determined as a target training strategy, wherein the reinforcement learning strategy is used to represent a rule for performing reinforcement learning on the diffusion model, and the supervised training strategy is used to represent a rule for performing supervised training on the diffusion model; According to the target training strategy, the diffusion model is trained using the preference data set to obtain the redrawing model.

19. The method according to claim 18, characterized in that According to the target training strategy, the diffusion model is trained using the preference data set to obtain the redrawing model, including: In a case where the target training strategy is the supervised training strategy, determining a first preference data set from the preference data sets, and acquiring external knowledge data, wherein the first preference data set includes target scene image samples whose evaluation results are higher than the evaluation result threshold, and the external knowledge data is used to represent knowledge allowed to be injected into the diffusion model; According to the supervised training strategy, the first preference data set and the external knowledge data are used to train the first drawing model to obtain a second drawing model, wherein the first drawing model is trained by the reinforcement learning strategy, and the redrawing model includes the second drawing model.

20. The method according to claim 18, characterized in that According to the target training strategy, the diffusion model is trained using the preference data set to obtain the redrawing model, including: In a case where the target training strategy is the reinforcement learning strategy, determining a first preference data set and a second preference data set from the preference data set, wherein the first preference data set includes target scene image samples whose evaluation results are higher than the evaluation result threshold, and the second preference data set includes target scene image samples whose evaluation results are lower than the evaluation result threshold; Using a second rendering model with frozen model weights, performing noise prediction on the first preference data set and the second preference data set to obtain a first noise prediction result, and decoding a first decoded image from the first noise prediction result, wherein the second rendering model is trained by the supervised training strategy; Using the second rendering model whose model weights are not frozen, performing noise prediction on the first preference data set and the second preference data set to obtain a second noise prediction result, and decoding a second decoded image from the second noise prediction result; A first rendering model is trained based on the first noise prediction result, the first decoded image, the second noise prediction result and the second decoded image, wherein the redrawing model includes the first rendering model.

21. The method according to claim 20, characterized in that Based on the first noise prediction result, the first decoded image, the second noise prediction result and the second decoded image, training a first rendering model includes: Determining a reinforcement learning loss based on the first noise prediction result and the second noise prediction result; determining a color space loss based on the first decoded image and the second decoded image; The first drawing model is trained by using the reinforcement learning loss and the color space loss.

22. The method according to any one of claims 16 to 21, characterized in that The method further comprises: Expanding the initial number channel of the diffusion model to obtain a target number channel; Using the preference data set to perform reinforcement learning on the diffusion model to obtain the redrawing model includes: using the preference data set to perform reinforcement learning on the diffusion model of the target number of channels to obtain the redrawing model.

23. A method for generating an image, characterized in that: include: Identifying an initial scene image to be processed from an e-commerce platform, wherein the image content of the initial scene image includes at least one displayed product; Determine an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the display product; Based on the auxiliary object image, the target position and the initial scene image, determining a posture image and an object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent an image region where the auxiliary display object is located in the initial scene image, and the image region includes the target position; Using a redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain a target scene image, wherein the redrawing model is obtained by performing reinforcement learning on a diffusion model; The target scene image is returned to the e-commerce platform.

24. A method for generating an image, characterized in that: include: In response to an input instruction acting on an operation interface, an initial scene image to be processed is displayed on the operation interface, wherein the image content of the initial scene image includes at least one display object; In response to an image generation instruction acting on the operation interface, a target scene image corresponding to the initial scene image is displayed on the operation interface, wherein the target scene image is obtained by analyzing a posture image of at least one auxiliary display object, an object mask image of the auxiliary display object and the initial scene image using a redrawing model, the redrawing model is obtained by reinforcement learning of a diffusion model, the auxiliary display object is used to assist in displaying at least one display object in the initial scene image, the posture image is used to represent the posture of the auxiliary display object, the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, the posture image and the object mask image are determined based on the auxiliary object image, the target position and the initial scene image, the auxiliary object image is an image to be drawn at the target position in the initial scene image, and the image content of the auxiliary object image includes the auxiliary display object, and the target position is included in the image area.

25. A system for generating an image, characterized in that: include: A client, used for uploading an initial scene image to be processed, wherein the image content of the initial scene image includes at least one display object; The server is used to determine an auxiliary object image to be drawn at a target position in the initial scene image, wherein the image content of the auxiliary object image includes at least one auxiliary display object, and the auxiliary display object is used to assist in displaying the display object; based on the auxiliary object image, the target position and the initial scene image, determine a posture image and an object mask image of the auxiliary display object, wherein the posture image is used to represent the posture of the auxiliary display object, and the object mask image is used to represent the image area where the auxiliary display object is located in the initial scene image, and the image area includes the target position; using a redrawing model, analyzing the posture image, the object mask image and the initial scene image to obtain a target scene image, wherein the redrawing model is obtained by performing reinforcement learning on a diffusion model; Wherein, the client is used to display the target scene image.

26. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 24 when running.

27. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 24.

28. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 24.

Images