Image generation method, device and equipment and computer storage medium

Abstract

The embodiment of the invention provides an image generation method and device, equipment and a computer storage medium. The image generation method comprises the following steps: acquiring a scene image and description information, wherein the description information is used for indicating to generate a preset object in the scene image; processing the description information and the scene image by using a pre-trained global positioning model, and determining a local scene image for generating the preset object, the resolution of the local scene image being smaller than the resolution of the scene image; processing the local scene image and the description information by using a pre-trained local drawing model to generate an object foreground image corresponding to the preset object; and carrying out image fusion on the local scene image and the object foreground image to generate a target scene image which comprises a preset object. In the embodiment of the invention, the target scene image comprising the preset object can be stably generated without user participation, and the authenticity and rationality of the target scene image are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image generation method and device, equipment and a computer storage medium. BACKGROUND

[0002] In the application field of artificial intelligence generated content, human synthesis technology is often implemented based on a local redrawing model of a diffusion model. When generating an image using the local redrawing model, a user needs to manually define an accurate generation mask region (i.e., a mask), and then the local redrawing model can perform a content generation operation in the mask region.

[0003] In the above image generation process, a complex composition planning task (including determining the appropriate position, size, and perspective relationship of a person, etc.) is assigned to the user, which makes the image generation task challenging for non-professional users and easily leads to an imbalance between the human body and the scene in the generated image, improper position, and thus destroys the authenticity and rationality of the generated image. SUMMARY

[0004] The embodiments of the present application provide an image generation method, device, equipment and computer storage medium, which can stably generate a target scene image including a preset object without user participation, and effectively ensure the authenticity and rationality of the generated target scene image.

[0005] In a first aspect, the embodiments of the present application provide an image generation method, comprising: obtaining a scene image and description information, the description information being used to indicate generation of a preset object in the scene image; processing the description information and the scene image using a pre-trained global positioning model to determine a local scene image used to generate the preset object, the resolution of the local scene image being less than the resolution of the scene image; processing the local scene image and the description information using a pre-trained local drawing model to generate an object foreground image corresponding to the preset object; performing image fusion on the local scene image and the object foreground image to generate a target scene image, the target scene image including the preset object.

[0006] In a second aspect, the embodiments of the present application provide an image generation device, comprising: a first obtaining module configured to obtain a scene image and description information, the description information being used to indicate generation of a preset object in the scene image; The first processing module is configured to process the description information and the scene image by using a pre-trained global positioning model, to determine a local scene image used to generate the preset object, and the resolution of the local scene image is less than the resolution of the scene image. The first processing module is further configured to process the local scene image and the description information by using a pre-trained local drawing model, to generate an object foreground image corresponding to the preset object. The first generation module is configured to perform image fusion on the local scene image and the object foreground image, to generate a target scene image, and the target scene image includes the preset object.

[0007] In a third aspect, an electronic device is provided, including a memory and a processor, the memory is configured to store one or more computer instructions, and the one or more computer instructions are configured to implement the method in the first aspect when executed by the processor.

[0008] In a fourth aspect, a computer storage medium is provided, configured to store a computer program, and the computer program is configured to implement the method in the first aspect when executed by a computer.

[0009] In a fifth aspect, a computer program product is provided, including a computer readable storage medium storing computer instructions, and the computer instructions are configured to cause one or more processors to perform the steps in the method in the first aspect when executed by the one or more processors.

[0010] The image generation method, device, electronic device and computer storage medium provided by the embodiment can obtain a scene image and description information, process the description information and the scene image by using a pre-trained global positioning model, determine a local scene image used to generate a preset object, then process the local scene image and the description information by using a pre-trained local drawing model, generate an object foreground image corresponding to the preset object, perform image fusion on the local scene image and the object foreground image, effectively realize automatic and high-quality image generation based on the global positioning model and the local drawing model, and do not need human operation, thereby liberating users from tedious and inefficient manual operation, solving the problem that the generated image is unreasonable in composition, out of proportion, and not real in related technologies, and effectively ensuring the authenticity and rationality of the generated target scene image, and then application operation can be performed based on the generated target scene image, and the practicability of the method is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A scene illustration of an image generation method provided for an exemplary embodiment of this application; Figure 2 A schematic flowchart of an image generation method provided for an exemplary embodiment of this application; Figure 3 A schematic diagram of a process for determining a local scene image for generating the preset object, provided as an exemplary embodiment of this application; Figure 4 A schematic flowchart illustrating the process of generating an object foreground image corresponding to the preset object, provided for an exemplary embodiment of this application; Figure 5 A schematic flowchart illustrating yet another image generation method provided as an exemplary embodiment of this application; Figure 6 A schematic diagram of an automated scene-aware human image generation method based on global localization and local fine-mapping provided for exemplary embodiments of this application; Figure 7 A schematic diagram of the structure of an image generation apparatus provided for an exemplary embodiment of this application; Figure 8 A schematic diagram of the structure of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] It should be noted that, in the case of user information involved in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0014] The various models involved in this application (including but not limited to language models or large models) comply with relevant laws and standards. Furthermore, the technical solutions provided in the embodiments of this application can employ deep learning models with relatively large parameter scales. The large model is merely an example, and the embodiments of this application do not limit the number of model parameters supported by the deep learning model used, aiming to meet actual needs. The deep learning models involved in the embodiments of this application can be artificial intelligence-based language models (LM) or multimodal models (MM).

[0015] Additionally, it should be noted that when user interaction operations or triggering operations are involved in the embodiments of this application, these operations include, but are not limited to, various interaction methods such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations. Touch operations include, but are not limited to, click operations, double-click operations, long-press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include, but are not limited to, straight-line swipes and curved-line swipes.

[0016] To facilitate understanding of the image generation method, apparatus, device, and computer storage medium provided in the embodiments of this application, the relevant technologies are briefly described below: With the rapid development of artificial intelligence technology, the application scope of content generation technology is becoming increasingly widespread. In the field of AI-generated content, solutions for human body synthesis within scene images face numerous challenges, often resulting in final effects that fail to meet commercial application standards. Specifically, existing implementations of human body synthesis technology often suffer from the following two irreconcilable bottlenecks: (1) Relying on manual planning by users.

[0017] In related technology 1, human body synthesis technology can be implemented based on the local redraw model of the diffusion model. When using the local redraw model to generate images, users often need to manually define a precise generation mask area (i.e., mask), and then the local redraw model can perform content generation operations within the aforementioned mask area.

[0018] The inherent flaw of the above implementation scheme lies in its passive architecture: First, it entrusts the complex task of composition planning (including determining the appropriate position, size, and perspective of the figure, etc.) to the user, lacking the ability to perform advanced semantic understanding and layout planning of the scene. This makes the image generation task challenging for non-professional users and easily leads to disproportionate human figures and improper positions in the generated images, thus compromising the realism and rationality of the generated images. Second, local redrawing models often lack structured prior knowledge about human skeletons and joint range of motion, resulting in frequent anatomical errors and highly unpredictable results when generating human figures. Finally, local redrawing models usually have limited understanding of the context outside the masked area, which may lead to the generated human figures being inconsistent with the background in terms of lighting, style, and visual "texture look."

[0019] (2) Lack of anatomical constraints.

[0020] In related technology 2, "generating a structurally accurate human body within a defined region" is currently a technical challenge. The objective function of mainstream generative models typically focuses on pixel-level similarity, meaning they learn to generate a set of pixels that resemble the target's appearance, rather than constructing a structured human body that conforms to anatomical principles. Therefore, their generated results frequently contain structural errors such as redundant limbs and inverted joints.

[0021] In addition, a human body synthesis scheme is implemented through a separate two-stage inference operation: "predict the skeleton first, then render the image." This scheme is based on a cascaded independent pose prediction model and an independent image generation model. Specifically, the first step is to predict the skeleton pose, and the second step is to render the image based on the predicted skeleton pose.

[0022] However, the core technical contradiction of this design lies in its "fragmented" process architecture. Specifically, since pose prediction and appearance rendering are two independently trained modules with unidirectional information flow, they lack awareness of each other's internal states. Therefore, any tiny deviation generated by the upstream pose prediction model can be amplified in the downstream rendering process, forming an irreversible error accumulation effect, thus affecting the stability and final quality of the entire system. Furthermore, the architecture lacks an end-to-end optimization mechanism; the quality of the rendering results cannot form a closed-loop feedback loop to optimize the upstream pose prediction through gradient backpropagation, etc. Simultaneously, the two modules cannot form a collaborative optimization force, limiting the overall system performance to the weakest link. It is worth noting that such solutions typically still require the user to pre-specify a bounding box, failing to address the problem of intelligently determining position and size in large scenes.

[0023] To address the aforementioned technical problems, embodiments of this application provide an image generation method, apparatus, device, and computer storage medium, as detailed in the appendix. Figure 1 As shown, the execution entity of this image generation method can be an image generation device 200, which can be implemented as a local server, a cloud server, or an edge server. When the image generation device 200 is implemented as a cloud server, the image generation method can be executed in the cloud. Several computing nodes (cloud servers) can be deployed in the cloud, each with computing, storage, and other processing resources. In the cloud, multiple computing nodes can be organized to provide a certain service; of course, a single computing node can also provide one or more services. The cloud can provide this service by providing a service interface, which users can call to use the corresponding service. Service interfaces include Software Development Kits (SDKs) and Application Programming Interfaces (APIs).

[0024] The image generation device 200 is communicatively connected to the client 100, which is used by the user to trigger image generation operations. The client 100 can be any computing device with certain information interaction capabilities; specifically, it can be a mobile phone, a personal computer (PC), a tablet computer, a configuration application, etc. Furthermore, the basic structure of the client 100 may include at least one processor. The number of processors depends on the client's configuration and type. The client 100 may also include memory, which can be volatile, such as Random Access Memory (RAM), or non-volatile, such as Read-Only Memory (ROM), flash memory, etc., or both types. The memory typically stores the operating system (OS), one or more applications, and may also store program data. In addition to the processing unit and memory, the client 100 also includes some basic configurations, such as a network interface card (NIC) chip, an I / O bus, a display component, and some peripheral devices. Optionally, some peripheral devices may include, for example, a keyboard, a mouse, a stylus, a printer, etc. Other peripheral devices are well known in the art and will not be described in detail here.

[0025] Image generation device 200 refers to a device capable of performing image generation operations in a network virtual environment, typically referring to a device that utilizes a network for information planning and image generation operations. The image generation device 200 can be implemented as an image generation model for performing image generation operations. Physically, the image generation device 200 can be any device capable of providing computing services and performing corresponding subsequent operations, such as a processor, server, etc. The image generation device 200 mainly consists of a processor, hard disk, memory, system bus, etc., and its architecture is similar to that of a general-purpose computer.

[0026] In the above embodiment, the image generating device 200 and the client 100 are connected via a network, which can be a wireless or wired network connection. If the image generating device 200 and the client 100 are connected via a communication connection, the network standard of the mobile network can be any one of 2G (Global System for Mobile Communications GSM), 2.5G (General Packet Radio Service GPRS), 3G (Wideband Code Division Multiple Access (WCDMA), Time Division Synchronous Code Division Multiple Access (TD-SCDMA), 4G (Long Term Evolution LTE), 4G+ (Enhanced Long Term Evolution LTE+), Global Microwave Access Interoperability (WiMax), 5G, 6G, etc.

[0027] In this embodiment, the client 100 is used by a user to generate an image generation request that triggers an image generation operation. The image generation request may include a scene image and descriptive information. The descriptive information is used to generate a preset object in the scene image. The preset object can be any of the following: a person, an animal, a virtual object, etc. The image generation request can then be sent to the image generation device 200 so that the image generation device 200 can perform a corresponding image generation operation based on the image generation request.

[0028] Image generation device 200 is used to acquire an image generation request sent by client 100, and then acquire a scene image and descriptive information based on the image generation request. To stably generate a preset object in the scene image and ensure the quality and effect of the image generation operation, a pre-trained global localization model can be used to process the descriptive information and the scene image to determine a local scene image for generating the preset object. This local scene image is a region within the scene image suitable for generating the preset object, for example: in Figure 1In the scene image shown, if the preset object is a person, the area corresponding to the sofa is suitable for generating a person. Therefore, the area corresponding to the sofa can be determined as a local scene image. For a local scene image, its resolution is smaller than that of the scene image, which means that the obtained local scene image is a low-fidelity image. This can improve the efficiency of subsequent image generation operations based on the local scene image.

[0029] After determining the local scene image used to generate the preset object, a pre-trained local rendering model can be used to process the local scene image and the description information to generate an object foreground image corresponding to the preset object. Then, an image fusion operation is performed on the local scene image and the object foreground image to generate a target scene image, which includes the preset object. This completes the image generation operation.

[0030] In this embodiment, image generation based on a global positioning model and a local rendering model is effectively automated and of high quality, without the need for manual operation. This frees users from tedious and inefficient manual operations and solves the problems of unreasonable composition, disproportionate scale, and unrealistic images caused by reliance on user operation in related technologies. This effectively ensures the authenticity and rationality of the generated target scene image, which can then be used for application operations, such as product publishing, further improving the practicality of the method.

[0031] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0032] Figure 2 A schematic flowchart of an image generation method provided for an exemplary embodiment of this application; see attached document. Figure 2 As shown, this embodiment provides an image generation method. The execution subject of this method is an image generation device, which can be implemented as software or a combination of software and hardware. When the image generation device is implemented as hardware, it can specifically be various electronic devices capable of performing image generation operations. In some instances, the image generation device can be implemented as a server, cloud server, etc. When the image generation device is implemented as software, it can be installed in the electronic devices exemplified above. Specifically, the image generation method provided in this embodiment may include: Step S201: Obtain scene image and description information. The description information is used to indicate the generation of a preset object in the scene image.

[0033] Step S202: Use a pre-trained global localization model to process the description information and scene image to determine the local scene image used to generate the preset object. The resolution of the local scene image is smaller than that of the scene image.

[0034] Step S203: Use a pre-trained local rendering model to process the local scene image and description information to generate an object foreground image corresponding to the preset object.

[0035] Step S204: Perform image fusion on the local scene image and the object foreground image to generate a target scene image, which includes a preset object.

[0036] The specific implementation methods and principles of each of the above steps are explained in detail below: Step S201: Obtain scene image and description information. The description information is used to indicate the generation of a preset object in the scene image.

[0037] When there is a need to generate images, the image generation device can acquire scene images and descriptive information. The scene image can refer to an image depicting a spatially structured environment, which typically includes multiple objects within the environment, background information, lighting information, and layout elements. The descriptive information instructs the generation of a preset object from the scene image. In some instances, the descriptive information can be user-input natural language, which can be in text form, such as "Generate an image including a male model" or "Generate an image including a standing female model," etc. Alternatively, the descriptive information can be in speech form. When the obtained descriptive information is in speech form, it can be converted from speech to text to obtain text-based descriptive information, thus ensuring the accuracy of image generation based on the descriptive information.

[0038] Regarding scene images and descriptive information, this embodiment does not limit the specific method of obtaining scene images and descriptive information. In some instances, scene images and descriptive information can be obtained through interactive operations with a client. In this case, obtaining scene images and descriptive information may include: determining a client that is communicatively connected to the image generation device, which stores scene images and descriptive information; and then the scene images and descriptive information can be actively or passively obtained through the client, thus effectively ensuring the accuracy and reliability of obtaining scene images and descriptive information.

[0039] In other instances, scene images and descriptive information can be obtained not only through interactive operations with the client, but also through data upload operations via different upload interfaces in the image generation interface. In this case, obtaining scene images and descriptive information may include: displaying the file upload interface for image generation; identifying the image upload interface and text upload interface in the file upload interface; performing an image upload operation via the image upload interface to obtain the scene image; and performing a text upload operation via the text upload interface to obtain the descriptive information. This also ensures the accuracy and reliability of obtaining scene images and descriptive information.

[0040] Step S202: Use a pre-trained global localization model to process the description information and scene image to determine the local scene image used to generate the preset object. The resolution of the local scene image is smaller than that of the scene image.

[0041] After acquiring the scene image and description information, in order to stably achieve image generation, a global localization model can be pre-trained to determine the coordinates of the adaptation region and the size of the bounding box where a preset object can be placed or deployed. Before processing the description information and scene image using the pre-trained global localization model, the method in this embodiment may include: acquiring a global localization training dataset, which includes multiple scene images and a ground truth map of object positions used to identify the position information of the preset object in the scene image; processing the data in the global localization training dataset using the network model to be trained to obtain the predicted velocity vector and actual velocity vector output by the network model to be trained in the flow matching dimension; determining the model loss function based on the predicted velocity vector and actual velocity vector; and training the network model to be trained using the model loss function to obtain the global localization model. The global localization model is trained to determine the coordinates of the adaptation region and the size of the bounding box where the preset object can be placed, and learns the aesthetics and spatial logic of harmonious composition.

[0042] To train the global localization model and obtain a model with high inference accuracy, a global localization training dataset can be acquired first. This dataset can be obtained by accessing a preset area or device. Then, the network model to be trained processes the data in the global localization training dataset to obtain the predicted velocity vector and the actual velocity vector output by the network model in the flow matching dimension. The predicted velocity vector represents the encoding deviation between the training image encoding and the generated scene image encoding during image processing, indicating the velocity information corresponding to adding noise to the training images in the global localization training dataset. The actual velocity vector represents the encoding deviation between the noisy image encoding and the ground truth of the local scene image during image processing, indicating the velocity information corresponding to the process of generating the target image from the noisy image.

[0043] Specifically, predicting the velocity vector The actual velocity vector can be based on the output of the network model being trained; in some instances, the predicted velocity vector... Process scene images can be generated based on the network model to be trained. Preset time step and network training conditions To determine, among them, network training conditions This can include: scene images and descriptive information; at this point, the predicted velocity vector... It can be represented as The actual velocity vector can be based on the true value of the local scene image. and noisy images To determine this, the actual velocity vector can be expressed as ( ).

[0044] Determining the predicted velocity vector and actual velocity vector ( After that, the predicted velocity vector can be... and actual velocity vector ( The analysis process is performed to determine the model loss function. In some instances, the training operation of the global localization model can be implemented based on the objective function of flow matching. In this case, the determined model loss function can be implemented as the L2 norm loss function, which can be expressed as follows: in, The model loss function, This is the predicted velocity vector determined based on the network model to be trained. To preset the time step, The image represents the process scene generated based on the network model to be trained; it can be a noisy image. to local scene image ground truth The interpolated image at time step t, These are the network hyperparameters corresponding to the network model to be trained. The training conditions for the network can specifically include scene images. and description information ; This is the actual velocity vector determined based on the network model to be trained.

[0045] After determining the model loss function, the model loss function can be used. The network model to be trained is trained to obtain a global localization model after training. The obtained global localization model is used to determine the coordinates of the fitting region and the size of the bounding box into which the preset object can be placed.

[0046] After obtaining the descriptive information and scene images, a pre-trained global localization model can be used. The descriptive information and scene images are analyzed and processed. Specifically, the descriptive information and scene images can be input into the global localization model through different interfaces for processing, thereby determining the local scene image output by the global localization model for generating the preset object. For the local scene image, in order to improve the quality and efficiency of the image generation operation to a certain extent, the resolution of the generated local scene image can be made smaller than the resolution of the scene image. That is, the global localization model can not only perform prediction operations on the local scene image, but also perform downsampling operations on the local scene image.

[0047] Step S203: Use a pre-trained local rendering model to process the local scene image and description information to generate an object foreground image corresponding to the preset object.

[0048] Since the determined local scene image does not include the preset object, but is only a local area image in the scene image that is suitable for generating the preset object, determined by the global localization model, in order to stably generate the target scene image that includes the preset object, after determining the local scene image used to generate the preset object, the local scene image and description information can be processed to generate the object foreground image corresponding to the preset object.

[0049] In some instances, the object foreground image can be obtained by processing a local scene image and descriptive information using a pre-trained local rendering model. This local rendering model is trained to generate a single, high-quality object image within a region defined by the scene image. In this case, processing the local scene image and descriptive information using the pre-trained local rendering model to generate an object foreground image corresponding to a preset object can include: inputting the local scene image and descriptive information into the local rendering model for analysis and processing, and obtaining the object foreground image output by the local rendering model corresponding to the preset object. This effectively ensures the stability and reliability of the object foreground image generation.

[0050] Step S204: Perform image fusion on the local scene image and the object foreground image to generate a target scene image, which includes a preset object.

[0051] Since the object foreground image is the generated object image used to be embedded into the environment region corresponding to the local scene image, after determining the local scene image and the object foreground image, image fusion processing can be performed on the local scene image and the object foreground image to generate a target scene image including the preset object.

[0052] In some instances, the target scene image can be generated based on a pre-trained image fusion model. In this case, image fusion of the local scene image and the object foreground image to generate the target scene image may include: determining the pre-trained image fusion model; inputting the local scene image and the object foreground image into the image fusion model for image fusion operation to obtain the target scene image output by the image fusion model. This effectively achieves stable and reliable generation of the target scene image.

[0053] In other instances, the target scene image can be generated not only based on a pre-trained image fusion model, but also based on a composite image obtained from a local scene image and an object foreground image. In this case, image fusion of the local scene image and the object foreground image to generate the target scene image may include: image fusion of the local scene image and the object foreground image to obtain a composite image; and generating the target scene image based on the composite image.

[0054] After obtaining the local scene image and the object foreground image, image fusion can be performed directly on the local scene image and the object foreground image. In some instances, the object foreground image can be overlaid on the local scene image using a direct overlay algorithm, or the object foreground image and the local scene image can be fused using an Alpha blending algorithm to obtain a composite image. The resolution of the composite image can be the same as or similar to the resolution of the local scene image.

[0055] After obtaining the composite image, a target scene image can be generated based on the composite image. In some instances, the composite image can be directly determined as the target scene image. Alternatively, the target scene image can be generated based on an image redrawing operation. In this case, generating the target scene image based on the composite image may include: performing an image redrawing operation on the composite image to obtain a redrawn image with a resolution greater than that of the composite image; and embedding the redrawn image into the scene image to generate the target scene image.

[0056] Specifically, since the composite image is generated based on a local scene image, its resolution is the same as or similar to that of the local scene image. However, the local scene image's resolution is lower than the scene image's resolution. Therefore, it can be determined that the resolution of the obtained composite image is lower than that of the scene image. Thus, to generate a high-quality target scene image, after obtaining the composite image, an image redrawing operation can be performed to obtain a redrawn image. This redrawn image has a higher resolution than the composite image, and its resolution can be the same as or similar to that of the scene image. After obtaining the redrawn image, it can be embedded into the scene image to generate a target scene image including the preset objects. This effectively achieves both quality and efficiency in generating the target scene image.

[0057] The image generation method provided in this embodiment acquires scene images and descriptive information, processes the descriptive information and scene images using a pre-trained global localization model to determine a local scene image for generating a preset object, and then processes the local scene image and descriptive information using a pre-trained local rendering model to generate an object foreground image corresponding to the preset object. Finally, the local scene image and object foreground image are fused. This effectively achieves automated and high-quality image generation based on a global localization model and a local rendering model, eliminating the need for manual operation and freeing users from tedious and inefficient manual work. It solves the problems of unreasonable composition, disproportionate scale, and unrealistic images caused by reliance on user operation in related technologies. This effectively ensures the authenticity and rationality of the generated target scene image, which can then be used for application operations, such as product publishing, further improving the practicality of the method.

[0058] Figure 3 A flowchart illustrating the process of determining a local scene image for generating a preset object is provided as an exemplary embodiment of this application; based on the above embodiment, refer to the appendix. Figure 3As shown, the process of determining a local scene image may include a downsampling stage and an image cropping stage. During this stage, a pre-trained global localization model is used to process the description information and scene image to determine the local scene image used to generate the preset object. This process may include: Step S301: Use the generative network unit to process the description information and scene image to generate an intermediate image including a preset object. The resolution of the intermediate image is smaller than that of the scene image.

[0059] The global localization model includes a generative network unit for generating low-resolution images and an object segmentation unit for image segmentation. To stably determine local scene images, the generative network unit processes the descriptive information and scene images. Specifically, the generative network unit performs semantic and spatial analysis on the scene images and can directly generate an intermediate image containing a preset object based on the intent of the descriptive information. This intermediate image can be a pre-generated low-fidelity image of the scene image that includes the preset object. Notably, the intermediate image can be a scene region image suitable for generating the preset object.

[0060] For example, a generative network unit can be implemented as a generative network. , These are the network hyperparameters in a generative network; their description information is... The scene image is The middle image is At this point, the intermediate image It can be determined using the following formula: This ensures the stable and reliable generation of intermediate images.

[0061] Step S302: Process the intermediate image using the object segmentation unit to generate a binary mask image corresponding to the preset object.

[0062] Since intermediate images can identify scene regions suitable for generating preset objects, but the pose or object details of the preset objects generated from intermediate images may not meet the high-quality requirements of the image, an object segmentation unit can be used to process the intermediate images to generate a binary mask image corresponding to the preset objects in order to ensure the quality and effect of the image generation operation. For example, the object segmentation unit can be implemented as a generative network. , These are the network hyperparameters in a generative network; the intermediate image is... At this point, the binary mask image It can be determined using the following formula: This achieves stable processing of the binary mask image. To ensure the accuracy and reliability of the generated data.

[0063] In some instances, the object segmentation unit can be flexibly trained based on the object type of the preset object. For example, if the preset object is a human object, the object segmentation unit can be implemented as a human segmentation unit; if the preset object is an animal object, the object segmentation unit can be implemented as an animal segmentation unit. This can improve the accuracy and reliability of generating the binary mask image of the preset object to a certain extent.

[0064] Step S303: Crop the scene image based on the binary mask image to determine the local scene image.

[0065] Since a binary mask can identify the size of the area and boundary information to be occupied by the preset object to be generated, after generating a binary mask corresponding to the preset object, the scene image can be cropped based on the binary mask to obtain a local scene image for generating the preset object.

[0066] In some instances, local scene images can be determined based on a pre-trained image cropping network model. In this case, cropping the scene image based on a binary mask to determine the local scene image may include: determining the pre-trained image cropping network model. Then, the binary mask image and the scene image are input into the image cropping network model. The cropping process is performed; for example, a partial scene image is... The image cropping network model is The scene image is The binary mask image is At this point, the local scene image can be determined using the following formula: This allows for the stable determination of local scene images.

[0067] In other instances, the local scene image can be determined by cropping the scene image based on boundary extension parameters. In this case, cropping the scene image based on a binary mask to determine the local scene image may include: determining the boundary extension parameters used to crop the scene image; and cropping the scene image based on the boundary extension parameters and the binary mask to determine the local scene image.

[0068] Specifically, in order to ensure the quality and effect of generating preset objects in the scene image, when it is necessary to determine the local scene image, the boundary expansion parameters used for cropping the scene image can be determined first. These boundary expansion parameters can be determined based on the parameter configuration operation input by the user, or the boundary expansion parameters can be pre-configured parameters stored in a preset area or preset device. In this case, the boundary expansion parameters used for cropping the scene image can be determined by accessing the preset area or preset device.

[0069] After determining the boundary expansion parameters used for cropping the scene image, the scene image can be cropped based on the boundary expansion parameters and the binary mask image, thereby stably determining the local scene image. In some instances, the boundary expansion parameters can have different representations, such as: the boundary expansion parameters can be implemented as pixel values ​​in the image, or the boundary expansion parameters can be implemented as the proportion information of the image; different representations can correspond to different cropping methods.

[0070] Example 1: When the boundary expansion parameter is implemented as a pixel value, the boundary of the area to be occupied by the preset object corresponding to the binary mask image can be determined first in the scene image. Then, based on the boundary expansion parameter, an expansion operation can be performed outward from the area boundary. The specific expansion distance can be determined based on the pixel value corresponding to the boundary expansion parameter, thereby obtaining the expanded area. The size of the expanded area is larger than the size of the area to be occupied by the preset object. Then, cropping can be performed on the scene image based on the expanded area, thereby stably determining the local scene image. The determined local scene image corresponds to the expanded area, thus ensuring that the local scene image obtained after the cropping operation can stably generate the preset object and guarantee the generation quality and effect of the preset object.

[0071] Example 2: When the boundary expansion parameter is implemented as proportion information, the boundary of the area to be occupied by the preset object corresponding to the binary mask image can be determined first in the scene image. Then, based on the boundary expansion parameter, an expansion operation can be performed outward from the area boundary. The specific expansion distance can be determined based on the proportion information corresponding to the boundary expansion parameter. For example, if the proportion information is 0.5 and the size of the area to be occupied by the preset object is 200*200 pixels, the expansion distance can be 0.5*200 pixels = 100 pixels. Or, if the proportion information is 0.1 and the preset object's size is 200*200 pixels, the expansion distance can be 0.5*200 pixels = 100 pixels. If the area required by the object is 500*500 pixels, then the outward expansion distance can be 0.1*500 pixels = 50 pixels. This results in an expanded area that is larger than the area required by the object. The expanded area can then be used to crop the scene image, thus stably determining the local scene image. The determined local scene image corresponds to the expanded area, ensuring the accuracy and reliability of the local scene image obtained through the cropping operation, and guaranteeing the generation quality and effect of the object.

[0072] In this embodiment, a generative network unit is used to process the description information and scene image to generate an intermediate image including a preset object. Then, an object segmentation unit is used to process the intermediate image to generate a binary mask image corresponding to the preset object. The scene image is then cropped based on the binary mask image, thereby stably generating local scene images. This effectively ensures the accuracy and reliability of generating local scene images.

[0073] Figure 4 A flowchart illustrating the generation of an object foreground image corresponding to a preset object is provided as an exemplary embodiment of this application; based on the above embodiment, refer to the appendix. Figure 4 As shown, the process of generating an object foreground image corresponding to a preset object can include an image generation stage and an image rendering stage. During this stage, a pre-trained local rendering model is used to process the local scene image and descriptive information. Generating the object foreground image corresponding to the preset object can include: Step S401: Use the local inference unit to perform image generation operation on the local scene image and description information to obtain the skeleton pose map of the preset object. The fit between the skeleton pose map and the local scene image is greater than or equal to a preset threshold.

[0074] The local rendering model may include a local inference unit for generating a skeletal pose map and a local rendering unit for implementing image rendering operations; the local inference unit and the local rendering unit included in the local rendering model may correspond to the same network architecture. Furthermore, the local inference units and the first adaptive weights loaded by the network architecture Correspondingly, the local rendering unit and the second adaptive weights loaded by the network architecture Correspondingly, the first adaptive weights are different from the second adaptive weights.

[0075] As shown above, the local rendering unit and the local inference unit are based on the same network architecture with different adaptive weights. Specifically, the network architecture loads the first adaptive weight. In this case, the local inference unit is awakened; the network architecture loads the second adaptive weights. In this case, the local rendering unit is activated. Therefore, before using the local inference unit to perform image generation operations on the local scene image and description information, when it is necessary to use the local inference unit to perform image generation operations on the local scene image and description information, the local inference unit needs to be activated. At this time, the method in this embodiment may further include: determining the image generation progress corresponding to the scene image; loading the first adaptive weight based on the image generation progress control network architecture, and determining the local inference unit.

[0076] Since the local inference unit and local rendering unit in the local rendering model are used to perform different image processing operations in different image processing stages, in order to wake up the required local inference unit or local rendering unit at the correct image processing stage, the image generation progress corresponding to the scene image can be determined before using the local inference unit to perform image generation operations on the local scene image and description information. This image generation progress can be determined by the processing operation of the scene image by the preset detection network model, or it can be determined by the data type generated during the processing operation of the scene image. For example, if the generated data type is a local scene image, the image generation progress can be determined to be the local skeleton inference stage; if the generated data type is a skeleton pose map, the image generation progress can be determined to be the local rendering stage, and so on.

[0077] When the determined image generation progress indicator requires the local inference unit to be awakened, the first adaptive weight can be loaded based on the image generation progress control network architecture, thereby waking up and determining the local inference unit. Then, based on the determined local inference unit, image generation operation can be performed on the local scene image and description information, so as to stably obtain the skeletal pose map of the preset object.

[0078] For example, a local scene image is The description information is as follows The skeletal pose diagram is Local reasoning units can be implemented as At this point, the skeletal pose diagram can be determined using the following formula: The above This refers to the model hyperparameters of the network architecture corresponding to the aforementioned local inference units. The first adaptive weight is assigned to the local inference unit, which effectively ensures the stability and reliability of generating the skeletal pose map of the preset object.

[0079] It is important to note that when the local inference unit generates the skeletal pose map of the preset object, the adaptation degree between the generated skeletal pose map and the local scene image is greater than or equal to a preset threshold. The adaptation degree between the skeletal pose map and the local scene image can include: the adaptation degree between the skeletal pose map and the local scene image in terms of image style, the adaptation degree between the skeletal pose map and the local scene image in terms of image size, the interaction adaptation degree between the pose information in the skeletal pose map and the scene environment of the local scene image, etc.

[0080] Step S402: Use the local rendering unit to perform image rendering operations on the skeleton pose map and description information to generate an object foreground image corresponding to the preset object.

[0081] Since the skeletal pose map only includes the anatomical features of the preset object, in order to generate a target scene image that includes a preset object with natural pose and rich details, after obtaining the skeletal pose map of the preset object, the local rendering unit can be used to perform image rendering operations on the skeletal pose map and description information to generate an object foreground image corresponding to the preset object.

[0082] In some instances, when the local rendering unit is currently active, the skeletal pose map and description information can be directly input into the local rendering unit for image rendering. This allows the local rendering unit to output an object foreground image corresponding to the preset object, thus effectively ensuring the accuracy and reliability of the object foreground image generation.

[0083] In other instances, when the local rendering unit is not currently awake, it can be awakened and determined first. Then, the awakened local rendering unit can be used to perform image rendering operations on the skeletal pose map and description information. In this case, using the local rendering unit to perform image rendering operations on the skeletal pose map and description information to generate an object foreground image corresponding to a preset object may include: controlling the network architecture to load the second adaptive weights, determining the local rendering unit; and using the local rendering unit to perform image rendering operations on the skeletal pose map and description information to generate an object foreground image.

[0084] Since the local rendering unit corresponds to the second adaptive weight, in order to wake up and determine the local rendering unit, the network architecture can be controlled to load the second adaptive weight, thereby stably waking up the local rendering unit. Then, the woken local rendering unit can be used to perform image rendering operations on the skeleton pose map and description information, thus stably generating the object foreground image, which can be realized as a high-fidelity object image.

[0085] For example, a skeletal pose diagram is The description information is as follows The skeletal pose diagram is Local rendering units can be implemented as At this point, the foreground image of the object can be determined using the following formula: The above This refers to the model hyperparameters of the network architecture corresponding to the aforementioned local rendering units. This is the second adaptive weight corresponding to the local rendering unit, which effectively ensures the stability and reliability of generating the foreground image of the object.

[0086] In this embodiment, the local inference unit performs image generation operations on the local scene image and description information to obtain the skeletal pose map of the preset object. Then, the local rendering unit performs image rendering operations on the skeletal pose map and description information, thereby stably generating the object foreground image corresponding to the preset object. This effectively achieves the accuracy and reliability of the two-stage generation of the object foreground image through the local inference unit and the local rendering unit.

[0087] Figure 5 A schematic flowchart of another image generation method provided as an exemplary embodiment of this application; based on any of the above embodiments, refer to the appendix. Figure 5 As shown, before processing the local scene image and description information using a pre-trained local rendering model, the local rendering model can be determined through model training operations. In this case, the method in this embodiment may further include: Step S501: Determine the first loss function corresponding to the inference unit to be trained and the second loss function corresponding to the rendering unit to be trained. The inference unit to be trained is used to be trained as a local inference unit in the local rendering model; the rendering unit to be trained is used to be trained as a local rendering unit in the local rendering model.

[0088] In order to enable the training operation of the local rendering model, a first loss function corresponding to the inference unit to be trained can be determined. Specifically, a first training dataset can be obtained first. The first training dataset includes multiple local scene images and a ground truth map of the pose of a preset object in the scene image. This first training dataset can be obtained by accessing a preset region or a preset device. Then, the data in the first training dataset is processed by the inference unit to be trained to obtain the predicted velocity vector and the actual velocity vector output by the inference unit to be trained in the flow matching dimension. The predicted velocity vector can be used to characterize the encoding deviation between the training image encoding and the generated pose image encoding during image processing. It can identify the velocity information corresponding to the noise addition process of the training images in the first training dataset. The actual velocity vector can be used to characterize the encoding deviation between the noise image encoding and the ground truth of the skeletal pose map during image processing. It can identify the velocity information corresponding to the process of generating the ground truth of the skeletal pose map from the noise image.

[0089] Specifically, predicting the velocity vector The actual velocity vector can be based on the output of the inference unit to be trained; in some instances, the predicted velocity vector... It can be based on the process pose image generated by the inference unit to be trained. Preset time step and network training conditions To determine, among them, network training conditions This can include: scene images and descriptive information; in this case, the predicted velocity vector can be represented as... The actual velocity vector can be based on the ground truth of the local scene image. and noisy images To determine this, the actual velocity vector can be expressed as ( ).

[0090] After determining the predicted velocity vector and the actual velocity vector, the predicted velocity vector and the actual velocity vector can be analyzed and processed to construct a first loss function. The first loss function is then used to train the inference unit to be trained, thereby obtaining a local inference unit after training. The obtained local inference unit is used to generate a skeletal pose map that is adapted to the local scene image.

[0091] In some instances, the training operation of the local inference unit can be implemented based on the objective function of flow matching. In this case, the determined first loss function can be implemented as the L2 norm loss function, which can be expressed as follows: in, For the first loss function, The predicted velocity vector is determined based on the inference unit to be trained. To preset the time step, The pose image generated based on the inference unit to be trained can be a noisy image. to local scene image ground truth The interpolated image at time step t, The training conditions for the network can specifically include scene images. and description information ; This is the actual velocity vector determined based on the inference unit to be trained.

[0092] Correspondingly, in order to achieve the training operation of the local rendering model, in addition to determining the first loss function corresponding to the inference unit to be trained, a second loss function corresponding to the rendering unit to be trained can also be determined. Specifically, a second training dataset can be obtained first. The second training dataset includes multiple skeletal pose maps and rendering ground truth maps generated by the local inference units. This second training dataset can be obtained by accessing a preset region or a preset device. Then, the data in the second training dataset is processed by the rendering unit to be trained to obtain the predicted velocity vector and the actual velocity vector output by the rendering unit to be trained in the flow matching dimension. The predicted velocity vector can be used to characterize the encoding deviation between the training image encoding and the generated foreground image encoding during image processing. It can identify the velocity information corresponding to the noise addition process of the training image in the second training dataset. The actual velocity vector can be used to characterize the encoding deviation between the noise image encoding and the object foreground image ground truth during image processing. It can identify the velocity information corresponding to the process of generating the object foreground image ground truth from the noise image.

[0093] Specifically, predicting the velocity vector The actual velocity vector can be based on the output of the inference unit to be trained; in some instances, the predicted velocity vector... The process foreground image can be generated based on the inference unit to be trained. Preset time step and network training conditions To determine, among them, network training conditions This can include: scene images and descriptive information; in this case, the predicted velocity vector can be represented as... The actual velocity vector can be based on the ground truth of the local scene image. and noisy images To determine this, the actual velocity vector can be expressed as ( ).

[0094] After determining the predicted velocity vector and the actual velocity vector, the predicted velocity vector and the actual velocity vector can be analyzed and processed to construct a second loss function. The second loss function is then used to train the rendering unit to be trained, thereby obtaining the local rendering unit after training. The obtained local rendering unit is used to generate a skeletal pose map that is adapted to the local scene image.

[0095] In some instances, the training operation of the local rendering unit can be implemented based on the objective function of stream matching. In this case, the determined second loss function can be implemented as the L2 norm loss function, which can be expressed as follows: in, For the first loss function, The predicted velocity vector is determined based on the inference unit to be trained. To preset the time step, The foreground image generated based on the inference unit to be trained can be a noisy image. to local scene image ground truth The interpolated image at time step t, This is a skeletal pose diagram. The training conditions for the network can specifically include scene images. and description information ; This is the actual velocity vector determined based on the inference unit to be trained.

[0096] Step S502: Use the first loss function to perform preliminary training on the inference unit to be trained, and obtain intermediate inference units.

[0097] In order to enable the local inference unit to generate a more reasonable skeletal pose map based on the local scene image, after determining the first loss function corresponding to the inference unit to be trained, the first loss function can be used to perform preliminary and separate training operations on the inference unit to be trained, so as to stably obtain the intermediate inference unit after the preliminary training operation.

[0098] Step S503: Train the intermediate inference unit using the first loss function and the second loss function to obtain the local inference unit. The local inference unit is trained to generate a skeletal pose map that adapts to the local scene image. The local inference unit is used as part of the pre-trained local rendering model.

[0099] To ensure the training quality and effectiveness of the local inference units, a first loss function and a second loss function can be used to jointly train the local inference units. Therefore, after determining the first loss function corresponding to the inference unit to be trained and the second loss function corresponding to the rendering unit to be trained, the intermediate inference units can be trained directly using the first and second loss functions. For example, the intermediate inference units can be initially trained using the first loss function, and then retrained using the second loss function. This yields the trained local inference units, which are then trained to generate skeletal pose maps that adapt to the local scene images. These local inference units are used as part of a pre-trained local rendering model. In other words, after obtaining the local inference units, the trained local inference units can be used to generate the local rendering model.

[0100] In other instances, local inference units can be determined based on training operations using a joint loss function. In this case, training intermediate inference units using a first loss function and a second loss function to obtain local inference units may include: determining a first weight corresponding to the first loss function and a second weight corresponding to the second loss function; performing a weighted summation of the first and second loss functions based on the first and second weights to obtain a joint loss function; and training intermediate inference units based on the joint loss function to obtain local inference units.

[0101] For example, in the first loss function is The second loss function is At that time, we can first determine the relationship with the first loss function. Corresponding first weight and the second loss function The corresponding second weight Among them, the first weight Second weight The weight can be determined based on the weight configuration operation input by the user, or, the first weight. Second weight The parameters can be pre-configured and stored in a preset area. In this case, the parameters can be stably determined by accessing the preset area and the first loss function. Corresponding first weight and the second loss function The corresponding second weight .

[0102] In determining the first loss function Corresponding first weight and the second loss function The corresponding second weight Then, the first loss function and the second loss function can be weighted and summed based on the first and second weights to obtain the joint loss function. Specifically, the joint loss function It can be achieved as = Then, based on the obtained joint loss function, By performing collaborative joint training operations on intermediate inference units, local inference units that have undergone training operations can be stably obtained. These local inference units are then used as part of a pre-trained local rendering model.

[0103] Furthermore, during the aforementioned collaborative training operation, the local inference unit and the local rendering unit are connected in series. The skeletal pose map (i.e., pose layout) generated online by the local inference unit is used as the input of the local rendering unit in real time. The final rendering loss gradient is backpropagated through a fully differentiable path, which not only updates the local rendering unit, but also corrects and optimizes the upstream local inference unit. This makes the collaborative training process form a closed-loop feedback system, which promotes the local inference unit to learn a high-quality pose layout that is "easier to render", thereby realizing the collaborative optimization operation of the two units.

[0104] Furthermore, after obtaining the local inference unit, the local rendering unit in the local rendering model can be trained separately. In this case, the method in this embodiment may also include: performing preliminary training on the rendering unit to be trained based on the joint loss function to obtain the intermediate rendering unit; training the intermediate rendering unit using the second loss function to obtain the trained local rendering unit. The local rendering unit is trained to generate the object foreground image corresponding to the skeleton pose map, and the local rendering unit is used as part of the pre-trained local rendering model.

[0105] Specifically, after weighted summing of the first and second loss functions based on the first and second weights to obtain the joint loss function, preliminary co-training operations can be performed on the rendering units to be trained using the joint loss function to obtain intermediate rendering units. After co-training the local inference units, the network hyperparameters corresponding to the local inference units can be frozen. To ensure the quality and effect of image rendering operations, high-quality ground truth skeleton maps can be used. As input to the model, and by using the second loss function separately to perform refined training operations on the intermediate rendering units, the trained local rendering units can be obtained, which can then focus on improving the texture, lighting and other expressiveness of the final image, effectively ensuring the quality and effect of image rendering.

[0106] In this embodiment, before processing the local scene image and description information using the pre-trained local rendering model, a first loss function corresponding to the inference unit to be trained and a second loss function corresponding to the rendering unit to be trained are determined. Then, the inference unit to be trained is initially trained using the first loss function to obtain an intermediate inference unit. The intermediate inference unit is then trained using the first and second loss functions to obtain a local inference unit. Similarly, a trained local rendering unit can also be obtained. Then, the trained local inference unit and local rendering unit can be combined to form a trained local rendering model. This ensures the quality and effect of generating the object foreground image corresponding to the preset object using the local rendering model, further improving the stability and reliability of the method.

[0107] For specific applications, please refer to the appendix. Figure 6 As shown, in a home application scenario, taking the human body as a preset object for illustration, this application embodiment provides a human image generation method based on global positioning technology and local fine drawing automated scene perception technology. It aims to solve technical problems such as unreasonable composition, disproportionate proportions, and distorted anatomical structures when generating human bodies in complex large scenes. The main execution body of this method is a human image generation system, which can provide an effective tool for all application scenarios that need to embed high-quality virtual characters, animated characters, or real characters into virtual or real scenes. In some examples, the human image generation system may include a global positioning model and a local fine drawing model, thereby realizing a cascaded generation architecture from macro to micro, "from coarse to fine." Thus, when performing human image generation operations based on the human image generation system, it can render high-fidelity human images with correct structure, rich details, and natural integration with the local environment.

[0108] Specifically, the method for generating this human image may include the following steps: Step 1: Import scene images .

[0109] Users upload a high-resolution, large-scale image that fully showcases their product or space. For example: a complete living room, an outdoor landscape, or a cityscape, where the scene image... It can be an image with a resolution of 768 or higher.

[0110] Step 2: Expressing Intent.

[0111] The human image generation display interface shows an image upload interface for importing scene images and a text upload interface for expressing intent. Users can then input natural language text descriptions through the text upload interface. For example: "A man in business attire stands by the window," or more vaguely, "a model."

[0112] Step 3: Intelligent Positioning and Region Determination (Macro Stage) The system's global localization model It can fully automate the processing of scene images. A deep analysis of the entire corresponding scene was conducted, combined with textual description information. Based on the corresponding text intent, it can autonomously infer and determine an adapted generation area that matches the scene in terms of position, scale, and perspective (i.e., the "local scene image" in the above embodiment), and the above operation does not require user selection.

[0113] In some instances, the inference process for adapting the generated region may include the following steps: low-fidelity image The generation stages include the generation of the human body binary mask and the generation stage of the local scene image, specifically: low-fidelity images The generation stage can be: taking scene images and text description information Input to global localization model In the context of global localization models It can process scene images It performs semantic and spatial analysis and can combine textual description information. Based on the corresponding text intent, a preliminary low-fidelity image containing the target human body in the global scene is directly generated. , specifically .

[0114] The generation stage of the human body binary mask image can be as follows: after obtaining the low-fidelity image... Then, a pre-trained, fixed-weight human segmentation model can be called. The low-fidelity image generated by the above steps Perform analysis and processing. This allows us to obtain an accurate binary mask for the human body. .

[0115] The generation stage of a local scene image can be as follows: after obtaining the human body binary mask... Then, the cropping model can be used on the original high-resolution scene image. Above, using a human body binary mask The corresponding boundary is cropped to create a binary mask that corresponds to the human body. Corresponding adaptation generation region image (That is, corresponding to the local scene image in the above embodiments), when performing a region cropping operation, the cropping model will intelligently expand the cropping area outward by a preset margin. The aforementioned margin can be determined by preset cropping parameters. To determine, specifically, the adapted generated region image It can be determined using the following formula: The generated adaptation region image can provide rich contextual information for subsequent local fine-mapping models.

[0116] Step 4: Interactive posture adjustment (microscopic stage) After determining the adapted generation region image, human body generation can be performed based on it. At this point, a complex global composition problem has been simplified into a clear local creation task, which can be implemented based on a local fine-drawing model. This local fine-drawing model can be achieved through a unified backbone network. By loading different low-rank adaptive weights (first adaptive weights) Second adaptive weights To perform different subtasks, which may include: with the first adaptive weights The corresponding sub-local reasoning task, and the second adaptive weights The corresponding sub-local rendering task.

[0117] The interactive pose adjustment can include a local skeletal inference stage. Specifically, this stage can be implemented using a skeletal inference unit. This unit can perform pose conception and generation within a defined range, ensuring the naturalness of the interaction between the pose and the local environment. For example, it can generate a pose where a hand rests on the edge of a chair armrest, or a pose where the feet are on a step. The aforementioned skeletal inference unit can be loaded with a first adaptive weight. backbone network (That is, corresponding to the local inference unit in the above embodiments), specifically, the skeleton inference unit can adapt and generate region images after the region cropping operation. and text description information The analysis process generates a potential scene-aware layout. After decoding, this layout becomes a precise and natural human skeletal pose image overlaid on a local background. The above-mentioned human skeletal posture diagram The reasoning process can be abstracted as follows: , As a local reasoning unit, For local scene images, For text description information, This refers to the model hyperparameters of the network architecture corresponding to the aforementioned local inference units. This is the first adaptive weight corresponding to the local inference unit.

[0118] Furthermore, within the already located adaptation generation area, the system's locally refined model can automatically perform preliminary pose inference and present the user with a structurally sound human skeleton pose diagram that interacts with the environment (such as sofa armrests and step heights). Then, regarding the human skeleton pose diagram, the user can: a. Direct Confirmation: If you are satisfied with the posture corresponding to the generated human skeleton posture diagram, you can proceed directly to the next step.

[0119] b. Optional fine-tuning: If you wish to obtain a more creative specific pose, you can make fine adjustments to the bones in the human skeleton pose diagram through intuitive editing tools (such as dragging joint nodes) to obtain and output a human skeleton pose diagram that meets the user's needs.

[0120] Step 5: Rendering and Compositing (Microscopic Stage) Skeletal pose diagram Once confirmed (whether automatically generated by the system or adjusted by the user), the system can use this final skeletal structure benchmark to render a high-fidelity model image (i.e., the object foreground image in the above embodiment) that conforms to the text description and blends light and shadow with the environment in a fine manner, so as to seamlessly composite it into the original scene corresponding to the scene image.

[0121] The above rendering and compositing operations can include: a local appearance rendering stage, and a blending and refining stage, specifically: Local appearance rendering stage: Implemented through the skeletal rendering unit, which can load a second adaptive weight. backbone network The skeletal rendering unit can receive the human skeletal pose map generated in the previous step. As a structural constraint, and parsing the appearance descriptions in the text description information (e.g., "silk dress", "leather jacket", etc.), the skeleton is filled with photorealistic details such as skin, clothing, and hair. Furthermore, it can match the correct lighting and reflection information based on the local ambient lighting. The above image rendering process can be abstracted as follows: , Foreground image of human body For skeletal rendering units, This is a skeletal pose diagram. For text description information, This refers to the model hyperparameters of the network architecture corresponding to the aforementioned local rendering units. This is the second adaptive weight corresponding to the local rendering unit, which effectively ensures the quality and effect of the generated human foreground image. Furthermore, the aforementioned human foreground image... It can be implemented as a high-fidelity human body image, which the system can then embed back into the original large-scene image. The corresponding position is used to complete the final image generation operation.

[0122] Fusion and Refinement Stage: After acquiring the foreground image of the human body and local scene images Then, the human foreground image can be... Precisely overlaid on local scene images A preliminary composite image is created at the corresponding position in the background. Then, a powerful image redrawing model is used to perform embedding operations on the composite image. The image redrawing model can focus on optimizing the seam edges and refining the lighting details, ensuring that the background content is not changed, and finally outputting a seamless, detailed, high-fidelity image. ,Right now: ,in, For high-fidelity images, Redraw the model for the image. For use in human foreground images Precisely overlaid on local scene images A preliminary composite model is formed at the corresponding position of the background.

[0123] In obtaining high-fidelity images Then, it can be embedded back into the original large-scene image. Generate the target scene image at the corresponding position. Based on target scene images Complete the final delivery operation.

[0124] Step 6: Immediate Delivery Users can quickly obtain a commercial-grade image of their target scene with high background fidelity, a well-positioned model, a natural pose, and excellent image quality. .

[0125] The technical solution provided in this application embodiment realizes a novel global-local cascaded generation architecture. This cascaded architecture offers a new technical path for achieving high-fidelity and robust scene human body synthesis. Furthermore, it can automatically and intelligently decompose the human body synthesis task into two orthogonal stages: "macroscopic composition planning" and "microscopic detail realization." This decouples the macroscopic composition and microscopic detail generation technologies, systematically ensuring the compositional rationality, dimensional coordination, and anatomical accuracy of the generated human body in complex scenes, resulting in a significant improvement in image generation quality and stability. Specifically, this technical solution can achieve the following effects: 1) Solves the problems of "blind composition" and "arbitrary structure": In the global positioning stage, a global positioning model is used to replace the user's manual operation. Moreover, the global positioning model can perform aesthetic and logical composition reasoning with the human body as the center, which effectively avoids composition errors such as disproportionate figures and floating positions, and improves the rationality of the generated human body in macro composition from the top-level design. In addition, the professional composition planning capability is fully automated, which solves the problem that ordinary users have difficulty determining the position of the human body in open scenes. Users only need to make optional and more creative pose fine-tuning, thus making the operation of this technical solution simpler. 2) Alleviating the problems of "process disruption" and "error amplification": In the local fine-drawing stage, the process of "first positioning and then fine drawing" is adopted, and an internal "skeleton inference unit" is introduced to perform pose inference operations. This provides reliable anatomical structural constraints for human body generation, effectively avoiding common generation errors such as human body proportion imbalance and structural distortion in general models, and ensuring high success rate and high image quality of the output results. In addition, for the local fine-drawing model, it is trained through a collaborative joint training strategy. This establishes a dynamic two-way information path between the inference and rendering stages. The feedback signal of the rendering result can guide and optimize the upstream skeleton inference process in real time and accurately. Specifically, by constructing a differentiable closed-loop feedback system that allows gradient signal backpropagation, the two functional units can "mutually guide and collaboratively optimize" during training. This solves the error accumulation problem of traditional staged methods and is the core technical guarantee for ensuring the quality of micro-detail generation. It ensures the rationality of the pose and the high degree of unity of the final rendering effect, so that the generated human anatomical structure is accurate, the pose is natural, and the details are rich.

[0126] 3) Quantitative evaluation metrics (such as Fraser start distance and kernel start distance) and manual assessment results show that the target scene images generated using the above method achieve a high level of realism and plausibility, with good scene integration and background fidelity. Specifically, in human image generation, all operations are precisely limited to the necessary range. Except for the newly generated character, the details of the original scene are highly preserved, achieving a "seamless" addition effect. By introducing optional pose editing functions in the local fine-drawing stage, users can finely adjust the skeletal poses generated by the system, achieving a combination of automation convenience and creative freedom. Furthermore, this solution can shorten the traditional image generation cycle from days or weeks to minutes, thereby providing users with massive, low-cost, and high-quality visual content production capabilities, further improving image generation efficiency and reducing image generation costs, effectively ensuring the practicality of the solution.

[0127] Figure 7 A schematic diagram of an image generation apparatus provided for an exemplary embodiment of this application; see attached diagram. Figure 7 As shown, this embodiment provides an image generation apparatus for performing the above-described... Figure 2 The image generation method shown, specifically, the image generation apparatus may include: The first acquisition module 11 is used to acquire scene images and description information, wherein the description information is used to indicate the generation of preset objects in the scene images; The first processing module 12 is used to process the description information and scene image using a pre-trained global localization model to determine a local scene image for generating a preset object, wherein the resolution of the local scene image is smaller than the resolution of the scene image. The first processing module 12 is also used to process the local scene image and description information using a pre-trained local rendering model to generate an object foreground image corresponding to the preset object. The first generation module 13 is used to perform image fusion on the local scene image and the object foreground image to generate a target scene image, which includes a preset object.

[0128] The image generation apparatus in this embodiment may also perform the above-described... Figures 1-6 The description of the embodiments shown is for reference only, and will not be elaborated upon here.

[0129] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as 11, 12, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0130] like Figure 8 As shown, this embodiment provides an electronic device for performing the above-described... Figure 2 The image generation method shown may include an electronic device comprising a memory 24 and a processor 25.

[0131] Memory 24 is used to store computer programs and can be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, data structures, contact data, phone book data, messages, pictures, videos, etc.

[0132] Processor 25, coupled to memory 24, is configured to execute a computer program in memory 24 for: acquiring a scene image and descriptive information, the descriptive information indicating the generation of a preset object in the scene image; processing the descriptive information and the scene image using a pre-trained global localization model to determine a local scene image for generating the preset object, the local scene image having a resolution lower than that of the scene image; processing the local scene image and the descriptive information using a pre-trained local rendering model to generate an object foreground image corresponding to the preset object; and performing image fusion on the local scene image and the object foreground image to generate a target scene image, the target scene image including the preset object.

[0133] Furthermore, such as Figure 8 As shown, the electronic device also includes other components such as a communication component 26, a display 27, a power supply component 28, and an audio component 29. Figure 8 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 8 The components shown. Additionally... Figure 8The components within the center frame are optional, not mandatory, and their specific requirements depend on the product form of the work node. In this embodiment, the work node can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device, or a server-side device such as a conventional server, cloud server, or server array. If the work node in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 8 The components within the center frame; if the working node in this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, it may not include... Figure 8 The component within the center frame.

[0134] The aforementioned memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0135] The aforementioned communication component is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel.

[0136] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0137] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0138] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0139] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile components, or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Accordingly, this application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to implement the steps in the above method embodiments. It should be understood that each step or combination of steps in the above method flow can be implemented by the computer program or instructions. In addition, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device can be implemented as a means to implement the corresponding functions in the above method embodiments.

[0140] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0141] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. An image generation method, characterized in that, include: Acquire a scene image and descriptive information, wherein the descriptive information is used to indicate the generation of a preset object in the scene image; The description information and the scene image are processed using a pre-trained global localization model to determine a local scene image for generating the preset object, wherein the resolution of the local scene image is smaller than the resolution of the scene image; The local scene image and the description information are processed using a pre-trained local rendering model to generate an object foreground image corresponding to the preset object; The local scene image and the object foreground image are fused to generate a target scene image, which includes the preset object.

2. The method according to claim 1, characterized in that, The global localization model includes a generative network unit and an object segmentation unit; the pre-trained global localization model processes the description information and the scene image to determine a local scene image for generating the preset object, including: The generator network unit processes the description information and the scene image to generate an intermediate image including a preset object, wherein the resolution of the intermediate image is smaller than that of the scene image. The intermediate image is processed using the object segmentation unit to generate a binary mask image corresponding to the preset object; The scene image is cropped based on the binary mask image to determine the local scene image.

3. The method according to claim 2, characterized in that, Cropping the scene image based on the binary mask image to determine the local scene image includes: Determine the boundary expansion parameters used to crop the scene image; The scene image is cropped based on the boundary expansion parameters and the binary mask image to determine the local scene image.

4. The method according to claim 1, characterized in that, The local rendering model includes a local inference unit and a local rendering unit; it processes the local scene image and the description information using a pre-trained local rendering model to generate an object foreground image corresponding to the preset object, including: The local inference unit is used to perform image generation operations on the local scene image and the description information to obtain the skeleton pose map of the preset object. The fit between the skeleton pose map and the local scene image is greater than or equal to a preset threshold. The local rendering unit is used to perform image rendering operations on the skeletal pose map and the description information to generate an object foreground image corresponding to the preset object.

5. The method according to claim 4, characterized in that, The local inference unit and the local rendering unit correspond to the same network architecture. The local inference unit corresponds to the first adaptive weight loaded by the network architecture, and the local rendering unit corresponds to the second adaptive weight loaded by the network architecture. The first adaptive weight and the second adaptive weight are different.

6. The method according to claim 5, characterized in that, Before performing image generation operations on the local scene image and the description information using the local inference unit, the method further includes: Determine the image generation progress corresponding to the scene image; Based on the image generation progress, the network architecture is controlled to load the first adaptive weights and determine the local inference unit.

7. The method according to claim 5, characterized in that, The local rendering unit performs image rendering operations on the skeletal pose map and the description information to generate an object foreground image corresponding to the preset object, including: The network architecture is controlled to load the second adaptive weights to determine the local rendering units; The local rendering unit is used to perform image rendering operations on the skeletal pose map and the description information to generate the object foreground image.

8. The method according to any one of claims 1-7, characterized in that, Image fusion is performed on the local scene image and the object foreground image to generate the target scene image, including: The local scene image and the object foreground image are fused to obtain a composite image; Based on the synthesized image, the target scene image is generated.

9. The method according to claim 8, characterized in that, Based on the synthesized image, the target scene image is generated, including: The composite image is redrawn to obtain a redrawn image, wherein the resolution of the redrawn image is greater than the resolution of the composite image. The redrawn image is embedded into the scene image to generate the target scene image.

10. The method according to any one of claims 1-7, characterized in that, Before processing the local scene image and the descriptive information using a pre-trained local rendering model, the method further includes: A first loss function is determined for the inference unit to be trained, and a second loss function is determined for the rendering unit to be trained. The inference unit to be trained is used to be trained as a local inference unit in the local rendering model. The rendering unit to be trained is used to be trained as a local rendering unit in the local rendering model. The first loss function is used to perform preliminary training on the inference unit to be trained, thereby obtaining intermediate inference units; The intermediate inference unit is trained using the first loss function and the second loss function to obtain the local inference unit. The local inference unit is trained to generate a skeletal pose map that adapts to the local scene image, and the local inference unit is used as part of a pre-trained local rendering model.

11. The method according to claim 10, characterized in that, The intermediate inference unit is trained using the first loss function and the second loss function to obtain the local inference unit, including: Determine the first weight corresponding to the first loss function and the second weight corresponding to the second loss function; Based on the first weight and the second weight, the first loss function and the second loss function are weighted and summed to obtain the joint loss function; The intermediate inference unit is trained based on the joint loss function to obtain the local inference unit.

12. The method according to claim 11, characterized in that, After obtaining the local inference unit, the method further includes: The training rendering unit is initially trained based on the joint loss function to obtain intermediate rendering units; The intermediate rendering unit is trained using the second loss function to obtain a trained local rendering unit, which is trained to generate an object foreground image corresponding to the skeletal pose map.

13. An image generation apparatus, characterized in that, include: The first acquisition module is used to acquire a scene image and descriptive information, wherein the descriptive information is used to indicate the generation of a preset object in the scene image; The first processing module is used to process the description information and the scene image using a pre-trained global localization model to determine a local scene image for generating the preset object, wherein the resolution of the local scene image is smaller than the resolution of the scene image. The first processing module is further configured to process the local scene image and the description information using a pre-trained local rendering model to generate an object foreground image corresponding to the preset object; The first generation module is used to perform image fusion on the local scene image and the object foreground image to generate a target scene image, wherein the target scene image includes the preset object.

14. An electronic device, characterized in that, include: A memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method of any one of claims 1-12.

15. A computer storage medium, characterized in that, Used to store a computer program that, when executed by a computer, implements the method of any one of claims 1-12.

16. A computer program product, characterized in that, include: A computer program, when executed by a processor of an electronic device, causes the processor to perform the steps of the method of any one of claims 1-12.

Images