An image generation method, device, equipment, computer readable storage medium and computer program product

Abstract

The application provides an image generation method, device and equipment, a computer readable storage medium and a computer program product. The method comprises: generating a first virtual scene image conforming to first content based on a first prompt word representing the first content; generating a binary mask based on a redraw area of the first virtual scene image; performing image redraw processing on the first virtual scene image based on a second prompt word representing image differences and the binary mask to obtain a second virtual scene image; wherein the differences between the second virtual scene image and the first virtual scene image conform to the image differences, and the redraw area corresponding to the image redraw processing is the same as the redraw area corresponding to the binary mask. Through the application, virtual scene images with consistent content can be generated in batches, and the redraw area thereof can be redrawn to ensure that there are image differences between the virtual scene images in the redraw area, avoid irrelevant areas from being modified by mistake, and improve the accuracy and efficiency of the redraw.

Description

Technical Field

[0001] This application relates to image processing technology, and more particularly to an image generation method, apparatus, device, computer-readable storage medium, and computer program product. Background Technology

[0002] In related technologies, when generating multiple virtual scene images with inherent differences, it is necessary to manually identify and draw the differences between these images. This manual identification and drawing process is time-consuming, and the representation of the multiple virtual scene images is difficult to standardize. When the manually identified differences are densely packed, multiple differences may be incorrectly identified as a single difference. Furthermore, virtual scenes with unacceptable differences require further manual modification, further increasing time costs and operational complexity, making it difficult to meet the requirements for efficient and standardized batch generation of virtual scene images. Summary of the Invention

[0003] This application provides an image generation method, apparatus, device, computer-readable storage medium, and computer program product, which can generate virtual scene images with consistent content in batches and redraw their redrawing areas to ensure that there are image differences between virtual scene images in the redrawing areas, avoid accidental modification of irrelevant areas, and improve the accuracy and efficiency of redrawing.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an image generation method, the method comprising:

[0006] Based on the first prompt word representing the first content, a first virtual scene image that conforms to the first content is generated;

[0007] A binary mask is generated based on the redrawn area of ​​the first virtual scene image;

[0008] Based on the second cue word representing the image difference and the binary mask, the first virtual scene image is redrawn to obtain the second virtual scene image;

[0009] Wherein, the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawing area corresponding to the image redrawing process is the same as the redrawing area corresponding to the binary mask.

[0010] This application provides an image generation apparatus, including:

[0011] The first virtual scene image generation module is used to generate a first virtual scene image that conforms to the first content based on a first prompt word representing the first content.

[0012] A binary mask generation module, configured to generate a binary mask based on the redrawing area of the first virtual scene image;

[0013] A second virtual scene image generation module, configured to perform image redrawing processing on the first virtual scene image based on a second prompt word characterizing image difference and the binary mask to obtain a second virtual scene image; wherein, the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawing area corresponding to the image redrawing processing is the same as the redrawing area corresponding to the binary mask.

[0014] In the above solution, the image generation device further includes a first image generation model training module, configured to add initialization parameters to a target network layer in a diffusion model to obtain a first image generation model; acquire first image samples belonging to the same style; keep the original parameters in the first image generation model unchanged, and perform transfer learning on the first image generation model based on the first image samples to update the initialization parameters of the first image generation model, so as to obtain an updated first image generation model, wherein the updated first image generation model is used to generate images conforming to the style.

[0015] In the above solution, the first virtual scene image generation module is further configured to perform text encoding processing on a first prompt word to obtain a first text encoding; perform denoising processing on a noise image based on the first text encoding to obtain a first virtual scene image conforming to the first content.

[0016] In the above solution, the first virtual scene image generation module is further configured to perform denoising processing on the input of the nth denoising network through the nth denoising network in N cascaded denoising networks, and transmit the nth denoising result output by the nth denoising network to the (n + 1)th denoising network to continue denoising processing to obtain the (n + 1)th denoising result corresponding to the (n + 1)th denoising network; use the Nth denoising result output by the Nth denoising network as the first virtual scene image; wherein, n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the nth denoising network is the noise image and the first text encoding. When n takes the value of 2 ≤ n < N, the input of the nth denoising network is the (n - 1)th denoising result output by the (n - 1)th denoising network and the first text encoding.

[0017] In the above solution, the binary mask generation module is further configured to obtain a to-be-filled image having the same size as the first virtual scene image; for each pixel position in the to-be-filled image: when the pixel at the pixel position in the first virtual scene image is within the redrawing area, set the pixel value at the pixel position in the to-be-filled image to a first pixel value; when the pixel at the pixel position in the first virtual scene image is not within the redrawing area, set the pixel value at the pixel position in the to-be-filled image to a second pixel value; and use the to-be-filled image with the pixel values set as the binary mask.

[0018] In the above solution, the second virtual scene image generation module is further configured to perform image encoding processing on the first virtual scene image to obtain a first image encoding, and perform image encoding processing on the binary mask to obtain a second image encoding; perform text encoding processing on the second prompt to obtain a second text encoding; denoise the input of the m-th denoising network through the m-th denoising network in M cascaded denoising networks, and transmit the m-th denoising result output by the m-th denoising network to the (m + 1)-th denoising network for continued denoising processing to obtain the (m + 1)-th denoising result corresponding to the (m + 1)-th denoising network; generate the second virtual scene image based on the M-th denoising result corresponding to the M-th denoising network; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m < M. When m takes the value of 1, the input of the m-th denoising network is the first image encoding, the second text encoding, and the second image encoding. When m takes the value of 2 ≤ m < M, the input of the m-th denoising network is the (m - 1)-th denoising result output by the (m - 1)-th denoising network, the second text encoding, and the second image encoding.

[0019] In the above solution, the second virtual scene image generation module is further configured to perform semantic segmentation on the first virtual scene image to obtain a semantic label map having the same size as the first virtual scene image; the image redrawing processing of the first virtual scene image based on the second prompt characterizing the image difference and the binary mask to obtain the second virtual scene image includes: performing image redrawing processing on the first virtual scene image based on the second prompt characterizing the image difference, the binary mask, and the semantic label map to obtain the second virtual scene image.

[0020] In the above solution, the second virtual scene image generation module is further configured to perform image encoding processing on the first virtual scene image to obtain a first image encoding, perform image encoding processing on the binary mask to obtain a second image encoding, and perform image encoding processing on the semantic label map to obtain a third image encoding; perform text encoding processing on the second prompt word to obtain a second text encoding; denoise the input of the p-th denoising network through the p-th denoising network in P cascaded denoising networks, and transmit the p-th denoising result output by the p-th denoising network to the p + 1-th denoising network for continuous denoising processing to obtain the p + 1-th denoising result corresponding to the p + 1-th denoising network; generate the second virtual scene image based on the P-th denoising result corresponding to the P-th denoising network; where p is an integer variable starting from 1 and increasing, and the value range of p is 1 ≤ p < P. When p takes the value of 1, the input of the p-th denoising network is the first image encoding, the second text encoding, the second image encoding, and the third image encoding. When p takes the value of 2 ≤ p < P, the input of the p-th denoising network is the p - 1-th denoising result output by the p - 1-th denoising network, the second text encoding, the second image encoding, and the third image encoding.

[0021] In the above solution, the image generation device further includes an image segmentation model training module, configured to obtain a second image sample corresponding to an image segmentation task; perform semantic segmentation on the second image sample through the image segmentation model to obtain a predicted semantic label map; determine a loss function based on the actual semantic label map corresponding to the second image sample and the predicted semantic label map, and update the image segmentation model based on the loss function to obtain an updated image segmentation model.

[0022] An embodiment of the present application provides an electronic device, which includes:

[0023] A memory for storing computer-executable instructions or computer programs;

[0024] A processor, configured to implement the image generation method provided by an embodiment of the present application when executing the computer-executable instructions or computer programs stored in the memory.

[0025] An embodiment of the present application provides a computer-readable storage medium, storing a computer program or computer-executable instructions, which are configured to implement the image generation method provided by an embodiment of the present application when executed by a processor.

[0026] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions, which implement the image generation method provided by an embodiment of the present application when the computer program or computer-executable instructions are executed by a processor.

[0027] The embodiments of this application have the following beneficial effects:

[0028] Based on a first prompt word representing the first content, a first virtual scene image conforming to the first content is generated. This ensures that the content represented by the first virtual scene image is consistent with the content required by the prompt word, allowing for the batch generation of first virtual scene images with consistent representations. A binary mask is generated based on the redrawing area of ​​the first virtual scene image. Based on a second prompt word representing image differences and the binary mask, the first virtual scene image undergoes image redrawing processing to obtain a second virtual scene image. The differences between the second and first virtual scene images conform to the image differences, and the redrawing area corresponding to the image redrawing processing is the same as the redrawing area corresponding to the binary mask. The generation of the binary mask clearly defines the redrawing area range, avoiding accidental modification of irrelevant areas. Combining the redrawing processing of the second prompt word and the binary mask accurately reflects image differences between virtual scene images within the redrawing area, while strictly limiting the redrawing operation to the area corresponding to the binary mask. This process effectively ensures that the redrawn second virtual scene image and the first virtual scene image have image differences within the redrawing area, avoiding accidental modification of irrelevant areas and improving the accuracy and efficiency of virtual scene image redrawing. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the image generation system architecture provided in the embodiments of this application;

[0030] Figure 2 This is a schematic diagram of the structure of the image generation apparatus provided in the embodiments of this application;

[0031] Figures 3A-3D This is a schematic flowchart of the image generation method provided in the embodiments of this application;

[0032] Figure 4 This is a traditional checkpoint production flow chart provided in the embodiments of this application;

[0033] Figure 5 This is a flowchart of the technical solution provided in the embodiments of this application. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0036] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0039] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0040] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0041] (1) LoRA (Low-Rank Adaptation): This is an efficient parameter fine-tuning method that captures task-specific information by freezing the weights of the pre-trained model and training only the low-rank matrix. It reduces the number of fine-tuning parameters, lowers computational costs, and alleviates overfitting. It can quickly adapt to downstream tasks while maintaining the basic capabilities of the model, thus improving the efficiency and effectiveness of fine-tuning.

[0042] (2) Diffusion Model: A deep learning model based on probability generation, which generates data through forward noise addition and reverse denoising. Forward noise addition gradually adds Gaussian noise to the data until it becomes pure noise, while reverse noise reduction uses a neural network to learn and recover patterns from the noise. It generates high-quality data with good detail, is stable during training and less prone to pattern collapse, and can balance generation speed and quality. It is widely used in image generation, style transfer, and other fields.

[0043] (3) Binary mask: A tool used for region marking in image processing, consisting of two pixel values, 0 and 1. 1 represents the redraw area, and 0 represents other areas besides the redraw area. By clearly defining the pixel division, the redraw area in the image can be precisely defined, and the redraw area can be processed in a targeted manner.

[0044] (4) Stable Diffusion Inpainting: An image inpainting technique based on the stable diffusion model. By utilizing the model’s powerful generation capabilities, it fills in the redrawing area of ​​the image with content, thereby repairing missing or damaged parts of the image or replacing the content in the redrawing area, maintaining the overall coherence and authenticity of the image.

[0045] In related technologies, when generating multiple virtual scene images with inherent differences, it is necessary to manually identify and draw the differences between these images. This manual identification and drawing process is time-consuming, and the representation of the multiple virtual scene images is difficult to standardize. When the manually identified differences are densely packed, multiple differences may be incorrectly identified as a single difference. Furthermore, virtual scenes with unacceptable differences require further manual modification, further increasing time costs and operational complexity, making it difficult to meet the requirements for efficient and standardized batch generation of virtual scene images.

[0046] This application provides an image generation method, apparatus, device, computer-readable storage medium, and computer program product, capable of batch generating virtual scene images with consistent content and redrawing their redrawn areas to ensure image differences between virtual scene images in the redrawn areas, avoiding accidental modification of irrelevant areas, and improving the accuracy and efficiency of redrawing. The following describes exemplary applications of the electronic devices provided in this application. The devices provided in this application can be implemented as various types of terminals such as laptops, tablets, and desktop computers, or as servers. The following will describe exemplary applications when the device is implemented as a terminal.

[0047] See Figure 1 , Figure 1This is a schematic diagram of the architecture of the image generation system provided in this application embodiment. In order to support an image generation application, in the image generation system 100, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The server 200 uploads data to the database 500.

[0048] Terminal 400 generates a first virtual scene image that conforms to the first content based on a first prompt word representing the first content; it generates a binary mask based on the redrawn area of ​​the first virtual scene image; based on a second prompt word representing image differences and the binary mask, it performs image redrawing processing on the first virtual scene image to obtain a second virtual scene image; wherein the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawn area corresponding to the image redrawing processing is the same as the redrawn area corresponding to the binary mask. The first and second virtual scene images are then uploaded to server 200, which in turn uploads them to database 500.

[0049] In some embodiments, server 200 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be connected directly or indirectly via wired or wireless communication, which is not limited in this embodiment.

[0050] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the image generation apparatus provided in the embodiments of this application. Figure 2 The terminal 400 shown includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0051] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0052] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0053] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0054] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0055] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0056] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0057] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0058] Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., a display screen, a speaker, etc.) associated with user interface 430;

[0059] The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0060] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 An image generation device 455 stored in memory 450 is shown. This device can be software in the form of programs and plugins, and includes the following software modules: a first virtual scene image generation module 4551, a binary mask generation module 4552, a second virtual scene image generation module 4553, an image segmentation model training module 4554, and a first image generation model training module 4555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0061] In other embodiments, the apparatus provided in this application can be implemented in hardware. For example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image generation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0062] In some embodiments, the terminal or server can implement the image generation method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be native applications (APPs), i.e., programs that need to be installed in the operating system to run; or they can be applets that can be embedded in any APP, i.e., programs that only need to be downloaded to a browser environment to run. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin.

[0063] The image generation method provided in this application will be described in conjunction with exemplary applications and implementations of the terminals provided in the embodiments of this application.

[0064] The image generation method provided in the embodiments of this application will be described below. As mentioned above, the electronic device implementing the image generation method of the embodiments of this application can be a terminal, a server, or a combination of both. Therefore, the executing entity of each step will not be described again below.

[0065] See Figure 3A , Figure 3A This is a flowchart illustrating the image generation method provided in the embodiments of this application, which will be combined with... Figure 3A The steps shown are explained below. Figure 3A The main component of the process is the terminal.

[0066] In step 101, a first virtual scene image that conforms to the first content is generated based on the first prompt word representing the first content.

[0067] In some embodiments, see Figure 3B , Figure 3A Step 101 shown can be implemented through steps 1011 to 1012, as described below. Figure 3B Please provide an explanation.

[0068] In step 1011, the first prompt word is processed by text encoding to obtain the first text encoding.

[0069] As an example, for text encoding of the first prompt, character encoding can be used to map each character in the first prompt, such as mapping to ASCII encoding, UTF-8 encoding, etc., and concatenating the encodings of each character in the order of characters in the first prompt. The result obtained is the first text encoding. Or the first prompt can be text-encoded using word embedding, such as using Word2Vec, GloVe, etc., to convert the first prompt into a low-dimensional vector, which is also the first text encoding.

[0070] In step 1012, denoising processing is performed on the noisy image based on the first text encoding to obtain a first virtual scene image that conforms to the first content.

[0071] In some embodiments, step 1012 can be implemented through the following technical solution: The nth denoising network in N cascaded denoising networks is used to perform denoising processing on the input of the nth denoising network, and the nth denoising result output by the nth denoising network is transmitted to the (n + 1)th denoising network to continue denoising processing, obtaining the (n + 1)th denoising result corresponding to the (n + 1)th denoising network; The Nth denoising result output by the Nth denoising network is used as the first virtual scene image; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the nth denoising network is the noisy image and the first text encoding. When n takes the value of 2 ≤ n < N, the input of the nth denoising network is the (n - 1)th denoising result output by the (n - 1)th denoising network and the first text encoding.

[0072] Through the embodiments of the present application, the denoising process is to make the noisy image gradually fit the first content expressed by the first text encoding. Specifically, it is implemented through N cascaded denoising networks: The first denoising network receives the noisy image and the first text encoding, and after denoising, transmits it to the second denoising network; The second to the (N - 1)th denoising networks all receive the denoising result of the previous denoising network and the first text encoding, and continuously perform denoising processing; Finally, the denoising result output by the Nth denoising network is the first virtual scene image that conforms to the first text encoding, that is, the first content. Each level of denoising network introduces the first text encoding, which can ensure that the denoising process always focuses on the first content expressed by the first text encoding, making the finally obtained first virtual scene image conform to the first content.

[0073] In this embodiment, the first prompt word is text-encoded, transforming the text-based prompt word into a computer-understandable first text code, laying a semantic foundation for subsequent image generation. Next, guided by the first text code, targeted denoising processing is performed on the noisy image. While removing noise, a first virtual scene image conforming to the first content is gradually constructed based on the first text code. This ensures a high degree of consistency between the first virtual scene image and the first content corresponding to the first text code. Through the collaborative processing of denoising the noisy image using the first text code, accurate generation from the first prompt word to the first virtual scene image is achieved.

[0074] In some embodiments, Figure 3A The first virtual scene image in step 101 is generated by calling a first image generation model. The training steps of the first image generation model are as follows: Initialization parameters are added to the target network layer of the diffusion model to obtain the first image generation model. First image samples belonging to the same style are obtained. Keeping the original parameters in the first image generation model unchanged, transfer learning is performed on the first image generation model based on the first image samples to update the initialization parameters of the first image generation model, resulting in an updated first image generation model.

[0075] As an example, the updated first image generation model is used to generate images conforming to the stated style. Here, style refers to the unique and stable overall appearance of an image in terms of form, representation, or features; that is, images share obvious similarities or identical features or content. First, trainable LoRA initialization parameters are introduced into the attention mechanism module of the target network layer of the diffusion model, such as the Transformer architecture, to obtain the first image generation model. These parameters exist in the form of low-rank matrices, serving as additional adapters for the first image generation model parameters. This design maintains the inconvenience of the original parameters of the first image generation model while constructing a lightweight first image generation model by introducing a small number of trainable parameters. Subsequently, first image samples belonging to the same style are collected as benchmark data for style feature learning. In the transfer learning stage, first image samples of the same style are input into the first image generation model. The difference between the generated result and the target style image is quantified by constructing a loss function. Then, when backpropagating the gradient based on the loss value, parameter update constraints are set, updating only the LoRA parameters without changing the diffusion model weights, allowing the first image generation model to gradually capture the statistical features of the target style. This parameter isolation strategy avoids the catastrophic forgetting problem, ensuring that the first image generation model trained retains its original semantic understanding ability while adapting to different styles.

[0076] Through the embodiments of this application, trainable initialization parameters are introduced into the target network layer of the diffusion model, resulting in a first image generation model that constructs a lightweight adaptive architecture that does not rely on full parameter tuning. By keeping the original parameters of the first image generation model unchanged and performing transfer learning only on the initialization parameters, the original cross-modal understanding capability of the first image generation model is preserved, while accurately capturing the distribution of visual features consistent with the style. This optimization strategy avoids the catastrophic forgetting problem in traditional fine-tuning, enabling the first image generation model to generate images with high style consistency while maintaining the accuracy of text-to-image mapping.

[0077] See also Figure 3A In step 102, a binary mask is generated based on the redrawing area of ​​the first virtual scene image.

[0078] In some embodiments, see Figure 3C , Figure 3A Step 102 shown can be implemented through steps 1021 to 1024, as described below. Figure 3C Please provide an explanation.

[0079] In step 1021, an image to be filled with the same size as the first virtual scene image is obtained.

[0080] In step 1022, when the pixel at the pixel position in the first virtual scene image is in the redrawing area, the pixel value at the pixel position in the image to be filled is set to the first pixel value.

[0081] In step 1023, when the pixel at the pixel position in the first virtual scene image is not in the redrawing area, the pixel value at the pixel position in the image to be filled is set to the second pixel value.

[0082] In step 1024, the image to be filled, after pixel value settings, is used as the binary mask.

[0083] As an example, when further processing the first virtual scene image, a mask is needed to precisely control the range of the processed first virtual scene image. Therefore, it is necessary to determine an image to be filled with the same size as the first virtual scene image. Here, steps 1022 and 1023 are performed on each pixel position in the image to be filled. When a pixel at position A in the first virtual scene image is in a redrawing area, the pixel value at that position in the image to be filled is set to a first pixel value. The first pixel value can be 1, meaning that the pixel corresponding to the pixel position with a pixel value of 1 in the image to be filled is in a redrawing area and needs to be redrawn. For example, if pixel A at position A in the first virtual scene image is in a redrawing area, then the pixel value at position A in the image to be filled is set to the first pixel value, which is 1. When a pixel at position B in the first virtual scene image is not in the redraw area, the pixel value at that position in the image to be filled is set to the second pixel value. The second pixel value can be 0, meaning that the pixel at position B in the image to be filled is not in the redraw area and does not need to be redrawn in subsequent processing. For example, if pixel b at position B in the first virtual scene image is not in the redraw area, the pixel value at position B in the image to be filled is set to the second pixel value, which is 0. Furthermore, the first and second pixel values ​​can also be RGB values ​​of different colors, as long as the RGB values ​​corresponding to the first and second pixel values ​​are different. Since the pixel value at each position in the image to be filled can only be either the first or second pixel value, and the image to be filled is used as a mask for the first virtual scene image, the image to be filled after pixel value setting is actually a binary mask.

[0084] In this embodiment, the pixel values ​​of the image to be filled corresponding to the pixels in the redrawn area of ​​the first virtual scene image are set as first pixel values, and the pixel values ​​of the image to be filled corresponding to the pixels in the remaining pixels of the first virtual scene image (excluding the redrawn area) are set as second pixel values. Using the image to be filled with these pixel values ​​as a binary mask, the redrawn area of ​​the first virtual scene image can be accurately defined, achieving efficient definition of the processing range of the first virtual scene image. Furthermore, the size of the binary mask perfectly matches the first virtual scene image, avoiding region misalignment. Binarization processing makes the mask structure clear, effectively reducing invalid calculations and improving the accuracy and efficiency of processing in the redrawn area.

[0085] See also Figure 3A In step 103, based on the second cue word representing the image difference and the binary mask, the first virtual scene image is redrawn to obtain the second virtual scene image.

[0086] As an example, the difference between the second virtual scene image and the first virtual scene image conforms to the image difference description, and the redrawing area corresponding to the image redrawing process is the same as the redrawing area corresponding to the binary mask. Image difference refers to the visual or data-level difference between the second virtual scene image and the first virtual scene image, typically manifested as changes in local pixels, colors, textures, or structures. Since the second virtual scene image is obtained by redrawing the redrawing area of ​​the first virtual scene image, the image difference between the second virtual scene image and the first virtual scene image is relatively small.

[0087] In some embodiments, step 103 can be implemented by the following technical solution: Image encoding processing is performed on the first virtual scene image to obtain a first image code, and image encoding processing is performed on the binary mask to obtain a second image code. Text encoding processing is performed on the second prompt word to obtain a second text code. The input of the m-th denoising network in an M-level cascaded denoising network is denoised, and the m-th denoising result output by the m-th denoising network is transmitted to the (m+1)-th denoising network for further denoising processing, resulting in the (m+1)-th denoising result corresponding to the (m+1)-th denoising network. The second virtual scene image is generated based on the M-th denoising result corresponding to the M-th denoising network.

[0088] As an example, m is an integer variable that starts from 1 and increases incrementally. The value range of m is 1 ≤ m < M. When m takes the value of 1, the input of the m-th denoising network is the first image encoding, the second text encoding, and the second image encoding. When m takes the value of 2 ≤ m < M, the input of the m-th denoising network is the (m - 1)-th denoising result output by the (m - 1)-th denoising network, the second text encoding, and the second image encoding. To ensure that the second virtual scene image generated during the image redrawing in the redrawing area of the first virtual scene image conforms to the second prompt word representing the image difference, it is necessary to perform denoising processing on the first virtual scene image in combination with the second prompt word. The binary mask is used to determine the redrawing area of the first virtual scene image. To enable the computer to understand the second prompt word, the first virtual scene image, and the binary mask, it is necessary to perform encoding processing on the second prompt word, the first virtual scene image, and the binary mask: For text encoding of the second prompt word, character encoding can be used. Each character in the second prompt word is mapped, for example, to ASCII encoding, UTF-8 encoding, etc. The encodings of each character are concatenated in the order of the characters in the second prompt word, and the resulting result is the second text encoding. Or use the method of word embedding to perform text encoding on the second prompt word, such as using Word2Vec, GloVe, etc., to convert the second prompt word into a low-dimensional vector, which is also the second text encoding; Since both the first virtual scene image and the binary mask are images, the encoding processing steps are similar. Specifically: First, preprocess the first virtual scene image or the binary mask, including color space conversion, such as converting from RGB to YUV, noise reduction, and normalization operations. The preprocessed result is transformed through spatial domain or frequency domain transformation, such as discrete cosine transform DCT or wavelet transform, to convert the pixel data into a coefficient representation that is more conducive to compression. The transformed coefficients are quantized, and the data volume is reduced by discarding secondary information such as high-frequency details. Entropy coding, such as Huffman coding or arithmetic coding, is used to perform lossless compression on the quantized data. Finally, the encoded data is encapsulated in a specific format, such as JPEG, PNG, or H.264 bitstream, to obtain the first image encoding of the first virtual scene image or the second image encoding of the binary mask.

[0089] Through the embodiments of this application, the first image encoding retains the structural information of the original scene image, that is, the first virtual scene image. The second image encoding can accurately locate the redrawing area, while the second text encoding introduces semantic guidance to represent the image difference. Through the processing of the cascaded denoising network, the first image encoding, the second image encoding, and the second text encoding can significantly improve the redrawing accuracy and global coordination of the generated second virtual scene image, and can perform high-quality redrawing operations on the redrawing area of the first virtual scene image.

[0090] In some embodiments, refer to Figure 3D , inFigure 3A Before step 103 shown, step 104 can also be performed to determine semantic tags and... Figure 3A Step 103 shown is updated to Figure 3D Step 105 shown below, in conjunction with Figure 3D Please provide an explanation.

[0091] In step 104, semantic segmentation is performed on the first virtual scene image to obtain a semantic label map with the same size as the first virtual scene image.

[0092] As an example, semantic segmentation of the first virtual scene image and generation of a semantic label map of the same size mainly includes the following steps: First, feature extraction: Deep features of the first virtual scene image are extracted using a convolutional neural network to generate a multi-scale feature map containing low-level details such as edges and textures, and high-level semantics such as object categories and scene structures. Second, semantic segmentation decoding: Features of different scales in the multi-scale feature map are fused using a decoder, and spatial resolution is restored through progressive upsampling. The preliminary category probability distribution of each pixel is output to obtain a probability map. Third, label mapping: An argmax operation is performed on the preliminary category probability distribution of each pixel, that is, the maximum preliminary category probability distribution is obtained from each preliminary category probability distribution. The maximum preliminary category probability distribution is determined as the final category label of each pixel. Finally, the final category labels of each pixel are combined according to the position of each pixel in the first virtual scene image to generate a semantic label map of the same size as the first virtual scene image.

[0093] In step 105, based on the second cue word representing the image difference, the binary mask, and the semantic tag map, the first virtual scene image is redrawn to obtain the second virtual scene image.

[0094] As an example, step 105 can be implemented by the following technical solution: perform image encoding processing on the first virtual scene image to obtain a first image encoding, perform image encoding processing on the binary mask to obtain a second image encoding, and perform image encoding processing on the semantic label map to obtain a third image encoding; perform text encoding processing on the second prompt word to obtain a second text encoding; denoise the input of the p-th denoising network through the p-th denoising network in P cascaded denoising networks, and transmit the p-th denoising result output by the p-th denoising network to the p + 1-th denoising network to continue denoising processing to obtain the p + 1-th denoising result corresponding to the p + 1-th denoising network; generate the second virtual scene image based on the P-th denoising result corresponding to the P-th denoising network; where p is an integer variable starting from 1 and increasing, and the value range of p is 1 ≤ p < P. When p takes the value of 1, the input of the p-th denoising network is the first image encoding, the second text encoding, the second image encoding, and the third image encoding. When p takes the value of 2 ≤ p < P, the input of the p-th denoising network is the p - 1-th denoising result output by the p - 1-th denoising network, the second text encoding, the second image encoding, and the third image encoding.

[0095] As an example, to ensure that the second virtual scene image generated when redrawing the redrawn area of ​​the first virtual scene image conforms to the second prompt word representing the image difference, and to ensure that the content generated in the redrawn area is highly consistent with the semantic logic of the first virtual scene image, it is necessary to perform denoising processing on the first virtual scene image by combining the second prompt word and the semantic label map. The binary mask is used to determine the redrawn area of ​​the first virtual scene image, and the semantic label map is obtained by semantic segmentation of the first virtual scene image. Therefore, it can accurately generate content corresponding to the semantic label map when redrawing the redrawn area. To enable the computer to understand the second prompt word, the first virtual scene image, the binary mask, and the semantic label map, encoding processing is required for these components: text encoding of the second prompt word can be performed using character encoding, mapping each character in the second prompt word to, for example, ASCII encoding, UTF-8 encoding, etc., and concatenating the encodings of each character according to the order of the characters in the second prompt word. The result is the second text encoding. Alternatively, word embedding can be used to encode the second prompt word, such as using Word2Vec or GloVe, to convert the second prompt word into a low-dimensional vector, i.e., the second text encoding. Since the first virtual scene image, binary mask, and semantic label image are all images, the encoding process is similar, specifically: First, preprocess the first virtual scene image, binary mask, or semantic label image, including color space conversion (e.g., from RGB to YUV), noise reduction, and normalization. The preprocessed result is then transformed using spatial or frequency domain transformations, such as Discrete Cosine Transform (DCT) or wavelet transform, to convert the pixel data into a more easily compressible coefficient representation. The transformed coefficients are then quantized, reducing the data volume by discarding secondary information such as high-frequency details. Lossless compression of the quantized data is performed using entropy coding, such as Huffman coding or arithmetic coding. Finally, the encoded data is encapsulated in a specific format, such as JPEG, PNG, or H.264, to obtain the first image encoding of the first virtual scene image, the second image encoding of the binary mask, or the third image encoding of the semantic label image.

[0096] In this embodiment, the first image encoding preserves the structural information of the original scene image, i.e., the first virtual scene image; the second image encoding accurately locates the redrawing region; the second text encoding introduces semantic guidance to represent image differences; and the third image encoding precisely restricts the redrawing content of the redrawing region, ensuring that the redrawing content conforms to the label restrictions in the semantic label graph. The first image encoding, second image encoding, second text encoding, and third image encoding, processed by a cascaded denoising network, significantly improve the redrawing accuracy and global consistency of the generated second virtual scene image. This enables high-quality redrawing operations on the redrawing region of the first virtual scene image and ensures that the redrawing content conforms to the restrictions of the semantic label graph.

[0097] This application's embodiments introduce a semantic tagging graph, enabling refined semantic control over the redrawing process of the first virtual scene image, ensuring that the content generated in the redrawing area is highly consistent with the semantic logic of the first virtual scene image. Specifically, the semantic tagging graph provides contextual constraints for the redrawing area, and the second virtual scene image generated in conjunction with the textual guidance of the second prompt word significantly improves the rationality and accuracy of the generated content when redrawing the redrawing area of ​​the first virtual scene image.

[0098] In some embodiments, Figure 3D The semantic label map in step 104 shown is generated by calling the image segmentation model. The training steps of the image segmentation model are as follows: obtain the second image sample corresponding to the image segmentation task; perform semantic segmentation on the second image sample through the image segmentation model to obtain the predicted semantic label map; determine the loss function based on the actual semantic label map corresponding to the second image sample and the predicted semantic label map, and update the image segmentation model based on the loss function to obtain the updated image segmentation model.

[0099] In this embodiment, a loss function, such as cross-entropy loss, is calculated based on the difference between the actual semantic label map and the predicted semantic label map corresponding to the second image sample. The parameters of the image segmentation model are then dynamically adjusted through backpropagation, enabling the model to gradually learn more accurate semantic segmentation capabilities. This effectively reduces semantic segmentation errors in complex scenes. By implementing an adaptive optimization strategy based on the second image sample, the accuracy and generalization ability of the image segmentation model's semantic segmentation are significantly improved.

[0100] Through the embodiments of this application, a first virtual scene image conforming to the first content is generated based on a first prompt word representing the first content; this ensures that the content represented by the first virtual scene image is consistent with the content represented by the prompt word, and allows for the batch generation of first virtual scene images with consistent content. A binary mask is generated based on the redrawing area of ​​the first virtual scene image; based on the second prompt word representing the image difference and the binary mask, the first virtual scene image is redrawn to obtain a second virtual scene image; wherein the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawing area corresponding to the image redrawing process is the same as the redrawing area corresponding to the binary mask. The generation of the binary mask clearly defines the range of the redrawing area, avoiding erroneous modification of irrelevant areas; combined with the redrawing process of the second prompt word and the binary mask, it can accurately realize the difference changes between specified virtual scene images, and strictly limit the redrawing operation to only the area corresponding to the binary mask. This process effectively ensures that the difference between the redrawn second virtual scene image and the first virtual scene image meets expectations, while maintaining the integrity of non-redrawing areas, improving the accuracy and efficiency of virtual scene image adjustments.

[0101] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0102] In generating scene images for spot-the-difference games, related technologies primarily employ manual image selection or drawing to produce the original level image. Difference points are then manually selected from this original image (the first virtual scene image). The process of generating a difference point involves removing, adding, or modifying an element on the original image. After modifying a certain number of differing elements, a comparison image (the second virtual scene image) is obtained. Once the level is successfully approved, it is exported through a corresponding level export system. This system compares the pixel differences between the original and comparison images, identifying areas with concentrated pixel differences as difference points, ultimately generating a level configuration directly usable in the game. See also... Figure 4 , Figure 4 This is a traditional checkpoint production flow chart provided in the embodiments of this application.

[0103] In related technologies, ensuring stylistic consistency in game scene images is difficult, as existing game scene images vary in style. Furthermore, the cost of finding, drawing, and modifying game scene images is too high, requiring manual screening and consideration of whether the selected images are prone to generating differences. Drawing game scene images is inefficient and requires matching a large number of different game levels. Modifying game scenes requires manual decision-making regarding differences, and determining the content and modifications of these differences is time-consuming. In addition, during the acceptance phase of game scene images, if acceptance fails, the images need to be modified again, requiring repeated adjustments to the differences. Moreover, level export systems automatically segment differences based on the density of pixel differences between the original and modified comparison images, which can lead to the misidentification of multiple differences as a single point when the pixel differences between the original and comparison images are too dense. Based on this, this application proposes a method that can automatically generate game scene images corresponding to levels in different games. It integrates text-based image generation and local redrawing techniques to achieve automated generation of game scene images, solving the bottlenecks of high cost and low efficiency associated with traditional manual drawing. This system can generate images with the same style in batches based on a training set, and can intelligently determine differences, including supporting parameterized automatic selection of differences or manual selection of differences, thus realizing an industrialized production process. Through the embodiments of this application, the style of game scene images corresponding to game levels can be kept uniform. By collecting a large number of game scene images with similar styles as a training set, a model can be trained to generate game scene images with a unified style in batches. Automated generation of comparison images can be achieved. Multiple comparison images can be automatically generated in batches for filtering by randomly generating differences or manually selecting difference regions, greatly reducing time costs. Furthermore, comparison images can be iteratively modified repeatedly through a redrawing algorithm. Since the differences in the comparison images are generated by given parameters, the problem of inaccurate difference identification when comparing the exported comparison image with the original image, which occurs after modifying the original image in traditional processes, is avoided.

[0104] See Figure 5 , Figure 5This is a flowchart of the technical solution provided in the embodiments of this application. First, stylized original images are generated in batches: a training set is prepared by collecting 50-200 images with consistent style and uniform resolution (first image samples). The LoRA fine-tuning model is selected, and a pre-trained stable diffusion model (first image generation model) is loaded. Fine-tuning is performed by configuring hyperparameters (initialization parameters) such as training set path, resolution, batch size, learning rate, and total number of iterations. Here, LoRA is a parameter fine-tuning technique applied to large models. The training hyperparameters are values ​​set before training begins. These parameters are adjusted by detecting training loss or checking output results during training to optimize the training results. After fine-tuning, the output weight file is loaded into the pre-trained stable diffusion model. New labels or prompt words (first prompt words) are used as input to verify whether the images generated in batches by the pre-trained stable diffusion model meet expectations.

[0105] The second step is region redrawing. This involves using a binary mask to outline the redrawing area on the original image (the first virtual scene image). This area can be randomly selected or specified. The original image and the binary mask are then input into the standard stable diffusion image inpainting pipeline. Simultaneously, style and detail requirements are added in the prompt (the second prompt). Parameters such as inference steps, guidance scale, and random seed are adjusted to generate a comparison image corresponding to the prompt. Here, after selecting the redrawing area on the original image, the system automatically generates the corresponding binary mask. The size of the binary mask is the same as the original image; 0 indicates areas that are preserved and not redrawn, and 1 indicates areas that need to be redrawn. The pipeline for image inpainting provided by the stable diffusion model is called, inputting the original image, binary mask, prompt (the second prompt), inference steps, guidance scale, and random seed to generate a comparison image (the second virtual scene image). The number of inference steps determines the number of denoising steps required for the stable diffusion model to generate comparison images from random noise, thus affecting the fineness of the generated comparison images. This is typically fine-tuned within an optimal range. The guiding scale determines the stable diffusion model's dependence on cue words. The random seed controls the randomness of the generated comparison images; different seeds produce inconsistent results, used to generate a large number of comparison images for selection. When other parameters are consistent, setting the same seed yields consistent results, allowing comparison of the impact of other parameters on the generated comparison images. During the development phase, each style of model has already obtained a set of optimal parameter values ​​through repeated debugging, and these parameters do not require significant modification in actual operation.

[0106] If you need to accurately draw a desired difference point, such as accurately drawing a tree in the redrawn area, you need to use a pre-trained image segmentation model to generate a semantic label map of the same size as the original image. Then, modify the semantic label map to RGB or grayscale format, add a binary mask, and redraw the original image. Figure 1The input integrates a stable diffusion image inpainting pipeline based on a neural network-based conditional generative model. The semantic label map is an RGB image that reflects the division of the main subject in the original image and is composed of color blocks. The neural network-based conditional generative model defines a mapping table between color codes and actual objects. For example, 04FA07 represents grass, and 06E6E6 represents the sky. In the semantic label map, the upper half is the solid color 04FA07, and the lower half is the solid color 06E6E6. This semantic label map can generate an image where the upper half is the sky and the lower half is grass. By selecting a redrawing region on the original image and setting its label, a semantic label map of the same size as the original image is generated. The color of the redrawing region is the color code of its corresponding semantic label. The semantic label map is passed as a parameter to the pipeline, along with a concise text prompt (second prompt) describing the content to appear in the redrawing region. Then, the appropriate control strength, inference steps, and guidance scale of the conditional generation model of the neural network are set. The pipeline will then generate a comparison map (second virtual scene image) that conforms to the specified category and style in the redrawing region based on the semantic label map and the prompt (second prompt) while keeping the rest of the original image unchanged.

[0107] The difference between generating a comparison image using a semantic label map and the ordinary method is that a semantic label map needs to be automatically generated from the original image. The semantic label map and the semantic influence strength are then fed into the pipeline. Semantic labels are set while the redrawn area is selected, and the semantic influence strength is adjusted.

[0108] The difficulty of the "spot the difference" game is dynamically controlled by modifying the parameters generated from the comparison images. The number, position, size, and extent of redrawn areas all affect the level's difficulty. Specifically, the number of redrawn areas and difficulty are not strictly positively or negatively correlated; too few or too many areas will make the game easier. During batch generation, three ranges are typically selected: small, medium, and large. The number of redrawn areas is then randomly generated within a specified range. The difficulty control of the redrawn area position is mainly reflected in the manual selection of the area. Larger redrawn areas and greater redrawn extents also make the game easier.

[0109] This application's embodiments improve production efficiency. Traditionally, creating a level by finding differences requires a series of processes: image sourcing, image modification, acceptance testing, and level generation. If the planner requests adjustments, this process becomes repetitive and time-consuming. The method provided in this application's embodiments can handle all modification requests for multiple images simultaneously, and can be entirely handled by the level planner. It essentially eliminates the time artists spend sourcing and modifying images, requiring only the generation of a large number of level images for the planner to select from. Furthermore, it ensures consistency in level style, generating different level styles through different training sets and compiling them into a style library for easy selection. This avoids inconsistencies caused by varying image styles or different artists. Traditional level creation is limited by the artist's individual experience, often reusing similar differences. This application's embodiments, however, can generate diverse differences. Moreover, the traditional method, with multiple artists each responsible for a portion, can lead to inefficiency during acceptance testing. This application's embodiments allow all levels to be centrally managed by the level planner.

[0110] The following description continues to illustrate the exemplary structure of the image generation apparatus 455 provided in the embodiments of this application as a software module. In some embodiments, see [link to relevant documentation]. Figure 2 The software module stored in the image generation device 455 of the memory 450 may include:

[0111] The first virtual scene image generation module 4551 is used to generate a first virtual scene image that conforms to the first content based on a first prompt word representing the first content;

[0112] Binary mask generation module 4552 is used to generate a binary mask based on the redrawing area of ​​the first virtual scene image;

[0113] The second virtual scene image generation module 4553 is used to perform image redrawing processing on the first virtual scene image based on the second prompt word representing the image difference and the binary mask to obtain the second virtual scene image; wherein the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawing area corresponding to the image redrawing processing is the same as the redrawing area corresponding to the binary mask.

[0114] In some embodiments, the image generation apparatus further includes a first image generation model training module 4555, configured to add initialization parameters to the target network layer in the diffusion model to obtain a first image generation model; obtain a first image sample belonging to the same style; keep the original parameters in the first image generation model unchanged, and perform transfer learning on the first image generation model based on the first image sample to update the initialization parameters of the first image generation model, thereby obtaining an updated first image generation model, wherein the updated first image generation model is used to generate images that conform to the style.

[0115] In some embodiments, the first virtual scene image generation module 4551 is further configured to perform text encoding processing on the first prompt word to obtain a first text encoding; perform denoising processing on the noise image based on the first text encoding to obtain a first virtual scene image that conforms to the first content.

[0116] In some embodiments, the first virtual scene image generation module 4551 is further configured to perform denoising processing on the input of the nth denoising network through the nth denoising network in N cascaded denoising networks, and transmit the nth denoising result output by the nth denoising network to the (n + 1)th denoising network to continue denoising processing to obtain the (n + 1)th denoising result corresponding to the (n + 1)th denoising network; use the Nth denoising result output by the Nth denoising network as the first virtual scene image; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n is 1, the input of the nth denoising network is the noise image and the first text encoding. When 2 ≤ n < N, the input of the nth denoising network is the (n - 1)th denoising result output by the (n - 1)th denoising network and the first text encoding.

[0117] In some embodiments, the binary mask generation module 4552 is further configured to obtain a to-be-filled image having the same size as the first virtual scene image; for each pixel position in the to-be-filled image: when the pixel point at the pixel position in the first virtual scene image is in the redrawing area, set the pixel value at the pixel position in the to-be-filled image to a first pixel value; when the pixel point at the pixel position in the first virtual scene image is not in the redrawing area, set the pixel value at the pixel position in the to-be-filled image to a second pixel value; use the to-be-filled image with the pixel values set as the binary mask.

[0118] In some embodiments, the second virtual scene image generation module 4553 is further configured to perform image encoding processing on the first virtual scene image to obtain a first image encoding, and perform image encoding processing on the binary mask to obtain a second image encoding; perform text encoding processing on the second prompt to obtain a second text encoding; perform denoising processing on the input of the m-th denoising network through the m-th denoising network in M cascaded denoising networks, and transmit the m-th denoising result output by the m-th denoising network to the (m + 1)-th denoising network to continue denoising processing to obtain the (m + 1)-th denoising result corresponding to the (m + 1)-th denoising network; generate the second virtual scene image based on the M-th denoising result corresponding to the M-th denoising network; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m < M. When m takes the value of 1, the input of the m-th denoising network is the first image encoding, the second text encoding, and the second image encoding. When m takes the value of 2 ≤ m < M, the input of the m-th denoising network is the (m - 1)-th denoising result output by the (m - 1)-th denoising network, the second text encoding, and the second image encoding.

[0119] In some embodiments, the second virtual scene image generation module 4553 is further configured to perform semantic segmentation on the first virtual scene image to obtain a semantic label map having the same size as the first virtual scene image; perform image redrawing processing on the first virtual scene image based on the second prompt characterizing image differences and the binary mask to obtain a second virtual scene image, including: performing image redrawing processing on the first virtual scene image based on the second prompt characterizing image differences, the binary mask, and the semantic label map to obtain a second virtual scene image.

[0120] In some embodiments, the second virtual scene image generation module 4553 is further configured to perform image encoding processing on the first virtual scene image to obtain a first image encoding, perform image encoding processing on the binary mask to obtain a second image encoding, and perform image encoding processing on the semantic label map to obtain a third image encoding; perform text encoding processing on the second prompt word to obtain a second text encoding; denoise the input of the p-th denoising network through the p-th denoising network in P cascaded denoising networks, and transmit the p-th denoising result output by the p-th denoising network to the (p + 1)-th denoising network to continue denoising processing to obtain the (p + 1)-th denoising result corresponding to the (p + 1)-th denoising network; generate the second virtual scene image based on the P-th denoising result corresponding to the P-th denoising network; where p is an integer variable starting from 1 and increasing, and the value range of p is 1 ≤ p < P. When p = 1, the input of the p-th denoising network is the first image encoding, the second text encoding, the second image encoding, and the third image encoding. When 2 ≤ p < P, the input of the p-th denoising network is the (p - 1)-th denoising result output by the (p - 1)-th denoising network, the second text encoding, the second image encoding, and the third image encoding.

[0121] In some embodiments, the image generation device further includes an image segmentation model training module 4554, configured to obtain a second image sample corresponding to an image segmentation task; perform semantic segmentation on the second image sample through the image segmentation model to obtain a predicted semantic label map; determine a loss function based on the actual semantic label map corresponding to the second image sample and the predicted semantic label map, and update the image segmentation model based on the loss function to obtain an updated image segmentation model.

[0122] An embodiment of the present application provides a computer program product, which includes a computer program or computer-executable instructions, and the computer program or computer-executable instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium, and the processor executes the computer-executable instructions, so that the electronic device executes the image generation method described above in the embodiments of the present application.

[0123] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions or a computer program are stored. When the computer-executable instructions or the computer program are executed by a processor, the processor will be caused to execute the image generation method provided in the embodiments of the present application.

[0124] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0125] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0126] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0127] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0128] In summary, this application's embodiments generate a first virtual scene image that conforms to the first content based on a first prompt word representing the first content; ensuring that the content represented by the first virtual scene image is consistent with the content required by the prompt word, and enabling the batch generation of first virtual scene images with consistent content. A binary mask is generated based on the redrawing area of ​​the first virtual scene image; based on the second prompt word representing image differences and the binary mask, the first virtual scene image undergoes image redrawing processing to obtain a second virtual scene image; wherein the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawing area corresponding to the image redrawing processing is the same as the redrawing area corresponding to the binary mask. The generation of the binary mask clearly defines the range of the redrawing area, avoiding accidental modification of irrelevant areas; combined with the redrawing processing of the second prompt word and the binary mask, it can accurately realize the difference changes between specified virtual scene images, and strictly limit the redrawing operation to only the area corresponding to the binary mask. This process effectively ensures that the differences between the redrawn second virtual scene image and the first virtual scene image meet expectations, while maintaining the integrity of non-redrawn areas and improving the accuracy and efficiency of adjusting the virtual scene image.

[0129] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, The method includes: Based on the first prompt word representing the first content, a first virtual scene image that conforms to the first content is generated; A binary mask is generated based on the redrawn area of ​​the first virtual scene image; Extract multi-scale feature maps from the first virtual scene image; By fusing features from different scales in the multi-scale feature map, a preliminary class probability distribution for each pixel is obtained; Obtain the maximum preliminary category probability distribution in each of the preliminary category probability distributions, and determine the maximum preliminary category probability distribution as the final category label for each pixel; The final category label of each pixel is combined according to the position of each pixel in the first virtual scene image to generate a semantic label map with the same size as the first virtual scene image; Based on the second cue word representing the image difference, the binary mask, and the semantic tag map, the first virtual scene image is redrawn to obtain the second virtual scene image. Wherein, the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawing area corresponding to the image redrawing process is the same as the redrawing area corresponding to the binary mask.

2. The method according to claim 1, characterized in that, The first virtual scene image is generated by calling a first image generation model, and the method further includes: Initialization parameters are added to the target network layer in the diffusion model to obtain the first image generation model; Obtain the first image sample belonging to the same style; Keeping the original parameters in the first image generation model unchanged, the first image generation model is subjected to transfer learning based on the first image sample to update the initialization parameters of the first image generation model, thereby obtaining an updated first image generation model, wherein the updated first image generation model is used to generate images that conform to the style.

3. The method according to claim 1, characterized in that, The step of generating a first virtual scene image that matches the first content based on a first prompt word representing the first content includes: The first prompt word is processed by text encoding to obtain the first text code; The noisy image is denoised based on the first text encoding to obtain a first virtual scene image that conforms to the first content.

4. The method according to claim 3, characterized in that, The step of performing denoising processing on the noisy image based on the first text encoding to obtain a first virtual scene image that conforms to the first content includes: The input of the nth denoising network is denoised through the nth denoising network in the N cascaded denoising networks, and the nth denoising result output by the nth denoising network is transmitted to the (n+1)th denoising network for further denoising, so as to obtain the (n+1)th denoising result corresponding to the (n+1)th denoising network. The Nth denoising result output by the Nth denoising network is used as the first virtual scene image; Where n is an integer variable that increments from 1, and the range of n is... n When n is 1, the input to the nth denoising network is the noisy image and the first text encoding; when n is 2... n At that time, the input of the nth denoising network is the (n-1)th denoising result output by the (n-1)th denoising network and the first text encoding.

5. The method according to claim 1, characterized in that, The step of generating a binary mask based on the redrawn area of ​​the first virtual scene image includes: Obtain an image to be filled that has the same size as the first virtual scene image; For each pixel position in the image to be filled: When the pixel at the specified pixel position in the first virtual scene image is located in the redrawing area, the pixel value at the specified pixel position in the image to be filled is set to the first pixel value. When the pixel at the specified pixel position in the first virtual scene image is not in the redrawing area, the pixel value at the specified pixel position in the image to be filled is set to the second pixel value; The image to be filled, after pixel value settings, is used as the binary mask.

6. The method according to claim 1, characterized in that, The second cue word based on the image difference representation, the binary mask, and the semantic label map are used to perform image redrawing processing on the first virtual scene image to obtain the second virtual scene image, including: The first virtual scene image is processed by image encoding to obtain a first image code; the binary mask is processed by image encoding to obtain a second image code; and the semantic tag map is processed by image encoding to obtain a third image code. The second prompt word is processed by text encoding to obtain the second text code; The input of the p-th denoising network is denoised through the p-th denoising network in the P cascaded denoising networks, and the p-th denoising result output by the p-th denoising network is transmitted to the p+1 denoising network for further denoising, so as to obtain the p+1 denoising result corresponding to the p+1 denoising network. The second virtual scene image is generated based on the Pth denoising result of the corresponding Pth denoising network; Where p is an integer variable whose value starts from 1 and increments, and the range of p is... When p is 1, the input to the p-th denoising network is the first image code, the second text code, the second image code, and the third image code. When p is 2... At that time, the input of the p-th denoising network is the p-1 denoising result output by the p-1 denoising network, the second text encoding, the second image encoding, and the third image encoding.

7. The method according to claim 1, characterized in that, The semantic label map is generated by calling an image segmentation model, and the method further includes: Obtain the second image sample for the corresponding image segmentation task; The image segmentation model is used to perform semantic segmentation on the second image sample to obtain a predicted semantic label map; Based on the actual semantic label map corresponding to the second image sample and the predicted semantic label map, a loss function is determined, and the image segmentation model is updated based on the loss function to obtain the updated image segmentation model.

8. An image generation apparatus, characterized in that, The device includes: The first virtual scene image generation module is used to generate a first virtual scene image that conforms to the first content based on a first prompt word representing the first content. The binary mask generation module is used to generate a binary mask based on the redrawing area of ​​the first virtual scene image; The second virtual scene image generation module is used to extract a multi-scale feature map of the first virtual scene image; fuse features of different scales in the multi-scale feature map to obtain a preliminary category probability distribution for each pixel; obtain the maximum preliminary category probability distribution in each preliminary category probability distribution, and determine the maximum preliminary category probability distribution as the final category label for each pixel; combine the final category labels of each pixel according to the position of each pixel in the first virtual scene image to generate a semantic label map with the same size as the first virtual scene image; and perform image redrawing processing on the first virtual scene image based on the second cue word representing image differences, the binary mask, and the semantic label map to obtain a second virtual scene image; wherein the difference between the second virtual scene image and the first virtual scene image conforms to the image difference, and the redrawing area corresponding to the image redrawing processing is the same as the redrawing area corresponding to the binary mask.

Images