Image generation method, apparatus, readable medium, device, and program product

Abstract

Embodiments of the present application relate to the technical field of computer, and disclose an image generation method and device, readable medium, equipment and program product. The method comprises: performing object recognition on a content text to obtain an object description text, and generating an object control image based on the object description text; performing object feature extraction on the object control image to obtain object features, and obtaining style guide information used for indicating an image style and content guide information used for indicating an image content element; performing text feature extraction on a target text in the content text which has a demand for a matching image to obtain text features of the target text; and performing reverse diffusion processing on a noise image based on the object features, the style guide information, the content guide information and the text features of the target text to generate a matching image corresponding to the target text. By using the embodiments of the present application, the image generation speed can be improved while ensuring the consistency of image content.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to image generation methods, apparatus, readable media, devices and program products. Background Technology

[0002] With the development of computer vision (CV) technology, image generation applications that combine computer vision technology with generative models are receiving increasing attention. In some text-based image generation scenarios, it is necessary to generate images that fit each text within the same article. Related technologies primarily utilize StoryDiffusion (a model focused on generating coherent image and video stories) to generate consistent images or videos to tell complex stories.

[0003] While this approach can control the consistency of generated images in terms of style, objects, and other image content to some extent, it consumes significant computational resources each time an image is generated, resulting in slow generation speed. Therefore, how to improve image generation speed while ensuring image content consistency is a pressing issue that needs to be addressed. Summary of the Invention

[0004] This application provides image generation methods, apparatus, readable media, devices, and program products, which can improve image generation speed while ensuring the consistency of image content.

[0005] On one hand, embodiments of this application provide an image generation method, including:

[0006] Perform object recognition on the content text to obtain object description text, and generate object control images based on the object description text;

[0007] Object features are extracted from the object control image to obtain object features, and style guidance information for indicating image style and content guidance information for indicating image content elements are obtained.

[0008] Text features are extracted from target text that requires accompanying images to obtain the text features of the target text;

[0009] Based on object features, style guidance information, content guidance information, and text features of the target text, a reverse diffusion process is performed on the noisy image to generate an image corresponding to the target text.

[0010] On one hand, embodiments of this application provide an image generation apparatus, which includes an acquisition unit, an image feature extraction unit, a text feature extraction unit, and a generation unit, wherein:

[0011] The acquisition unit is used to perform object recognition on the content text, obtain object description text, and generate object control images based on the object description text;

[0012] The image feature extraction unit is used to extract object features from the object control image, obtain object features, and acquire style guidance information for indicating image style and content guidance information for indicating image content elements.

[0013] The text feature extraction unit is used to extract text features from target text that requires accompanying images, thereby obtaining the text features of the target text.

[0014] The generation unit is used to perform reverse diffusion processing on noisy images based on object features, style guidance information, content guidance information, and text features of the target text to generate an image corresponding to the target text.

[0015] In one embodiment of this application, based on the aforementioned scheme, when the text feature extraction unit extracts text features from target text that has image requirements in the content text to obtain the text features of the target text, it can specifically perform the following: obtaining key content text in the target text; wherein, the key content text includes at least one of object description text and scene description text contained in the target text; performing feature extraction processing on the key content text to obtain the text features of the target text.

[0016] In one embodiment of this application, based on the foregoing scheme, the text feature extraction unit can also be used to: acquire a style control image for generating style guidance information; classify the style control image to obtain style description text of the style to which the style control image belongs; and concatenate the style description text and the key content text to obtain concatenated text. When performing feature extraction processing on the key content text to obtain the text features of the target text, the text feature extraction unit can specifically be used to: encode the text features of the concatenated text to obtain the text features of the target text.

[0017] In one embodiment of this application, based on the aforementioned scheme, when the image feature extraction unit obtains style guidance information for indicating image style and content guidance information for indicating image content elements, it can also specifically perform the following: obtaining a style control image and performing style feature extraction on the style control image to obtain style guidance information; obtaining a content control image and performing content feature extraction on the content control image to obtain content guidance information.

[0018] In one embodiment of this application, based on the foregoing scheme, when the image feature extraction unit obtains style guidance information for indicating image style and content guidance information for indicating image content elements, it can also specifically perform: style feature extraction on the object control image to obtain style guidance information; and content feature extraction on the object control image to obtain content guidance information.

[0019] In one embodiment of this application, based on the aforementioned scheme, when the image feature extraction unit extracts object features from the object control image to obtain object features, it can also specifically perform the following: extracting face features from the face control image included in the object control image to obtain face features; acquiring a pose control image and extracting pose features from the pose control image to obtain pose features; and obtaining object features based on the pose features and face features.

[0020] In one embodiment of this application, based on the aforementioned scheme, when the generation unit performs reverse diffusion processing on the noisy image based on object features, style guidance information, content guidance information, and text features of the target text to generate an image corresponding to the target text, it can also specifically perform the following: calling the trained image generation model to encode the noisy image to obtain an initial feature map; performing feature sampling on the initial feature map based on object features, style guidance information, content guidance information, and text features to obtain a target feature map; and performing decoding processing on the target feature map to obtain an image corresponding to the target text.

[0021] In one embodiment of this application, based on the aforementioned scheme, the trained image generation model includes a feature processing module. The feature processing module includes a downsampling network, a transition network, and an upsampling network connected in sequence. The signal output of the downsampling network and the signal input of the upsampling network are connected by a skip connection. When the generation unit performs feature sampling on the initial feature map based on object features, style guidance information, content guidance information, and text features to obtain the target feature map, it can specifically perform the following: calling the downsampling network to perform downsampling processing on the initial feature map based on style guidance information, content guidance information, and text features to obtain the downsampling feature map; calling the transition network to perform feature encoding on the downsampling feature map based on object features to obtain the transition feature map; and calling the upsampling network to perform upsampling processing on the transition feature map based on the feature map output by the signal output of the downsampling network, style guidance information, content guidance information, object features, and text features to obtain the target feature map.

[0022] In one embodiment of this application, based on the aforementioned scheme, the downsampling network includes multiple downsampling sub-networks connected in sequence, and the upsampling network includes multiple upsampling sub-networks connected in sequence, with each downsampling sub-network corresponding to each upsampling sub-network; when the generation unit calls the upsampling network and performs upsampling processing on the transition feature map based on the feature map, style guidance information, content guidance information, object features, and text features output by the signal output terminal of the downsampling network to obtain the target feature map, it can also be used to perform: for any upsampling sub-network in the upsampling network, obtain the corresponding... The specified feature map is output by the signal output terminal of the downsampling subnetwork; the specified feature map, object features, and the feature map output by the previous network connected to any upsampling subnetwork are concatenated to obtain a concatenated feature map; wherein, when any upsampling subnetwork is the first upsampling subnetwork in the upsampling network, the feature map output by the previous network is a transitional feature map; based on the concatenated feature map, style guidance information, content guidance information, and text features, the feature map output by the signal output terminal of any upsampling subnetwork is obtained; the feature map output by the signal output terminal of the last upsampling subnetwork in the upsampling network is used as the target feature map.

[0023] In one embodiment of this application, based on the aforementioned scheme, any upsampling sub-network includes a feature fusion network layer and an upsampling convolutional network layer connected in sequence. The feature fusion network layer includes a residual network and a cross-attention network connected in sequence. Based on the aforementioned scheme, when the generation unit obtains the feature map output by the signal output terminal of any upsampling sub-network based on the spliced ​​feature map, style guidance information, content guidance information, and text features, it can specifically perform the following: calling the residual network to perform residual encoding on the spliced ​​feature map to obtain a residual feature map; calling the cross-attention network to fuse the residual feature map, style guidance information, content guidance information, and text features to obtain a fused feature map; and calling the upsampling convolutional network layer to perform convolution processing on the fused feature map to obtain the feature map output by the signal output terminal of any upsampling sub-network.

[0024] On one hand, embodiments of this application provide an electronic device, which includes an input interface and an output interface, and further includes:

[0025] A processor, adapted to implement one or more instructions; and,

[0026] A computer storage medium storing one or more instructions adapted to be loaded by the processor and executed by the image generation method described above.

[0027] On one hand, embodiments of this application provide a computer-readable medium storing computer program instructions, which, when executed by a processor, are used to perform the above-described image generation method.

[0028] On one hand, embodiments of this application provide a computer program product or computer program, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium; a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, and when the computer instructions are executed by the processor, they are used to perform the above-described image generation method.

[0029] In the technical solution provided by the embodiments of this application, by generating object control images based on object description text in the content text, the image of the object in the content text can be visualized, so that the object features extracted from the object control images can represent specific objects, thereby ensuring high-precision consistency of objects contained in the subsequently generated images under the influence of object features. Simultaneously, the final image generated in this embodiment is influenced not only by text features but also by object features, style guidance information, and content guidance information; under the influence of the same object features, the same or similar style guidance information, and the same or similar content guidance information, the objects, styles, and content elements in the generated image can maintain a high degree of consistency. Thus, compared to existing solutions that require attention interaction between different images and simultaneous generation of multiple scenes, this embodiment only needs to perform interaction and fusion between object features, style guidance information, content guidance information, and text features, consuming relatively fewer computational resources and achieving faster image generation speed. Therefore, the embodiments of this application can effectively improve image generation speed while ensuring image content consistency. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of the structure of an image generation system provided in an embodiment of this application;

[0032] Figure 2 This is a schematic flowchart of an image generation method provided in an embodiment of this application;

[0033] Figure 3This is a schematic diagram illustrating a process for obtaining key content text provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of a style-controlled image processing procedure provided in an embodiment of this application;

[0035] Figure 5 This is a flowchart illustrating another image generation method provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the structure of a feature processing module provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of the structure of a trained image generation model provided in an embodiment of this application;

[0038] Figure 8 This is a schematic diagram illustrating the process of a trained image generation model generating an accompanying image, as provided in an embodiment of this application.

[0039] Figure 9 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this application;

[0040] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0044] The flowcharts shown in the accompanying diagrams are merely illustrative and do not necessarily include all content and operations, nor do they necessarily have to be executed in the described order. For example, some operations may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0045] It should also be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0046] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0047] Image generation, on the other hand, is an application area that combines computer vision technology with generative models. It involves using algorithms to create or synthesize images. In some scenarios where images are generated based on text, it is necessary to generate images that fit each text within the same article, resulting in multiple images.

[0048] In related technologies, StoryDiffusion (a model focused on generating coherent image and video stories) is primarily used to generate consistent images or videos to tell complex stories. This approach proposes Consistent Self-Attention, which establishes connections between images within a batch during image generation to maintain consistency. While this method can control the consistency of generated images in terms of style, objects, and other image content to some extent, it requires attention interactions between different images each time an image is generated, necessitating the simultaneous generation of multiple scenes. This results in high computational resource consumption and a relatively slow generation speed.

[0049] Based on this, embodiments of this application provide an image generation scheme. This scheme first identifies object description text within the content text and generates an object control image based on the object description text. Then, it extracts object features from the object control image. Additionally, it obtains style guidance information indicating the image style and content guidance information indicating image content elements. Thus, when generating an image corresponding to target text with an image requirement within the content text, it can perform reverse diffusion processing on the noisy image based on object features, style guidance information, content guidance information, and the text features of the target text to generate the corresponding image.

[0050] It is easy to see that this solution generates object control images based on object description text in the content text, which can visualize the image of the object in the content text. This allows the object features extracted from the object control images to represent specific objects, thus ensuring consistency of objects in the subsequently generated images under the influence of object features. Simultaneously, this solution uses a reverse diffusion process on noisy images based on object features, style guidance information, content guidance information, and text features of the target text. This ensures that the final generated illustrations are influenced not only by text features but also by object features, style guidance information, and content guidance information. Furthermore, the objects, styles, and content elements in the illustrations generated under the influence of the same object features, the same or similar style guidance information, and the same or similar content guidance information can maintain a high degree of consistency.

[0051] Furthermore, this solution controls the consistency of generated images in terms of content through object features, style guidance information, and content guidance information. Compared to existing solutions that require attention interactions between different images and simultaneous generation of multiple scenes, this solution only needs to perform interactive fusion between object features, style guidance information, content guidance information, and text features, consuming relatively fewer computational resources and generating images faster. Therefore, this solution can effectively improve image generation speed while ensuring image content consistency.

[0052] Based on the above image generation method, this application provides an image generation system, which can be found in [reference needed]. Figure 1 , Figure 1 The image generation system shown may include multiple terminal devices 101 and multiple servers 102, wherein a communication connection is established between any terminal device and any server. Terminal devices 101 may include any one or more of the following: sensors, smartphones, tablets, laptops, desktop computers, smart vehicles, and smart wearable devices. Terminal devices 101 may run clients for generating or displaying images, and may also run various applications (APPs), such as game clients, virtual interaction clients, multimedia playback clients, social clients, information streaming clients, etc.

[0053] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal device 101 and server 102 can communicate directly or indirectly via wired or wireless communication; this application does not impose any restrictions on this.

[0054] In one embodiment, the above image generation method can be performed solely by... Figure 1 The terminal device 101 in the image generation system described above executes the following process: the terminal device 101 identifies the object description text in the content text and generates an object control image based on the object description text; then, the terminal device 101 extracts object features from the object control image; subsequently, the terminal device 101 can acquire style guidance information for indicating the image style and content guidance information for indicating the image content elements; then, the terminal device 101 can extract the text features corresponding to the target text in the content text that has an image matching requirement; finally, the terminal device 101 can perform reverse diffusion processing on the noisy image based on the object features, style guidance information, content guidance information, and text features of the target text, thereby generating an image matching the target text.

[0055] Optionally, the terminal device 101 can also output an image corresponding to the target text. Alternatively, the above image generation method can also be implemented solely by... Figure 1 The image generation system shown is executed by server 102. The specific execution process can be found in the specific execution process of terminal device 101 mentioned above, and will not be repeated here.

[0056] In another embodiment, the above-described image generation method can be derived from... Figure 1 The image generation system shown in the diagram is executed jointly by terminal device 101 and server 102. Specifically, server 102 identifies object description text within the content text and generates an object control image based on this text. Then, server 102 extracts object features from the object control image, as well as style guidance information indicating the image style and content guidance information indicating image content elements. Before generating the image, terminal device 101 can respond to a text selection operation within the content text, obtain target text within the content text that requires an accompanying image, and send the target text to server 102. After receiving the target text, server 102 extracts its text features. Finally, server 102 performs reverse diffusion processing on the noisy image based on the object features, style guidance information, content guidance information, and the text features of the target text, thereby generating an accompanying image corresponding to the target text. Optionally, server 102 can send the accompanying image corresponding to the target text to terminal device 101, causing terminal device 101 to output the accompanying image corresponding to the target text.

[0057] It should be noted that the embodiments of this application can be applied to various scenarios, including but not limited to cloud reading, AI (Artificial Intelligence), smart home, smart transportation, intelligent driving and other application scenarios containing text and requiring the generation of images from the text, and are not limited thereto.

[0058] For example, embodiments of this application can be applied to novel reading applications to generate illustrations for various novels in the application; embodiments of this application can also be applied to smart homes to generate illustrations for furniture and appliance instruction manuals. Furthermore, embodiments of this application can be applied to image editing applications to generate images adapted to text; and embodiments of this application can also be applied to video editing applications to generate multiple consecutive images adapted to text, thereby obtaining a video.

[0059] Furthermore, in specific embodiments of this application, if the content text, style guidance information, or other data or information involve objects, when the embodiments of this application are applied to specific products or technologies, permission or consent from the objects is required, and the collection, use, and processing of related data or information must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0060] Based on the above image generation scheme and system, this application provides an image generation method. See also... Figure 2 This is a flowchart illustrating an image generation method provided in an embodiment of this application. Figure 2 The image generation method shown can be derived from... Figure 1The method is executed by a server or terminal device in the image generation system shown. In this embodiment, the method is described using the example of it being executed by a server.

[0061] in, Figure 2 The image generation method shown may include steps S201 to S204:

[0062] S201, perform object recognition on the content text to obtain object description text, and generate object control images based on the object description text.

[0063] In this embodiment, the content text can be text in various forms such as books, scripts, movies, TV series, novels, essays, and poems. The object description text, on the other hand, is text used to describe the objects contained within the content text; specifically, the objects can be at least one of several types of objects, such as people, animals, and items.

[0064] Specifically, the object description text may include at least one of the following: the original text used to describe the object in the content text; integrated text generated based on the original text (such as text obtained by stringing together the original text using some modifying language; or text obtained by reasonably expanding the original text); there is no limitation on this. At the same time, object identification can be performed on part or all of the text in the content text to obtain the object description text; there is no limitation on this either.

[0065] In practical implementation, methods such as named entity recognition, relation extraction, and keyword matching can be used to identify objects in the content text and obtain object description text. For example, at least one of the following methods can be used to identify object description text from the content text: large language models (such as Mixed Language Modeling, GPT-4 (a large multimodal language model)), SpaCy (an open-source natural language processing library that supports multiple languages ​​and provides powerful named entity recognition and other text processing capabilities), and Stanford NLP (an advanced text analysis tool that includes high-precision named entity recognition and relation extraction).

[0066] In one embodiment, a generative model for generating images can be invoked to generate object-controlled images based on object description text. For example, the generative model for generating images may include at least one of the following models: DALL·E3 (a model designed to generate images using text descriptions), Stable Diffusion (a type of diffusion model that generates data by treating the data distribution as a steady state of a diffusion process and using a reverse diffusion process; currently commonly used for image generation, text generation, audio generation, etc.), and OmniGen (a multimodal model for unifying image generation, supporting various image generation tasks such as text-to-image, image editing, topic-driven generation, and visual conditional generation, and featuring a highly simplified architecture).

[0067] Optionally, a generative model for generating images can be invoked to generate candidate images of objects based on object description text and output the candidate images of objects. If a confirmation operation on the candidate images of objects is detected, the candidate images of objects are used as object control images. If no confirmation operation on the candidate images of objects is detected, the step of invoking the generative model for generating images of objects based on object description text is triggered.

[0068] Optionally, a generative model for generating images can be invoked to generate multiple object candidate images based on the object description text, and output multiple object candidate images; if a confirmation operation is detected for any of the multiple object candidate images, then any object candidate image is used as the object control image.

[0069] Optionally, object description text can also be output so that staff can design and generate object control images based on the object description text, or staff can select a suitable image based on the object description text and use the selected image as the object control image.

[0070] S202, extract object features from the object control image to obtain object features, and obtain style guidance information for indicating image style and content guidance information for indicating image content elements.

[0071] In the embodiments of this application, convolutional neural networks (CNNs), models based on the Transformer architecture, U-Net (a network architecture with a U-shaped structure), and other models for extracting image features can be used to extract object features from object control images and obtain object features. No limitation is made here.

[0072] Furthermore, the style guidance information in this application embodiment can be at least one of an image expressing a certain style, text describing a style, or features characterizing a certain style (specifically, it can be extracted from an image expressing a certain style or text describing a style). Content guidance information can be at least one of an image containing a certain content element, text describing a certain content element, or features characterizing a certain content element (specifically, it can be extracted from an image expressing a certain content element or text describing a certain content element).

[0073] Specifically, features extracted solely from object description text can characterize an object's features, such as blond hair, large eyes, and medium build. However, these features are not specific. For example, blond hair can be described in more detail as long, curly blond hair or short, straight blond hair; large eyes can be described in more detail as large, dark eyes or large, round eyes; and medium build can be described in more detail as a 40 / 60 or 50 / 50 proportion. Therefore, the image of the same object in different images generated under the influence of these features will inevitably differ and cannot maintain a high degree of consistency.

[0074] However, the image of an object in a control image is specific and detailed; after all, it's impossible for an object in one image to have both long, curly blonde hair and short, straight blonde hair. Therefore, the object features extracted from the control image can represent the specific details of the object; and the image details of the same object in different images subsequently generated under the influence of these object features will remain consistent, which greatly improves the accuracy of controlling the consistency of the object.

[0075] In one embodiment, considering that the main objects in most types of text are people, and that the illustrations for the articles are also mainly people, the object control images may specifically include face control images and pose control images.

[0076] The specific process of obtaining object features can include: extracting facial features from the face control image included in the object control image to obtain facial features; and extracting pose features from the pose control image included in the object control image to obtain pose features; and obtaining object features based on pose features and facial features. Alternatively, pose features and facial features can be directly concatenated to obtain object features; or they can be fused together to obtain object features. This is not limited here.

[0077] Specifically, the feature extraction network in a face recognition model can be invoked to extract facial features from the face control image, or the aforementioned model for image feature extraction can be used to extract facial features from the face control image; this is not limited to either approach. Similarly, the feature extraction network in a pose recognition model can be invoked to extract pose features from the pose control image, or the aforementioned model for image feature extraction can be used to extract pose features from the pose control image; this is not limited to either approach.

[0078] Optionally, considering that the faces of the same object in multiple images need to be completely consistent, but the poses may not need to be completely consistent, the pose control image can be generated based on the object description text. That is, the pose control image can be a part of the object control image, or it can be entered by the staff each time the image is generated.

[0079] The specific process of acquiring object features can include: extracting facial features from the face control image included in the object control image to obtain facial features; acquiring a pose control image and extracting pose features from the pose control image to obtain pose features; and finally, obtaining object features based on the pose features and facial features. The process of acquiring the pose control image can include: the terminal device responding to an image input operation by using the input image as the pose control image.

[0080] Specifically, for text containing people, the people's poses may change with the scene, but the faces remain the same. Therefore, by using a unified face control image and potentially different pose control images, when generating multiple images corresponding to this type of text, we can maintain the unique characteristics of the people in different image scenes while effectively achieving consistent control over the people in multiple images.

[0081] S203, extract text features from the target text that requires accompanying images to obtain the text features of the target text.

[0082] In the embodiments of this application, the target text can be selected manually from the content text, or it can be selected by the terminal device or server in the aforementioned image generation system from the content text; no limitation is made here. Specifically, each paragraph in the content text can be traversed, and the currently traversed paragraph can be used as the target text; alternatively, a text segment can be randomly selected from the content text as the target text; no limitation is made here.

[0083] Specifically, text features can be extracted from the target text using models such as the bag-of-words model, Word2Vec (Word to Vector, a technique for converting words into vector form), FastText (a technique for efficiently learning word vectors and classifying sentences), and Bidirectional Encoder Representations from Transformers (BERT, a pre-trained language model).

[0084] In one possible implementation, the specific process of text feature extraction may include: obtaining key content text in the target text; wherein, the key content text includes at least one of object description text and scene description text contained in the target text; performing feature extraction processing on the key content text to obtain the text features of the target text.

[0085] Specifically, in scenarios where images are added to long articles or stories are converted into comics, the images mainly need to present the objects and scenes. Therefore, the key content text in the target text can be obtained, and the text features of the key content text can be extracted as the text features of the target text. This ensures that the images are appropriate for the text, and because the text length that needs to be processed during feature extraction is shorter (i.e., the text length of the key content text is shorter than the text length of the target text), the time consumed by text feature extraction can be effectively reduced, which is conducive to improving the overall image generation speed.

[0086] In practical applications, key content text within target text can be extracted using large language models such as Mixed Language and GPT-4. These large language models are trained on massive amounts of data from the internet, possess cross-domain knowledge and natural language understanding capabilities, and can understand input instructions and execute tasks through human-computer natural language dialogue, thus assisting in the acquisition of specified information.

[0087] For example, please see the appendix. Figure 3 This illustrates a process for obtaining key content text. For example... Figure 3 As shown, the target text can be input into the large language model 301, and the key content text in the target text can be obtained by inputting question commands into the large language model 301.

[0088] For example, after inputting a passage from a novel into the large language model 301, the following question sequence can be used to extract the key descriptive information of the text: (1) Describe the protagonist's appearance, including gender, age, face shape, facial features, hair, height, body type, clothing, etc.; (3) Extract the scene in which the protagonist is located, including weather, environment, foreground and background, clothing, posture, requiring specific and vivid imagery. Finally, for the protagonist in the text, the large language model 301 will output the character description text as shown in text 302; for the scene in the text, the large language model 301 will output the scene description text as shown in text 303.

[0089] In addition, step S201 mentions that object description text in the content text can also be identified using a large language model. Then, the object description text in the content text can be extracted by asking the following questions in sequence: (1) Who is the protagonist of this content text? Only one person's name is needed; (2) Describe the protagonist's appearance, including gender, age, face shape, facial features, hair, height, body type, clothing, etc.; (3) Extract 10 classic scenes of the protagonist, including weather, environment, foreground and background, clothing, and posture, which should be specific and vivid.

[0090] In another possible implementation, in order to further control the image style of the final generated image, text features can also be made to characterize the specific image style based on style guidance information.

[0091] Specifically, the process can begin by acquiring a style control image used to generate style guidance information; then, the style control image is classified to obtain style description text for the style to which it belongs; finally, the style description text and key content text are concatenated to obtain the concatenated text. The specific process of extracting text features from the key content text can include: encoding text features into the concatenated text to obtain the text features of the target text.

[0092] The style control image can be classified by calling a pre-trained style classification model or a classification function to determine its style. Optionally, each style can correspond to descriptive text, so the descriptive text corresponding to the style of the style control image can be used as the style description text. The specific implementation process of text feature encoding can be found in the aforementioned detailed implementation of text feature extraction, and will not be elaborated upon here.

[0093] Since the concatenated text contains style description text, and text features can characterize the semantics of the corresponding text, the text features obtained from the concatenated text can effectively represent the style described by the style description text. Thus, the accompanying images generated subsequently under the dual influence of style guidance information and the style represented by text features have higher style consistency with the style control image, which is beneficial for accurately controlling the style of the generated images, thereby achieving style consistency control.

[0094] S204. Based on object features, style guidance information, content guidance information, and text features of the target text, the noisy image is subjected to reverse diffusion processing to generate an image corresponding to the target text.

[0095] In this embodiment, backdiffusion is a process of recovering original data from noisy data. Backdiffusion processing of noisy images based on object features, style guidance information, content guidance information, and text features of the target text essentially influences the direction of change in the noisy image during the process of restoring it into an accompanying image, thereby ensuring that the final generated image is compatible with the aforementioned features and information.

[0096] In one embodiment, if the style guidance information includes style control images or style control text, the features of the style control images or style control text can be extracted first as style control features; simultaneously, if the content guidance information is content control images or content control text, the features of the content control images or content control text can be extracted first as style control features; then, based on the object features, style control features, style control features, and text features of the target text, the noisy image is subjected to reverse diffusion processing to generate an image corresponding to the target text.

[0097] Optionally, because the style displayed in the image is more concrete and detailed, such as expressing specific hues (e.g., Angora red in red), brushstrokes, and lines, the style guidance information can include style control features extracted from style-controlled images. Similarly, because the content elements displayed in the image are more concrete and detailed, such as expressing the specific style and hue of a person's shoes, or the specific style and hue of a vase, the content guidance information can include content control features extracted from content-controlled images.

[0098] Therefore, the process of obtaining style guidance information and content guidance information may include: obtaining a style control image and extracting style features from the style control image to obtain style guidance information; obtaining a content control image and extracting content features from the content control image to obtain content guidance information.

[0099] In practice, a pre-trained model for style classification can be invoked to extract style features from the style control image, thereby obtaining style guidance information. Specific implementation methods for style feature extraction and content feature extraction can be found in the aforementioned implementation methods for image feature extraction, and will not be repeated here.

[0100] Specifically, the more specific and clear the style represented by the style guidance information and the content elements represented by the content guidance information, the more consistent the style and content elements of the multiple accompanying images generated under the influence of the style guidance information and content guidance information will be. It is evident that by extracting style control features from style control images, the style of the generated images can be precisely controlled, thereby achieving style consistency control; and by extracting content control features from content control images, the content elements in the generated images can be precisely controlled, thereby achieving consistency control of content elements, which in turn facilitates further precise control of the image content consistency of the generated accompanying images.

[0101] Optionally, style feature extraction can specifically involve extracting high-dimensional style features of the style-controlling image. The higher the feature dimension, the more details the feature represents. Therefore, high-dimensional style features can better represent details such as tone, brushstrokes, and lines, allowing for subsequent control of the tone, brushstrokes, and lines of the generated image. This facilitates precise control of the style of the generated image and, consequently, helps maintain style consistency.

[0102] For example, please see the appendix. Figure 4 This illustrates a schematic diagram of style-controlled image processing. For example... Figure 4 As shown, after obtaining the style control image 401, the style control image 401 can be classified to obtain the style category 402 to which the style control image 401 belongs, and a style description text 403 matching the style category 402 can be generated, namely "This is a watercolor painting".

[0103] In practical implementation, a style classification model can be trained on 40 or more style datasets. When classifying the style control image 401, the style control image 401 can be input into the trained style classification model to obtain the style category 402 output by the trained style classification model. In practical applications, a trained style classification model can be obtained by training MobileNetV3 (a deep neural network model mainly used for image classification, object detection, semantic segmentation, and other tasks on mobile devices) on multiple style datasets.

[0104] At the same time, such as Figure 4As shown, style features can also be extracted from the style control image 401 to obtain 1*768-dimensional style features corresponding to the style control image 401. In practical applications, CSD ViT-L (a network model based on the VisionTransformer architecture) can be used to extract style features from the style control image to obtain high-dimensional style features.

[0105] In one possible implementation, if a new target text requiring an accompanying image is detected within the existing content text, text features can be extracted from this new target text to obtain its corresponding text features. Then, based on the aforementioned object features, style guidance information, content guidance information, and the text features corresponding to the new target text, a backdiffusion process is performed on the noisy image to generate an accompanying image corresponding to the new target text. In this way, under the influence of the same object features, style guidance information, and content guidance information, the style and content elements in the accompanying images generated from multiple different target texts will remain consistent, and the objects they contain will also remain consistent.

[0106] Optionally, if a new target text requiring an accompanying image is detected within the content text, text features can be extracted from this new target text to obtain corresponding text features. Then, new style guidance information and content guidance information can be obtained for the new target text. Finally, based on the aforementioned object features, new style guidance information, new content guidance information, and the text features corresponding to the new target text, the noisy image is subjected to reverse diffusion processing to generate an accompanying image corresponding to the new target text. Specifically, when generating multiple accompanying images for the same content text, designers may want different images to have different styles and content elements, but the main features of the protagonist (i.e., the object) in the content text will not change significantly. Therefore, each time an accompanying image is generated, new style guidance information and content guidance information can be introduced, but the object features remain the same; this can control the same objects in multiple accompanying images corresponding to the same content text, effectively ensuring the consistency of objects in multiple accompanying images corresponding to the same content text.

[0107] In this embodiment, by generating an object control image based on the object description text in the content text, the image of the object in the content text can be visualized, so that the object features extracted from the object control image can represent the specific object, thereby ensuring high-precision consistency of the objects contained in the subsequently generated image under the influence of object features. Simultaneously, this embodiment uses a reverse diffusion processing method based on object features, style guidance information, content guidance information, and text features of the target text to process noisy images. This ensures that the final generated illustration is influenced not only by text features but also by object features, style guidance information, and content guidance information. Under the influence of the same object features, the same or similar style guidance information, and the same or similar content guidance information, the objects, styles, and content elements in the generated illustration can maintain a high degree of consistency.

[0108] Furthermore, this embodiment controls the consistency of generated images in terms of content by using object features, style guidance information, and content guidance information. Compared to existing solutions that require attention interactions between different images and simultaneous generation of multiple scenes, this embodiment only needs to perform interactive fusion between object features, style guidance information, content guidance information, and text features, consuming relatively fewer computational resources and achieving faster image generation. Therefore, this embodiment can effectively improve image generation speed while ensuring image content consistency.

[0109] Based on the above image generation scheme and system, this application provides another image generation method. See also... Figure 5 This is a flowchart illustrating another image generation method provided in an embodiment of this application. Figure 5 The image generation method shown can be derived from... Figure 1 The method is executed by the server or terminal device shown. In this embodiment, the method is described using the server as an example. Figure 2 The method shown has been extended.

[0110] in, Figure 5 The image generation method shown may include the following steps S501 to S506:

[0111] S501, perform object recognition on the content text to obtain object description text, and generate an object control image based on the object description text.

[0112] In this application embodiment, the specific implementation of step S501 can be found in the specific implementation of step S201, and will not be repeated here.

[0113] S502, extract object features from the object control image to obtain object features, and obtain style guidance information for indicating image style and content guidance information for indicating image content elements.

[0114] In the embodiments of this application, the style guidance information is specifically the style features obtained by extracting style features from the style control image; and the content guidance information is specifically the style features obtained by extracting content features from the content control image.

[0115] In one embodiment, the object control image can be used as the style control image, meaning the style control image can be the same as the object control image; alternatively, the object control image can be used as the content control image, meaning the content control image can also be the same as the object control image.

[0116] Therefore, the process of obtaining style guidance information may include: extracting style features from the object control image to obtain style guidance information; and the process of obtaining content guidance information may include: extracting content features from the object control image to obtain content guidance information.

[0117] Optionally, the style control image can be the same as the content control image; this is not a limitation.

[0118] Furthermore, the specific implementation of step S502 can be found in the specific implementation of step S202, and will not be repeated here.

[0119] S503, extract text features from target text that requires accompanying images to obtain the text features of the target text.

[0120] In the embodiments of this application, the specific implementation of step S503 can be found in the specific implementation of step S203, and will not be repeated here.

[0121] S504 calls the trained image generation model to encode the noisy image and obtain the initial feature map.

[0122] In this embodiment, the trained image generation model may include an encoding module, a feature processing module, and a decoding module. Specifically, the encoding module may include an encoder, such as the encoding network in a Variational Autoencoder (VAE); the decoding module may include an encoder, such as the decoding network in a VAE; and the feature processing module can be built based on U-Net. A Variational Autoencoder typically includes an encoding network and a decoding network. The encoding network compresses the input data into a new set of data, and the decoding network restores the compressed new data to the input data.

[0123] Therefore, the encoding module in the trained image generation model can be invoked to encode the noisy image, obtaining the initial feature map. The variational autoencoder encodes the image into a set of data, which is called the latent image; thus, the initial feature map also belongs to the latent image. The latent space (also called hidden space) is a low-dimensional space used to represent data in machine learning and deep learning. It is a set of latent variables obtained by encoding and dimensionality reduction of the original data. The dimension of the latent space is usually lower than that of the original data, thus allowing the extraction of the most important features and structures from the data.

[0124] S505, based on object features, style guidance information, content guidance information and text features, performs feature sampling on the initial feature map to obtain the target feature map.

[0125] In this embodiment, the feature processing module in the trained image generation model can be invoked to perform feature sampling on the initial feature map based on object features, style guidance information, content guidance information and text features to obtain the target feature map.

[0126] In one embodiment, the feature processing module may include a downsampling network, a transition network, and an upsampling network connected in sequence, with a skip connection established between the signal output of the downsampling network and the signal input of the upsampling network. The downsampling network is used to reduce the feature dimension of the feature map, and the upsampling network is used to restore the feature dimension of the feature map to the same level as the initial feature map.

[0127] The specific process of feature sampling can include: calling a downsampling network to downsample the initial feature map based on style guidance information, content guidance information, and text features to obtain a downsampled feature map; calling a transition network to encode the downsampled feature map based on object features to obtain a transition feature map; and calling an upsampling network to upsample the transition feature map based on the feature map, style guidance information, content guidance information, object features, and text features output from the signal output of the downsampling network to obtain the target feature map.

[0128] Among them, the feature dimension of the downsampled feature map is smaller than that of the initial feature map, the feature dimension of the downsampled feature map is the same as that of the transition feature map, and the feature dimension of the target feature map is the same as that of the initial feature map.

[0129] In one embodiment, the downsampling network includes a plurality of downsampling subnetworks connected in sequence, and the upsampling network includes a plurality of upsampling subnetworks connected in sequence, with each downsampling subnetwork corresponding to each upsampling subnetwork.

[0130] The specific process of upsampling can be as follows: First, for any upsampling subnetwork in the upsampling network, obtain the specified feature map output by the signal output terminal of the corresponding downsampling subnetwork; then, concatenate the specified feature map, object features, and the feature map output by the previous network connected to any upsampling subnetwork to obtain a concatenated feature map; next, based on the concatenated feature map, style guidance information, content guidance information, and text features, obtain the feature map output by the signal output terminal of any upsampling subnetwork; finally, use the feature map output by the signal output terminal of the last upsampling subnetwork in the upsampling network as the target feature map.

[0131] It should be noted that the upsampling subnetworks in the upsampling network are called sequentially according to the connection order of the upsampling subnetworks. In addition, when any upsampling subnetwork is the first upsampling subnetwork in the upsampling network, since the previous network connected to the first upsampling subnetwork is a transition network, the feature map output by the previous network at this time is a transition feature map.

[0132] In one embodiment, the network structure of all upsampling subnetworks in the upsampling network can be the same or different, and this is not limited here.

[0133] Optionally, any upsampling sub-network in the upsampling network may include a feature fusion network layer and an upsampling convolutional network layer connected in sequence. The feature fusion network layer may include a residual network and a cross-attention network connected in sequence. The feature fusion network layer may include at least one.

[0134] Therefore, when the feature fusion network layer includes one, the processing of the feature map by any sampling sub-network may include: calling the residual network to perform residual encoding on the concatenated feature map to obtain the residual feature map; calling the cross-attention network to fuse the residual feature map, style guidance information, content guidance information and text features to obtain the fused feature map; and calling the upsampling convolutional network layer to perform convolution processing on the fused feature map to obtain the feature map output by the signal output end of any upsampling sub-network.

[0135] Optionally, the cross-attention network can first perform self-attention calculation on style guidance information based on the residual feature map to obtain style attention features; then perform self-attention calculation on content guidance information based on the residual feature map to obtain content attention features; and finally, perform self-attention calculation on text based on the residual feature map to obtain text attention features; and then add the style attention features, content attention features, and text attention features together to obtain a fused feature map.

[0136] When the feature fusion network layer comprises multiple layers, for any given feature fusion network layer, the residual network within that layer can be invoked to perform residual encoding on the feature map output by the previous network, yielding the corresponding residual feature map. Then, the cross-attention network within that layer can be invoked to fuse the residual feature map, style guidance information, content guidance information, and text features, resulting in the corresponding fused feature map. Finally, an upsampling convolutional network layer can be invoked to convolve the fused feature map output by the last feature fusion network layer, obtaining the feature map output by the signal output of any upsampling sub-network.

[0137] In one embodiment, the downsampling network includes multiple sequentially connected downsampling sub-networks. The specific process of downsampling can then include: inputting an initial feature map into the downsampling network, causing the downsampling network to sequentially call the downsampling sub-networks to process the initial feature map, thereby obtaining a downsampled feature map.

[0138] Optionally, the network structure of all downsampling subnetworks in the downsampling network can be the same or different, and this is not limited here.

[0139] Optionally, any downsampling subnetwork may include a feature fusion network layer and a downsampling convolutional network layer connected in sequence, wherein the feature fusion network layer includes a residual network and a cross-attention network connected in sequence. The number of feature fusion network layers in the downsampling subnetwork may include at least one.

[0140] The structure of the feature fusion network layer in the downsampling sub-network can be the same as or different from that in the upsampling sub-network, and is not limited here. The processing procedure of the feature map by the feature fusion network layer in the downsampling sub-network can be referred to the processing procedure of the feature map by the feature fusion network layer in the upsampling sub-network, and is not limited here.

[0141] Optionally, the signal output of the cross-attention network contained in the feature fusion network layer of the downsampling subnetwork can be connected to the signal input of the upsampling subnetwork corresponding to the downsampling subnetwork.

[0142] For specific implementation details, please refer to the appendix. Figure 6 The diagram illustrates the structure of a feature processing module. Figure 6 The feature processing module shown includes a downsampling network, a transition network 601, and an upsampling network connected in sequence. The downsampling network includes downsampling sub-networks d1 to d4 connected in sequence, and the upsampling network includes downsampling sub-networks u1 to u4 connected in sequence. There is a correspondence between d1 and u4, d2 and u3, d3 and u2, and d4 and u1.

[0143] In one embodiment, the trained image generation model may further include an object feature integration module. The object feature integration module may include a downsampling network and a mid-level convolutional network connected in sequence. Specifically, the signal output of the mid-level convolutional network may establish a skip connection with the signal input of the transition network, and the signal output of the downsampling network may establish a skip connection with the signal input of the upsampling network.

[0144] Therefore, the object features from the object control image can be input into the object feature processing module, so that the downsampling network processes the object features to obtain low-dimensional object features, and the middle-layer convolutional network processes the low-dimensional object features to obtain the final output target object features.

[0145] The specific process of calling the transition network to encode the downsampled feature map based on the object features can include: calling the transition network to obtain the target object features output by the signal output terminal of the middle convolutional network; and encoding the downsampled feature map based on the target object features to obtain the transition feature map.

[0146] Optionally, the downsampling network includes multiple downsampling sub-networks connected in sequence. Each downsampling sub-network corresponds to each upsampling sub-network in the feature processing module. Then, the process of obtaining the spliced ​​feature map corresponding to any upsampling sub-network may include: obtaining the specified object features output by the downsampling sub-network corresponding to any upsampling sub-network; splicing the specified feature map, the specified person features, and the feature map output by the previous network connected to any upsampling sub-network to obtain the spliced ​​feature map.

[0147] In the specific implementation, the architecture of the object feature processing module can be referenced from ControlNet (a control network in Stable Diffusion), and no specific limitations will be imposed on the architecture of the object feature processing module here.

[0148] S506, decode the target feature map to obtain the corresponding image for the target text.

[0149] In this embodiment, the decoding module in the trained image generation model can be invoked to decode the target feature map and obtain the corresponding image for the target text. Specifically, the decoding module can be the decoding network in a variational autoencoder.

[0150] For specific implementation details, please refer to the appendix. Figure 7 This illustrates a schematic diagram of the structure of a trained image generation model. Figure 7As shown, the trained image generation model includes an object adapter (including an object feature extraction module 701 and an object feature integration module 702), a style adapter (including a style feature extraction module 703, a style feature mapping module 704 and a style attention module 705), a content adapter (including a content feature extraction module 706, a content feature mapping module 707 and a content attention module 708), a text feature encoding module 709, a text attention module 710, an encoding module, a feature processing module 712, and a decoding module.

[0151] Among them, the object adapter is used to control the objects contained in the generated image to achieve object consistency; the style adapter is used to control the style of the generated image to achieve style consistency; and the content adapter is used to control the content elements in the generated image to achieve content element consistency, and further strengthens object consistency and style consistency through consistent content elements.

[0152] In practical applications, the style feature extraction module 703 can specifically be a trained CSD ViT-L network, and the style feature mapping module 704 can map the features extracted by the style feature extraction module 703 into higher-dimensional features, such as mapping from 768 dimensions to 1024 dimensions. Similarly, the content feature extraction module 706 can specifically be a trained Contrastive Language–Image Pretraining (CLIP, a multimodal model); and the content feature mapping module 707 can also map the features extracted by the content feature extraction module 706 into higher-dimensional features.

[0153] Furthermore, each of the cross-attention networks 715 in the feature processing module 712 may include a style attention module 705, a content attention module 708, and a text attention module 710. The network parameters of the style attention module 705, content attention module 708, and text attention module 710 in different cross-attention networks may be different.

[0154] In practical applications, the specific implementations of the content attention module 708 and the text attention module 710 can be the same as the trained IP-Adapter (a technique or method designed to enhance the capabilities of image processing and generative models). The style attention module 705 can be implemented in a similar way to the content attention module 708.

[0155] Optionally, since high feature dimensions are needed to better represent style details, the style attention module 705 may only exist in the various cross-attention networks in the transition network and upsampling network included in the feature processing module 712; this can effectively control the style of the generated image while saving computational resources as much as possible, which is beneficial to improving the image generation speed.

[0156] Specifically, the text features output by the text feature encoding module 709, the content features output by the content feature mapping module 707, and the style features output by the style feature mapping module 704 can be fused in various cross-attention networks to jointly influence the output of the feature processing module 712.

[0157] For example, the feature maps Z output by each cross-attention network in the feature processing module 712 ‘ It can be shown in formula (1-1):

[0158]

[0159] Where Q = ZW q Z represents the feature map output by the previous network to which each cross-attention network is connected, and K = c t W k V = c t W v c t W represents the text features of the target text (in this example, these features could be those output by the text feature encoding module 709). q W k and W v These are the query projection matrix (i.e., the query projection matrix), the key projection matrix (i.e., the key matrix), and the value projection matrix (i.e., the value matrix), respectively, after passing through various cross-attention networks. q W k and W v These are all parameters obtained through model training; W in different cross-attention networks q W k and W v They may be different.

[0160] In addition, K content =c content W k-content V content =c content W v-content K style =c style W k-style V style =c style W v-style c content For the content features included in the content guidance information (specifically, in this example, the features output by the content feature mapping module 707), c style This refers to the style features in the style guidance information (specifically, in this example, the features output by the style feature mapping module 704). And W... k-content and Wv-content These are the key projection matrix for content features and the value projection matrix for text features, respectively, and W. k-style and W v-style These are the key projection matrix for content features and the value projection matrix for text features, respectively.

[0161] In general, combining Figure 7 The specific process of the trained image generation model generating accompanying images is as follows:

[0162] (1) In response to the input object control image 700, the object control image 700 is input into the style feature extraction module 703, so that the style feature extraction module 703 extracts style features from the object control image 700, and the style feature mapping module 704 performs feature mapping on the features output by the style feature extraction module 703 to obtain the mapped style features; and the mapped style features are input into the style attention module 705 in each cross attention network within the feature processing module 712.

[0163] In response to the input object control image 700, the object control image 700 is input to the content feature extraction module 706, so that the content feature extraction module 706 performs style feature extraction on the object control image 700, and the content feature mapping module 707 performs feature mapping on the features output by the content feature extraction module 706 to obtain the mapped content features; and the mapped content features are input to the content attention modules 708 in each cross-attention network within the feature processing module 712.

[0164] Simultaneously, in response to the input object control image 700, the object control image 700 is input into the object feature extraction module 701, so that the object feature extraction module 701 performs object feature extraction on the object control image 700 to obtain object features including face features and pose features; and the object features are input into the object feature integration module 702, so that the object features are input into the feature processing module 712 through the object feature integration module 702.

[0165] (2) In response to the input target text “This is a watercolor painting, a little boy is sitting on the grass, and sheep are scattered all around”, the target text is input into the text feature encoding module 709 so that the text feature encoding module 709 encodes the target text to obtain text features; then, the text features are input into the text attention module 710 in each cross attention network in the feature processing module 712.

[0166] (3) Input the noisy image into the encoding module so that the encoding module encodes the noisy image to obtain the initial feature map 711; then, input the initial feature map 711 into the feature processing module 712 so that the feature processing module 712 processes the initial feature map 711 based on the mapped style features, mapped content features, object features input by the object feature integration module 702, and text features, and outputs the target feature map 713.

[0167] (4) Input the target feature map 713 into the decoding module so that the decoding module can decode the target feature map 713 to obtain the matching map 714 generated for the target text. It should be noted that this example does not limit the execution order of steps (1) to (4).

[0168] For specific implementation details, please refer to the appendix. Figure 8 This illustrates a schematic diagram of the process by which a trained image generation model generates accompanying images. For example... Figure 8 As shown, a style control image 801 can be input into the style feature extraction module of the trained image generation model, and a face control image 802 can be input into the content feature extraction module of the trained image generation model (at this time, the face control image 802 serves as the content control image).

[0169] In addition, a face control image 802 is input into the object feature extraction module of the trained image generation model to extract face features, and a pose control image 803 is input into the object feature extraction module to extract pose features, thereby obtaining object features containing face features and pose features.

[0170] Simultaneously, the target text "This is a watercolor painting; he walks in a desert with occasional cacti and camel thorns" is input into the text feature encoding module of the trained image generation model to obtain the text features of the target text. Finally, the trained image generation model can output an image 804 corresponding to the target text. For details on the specific processing procedures of each module in the trained image generation model, please refer to [link to relevant documentation]. Figure 7 Examples are provided, but will not be elaborated upon here.

[0171] In one embodiment, the feature processing module in the image generation model can be a pre-trained network model capable of generating images (such as a trained U-Net). Optionally, the object feature extraction module, style feature extraction module, content feature extraction module, and text feature extraction module can also be pre-trained modules.

[0172] After preparing the training text and training control images, the training text can be input into the text feature encoding module of the image generation model, and the training control images can be input into the style feature extraction module, content feature extraction module, and object feature extraction module of the image generation model, respectively, to obtain the corresponding feature information. The purpose of training is to transform the latent space image into a noisy image through the feature processing module under the action of the above feature information. The loss function is the loss value between the noisy image and the ground truth noise. Finally, the parameters of the style feature mapping module, content feature mapping module, object feature integration module, style attention module, content attention module, and text attention module of the image generation model are updated in the direction of reducing the loss value to obtain the trained image generation model.

[0173] Optionally, the parameters of the style feature mapping module, content feature mapping module, object feature integration module, style attention module, content attention module, text attention module, object feature extraction module, style feature extraction module, content feature extraction module, and text feature extraction module in the image generation model can be updated in the direction of reducing the loss value to obtain the trained image generation model.

[0174] In the actual experiment, quantitative and qualitative evaluations were conducted on the above image generation schemes. The quantitative evaluation used cosine similarity to measure the correlation between faces in multiple images, and style similarity to measure the similarity between the style of the generated image and the target style. The basic scheme for face similarity was InstantID, and the basic scheme for style similarity was IP-Adapter. The specific results are shown in Table 1.

[0175] Table 1

[0176] CosineSimilarity StyleSimilarity Basic Solution 0.72 0.28 This plan 0.79 0.63

[0177] As shown in Table 1, compared to the basic scheme, this scheme shows a slight improvement in facial similarity; and compared to the basic scheme, this scheme shows a significant improvement in style similarity. It is evident that this scheme not only improves image generation speed but also enhances the similarity between faces in the generated images, thereby further improving the accuracy of facial consistency control; and it also significantly improves the similarity between the style of the generated images and the target style, thus greatly enhancing the accuracy of style consistency control.

[0178] In this embodiment, by encoding a noisy image to obtain an initial feature map, the noisy image can interact with object features, style guidance information, content guidance information, and text features in the form of a feature map. This allows subsequent feature sampling of the initial feature map under the influence of object features, style guidance information, content guidance information, and text features to obtain a target feature map. Thus, the image obtained by decoding the target feature map maintains consistency in image content such as object, style, and content elements.

[0179] Based on the above description of the image generation method, this application also discloses an image generation apparatus. This image generation apparatus can be a computer program (including program code) running on the aforementioned computer device. The image generation apparatus can perform actions such as... Figure 2 and Figure 3 For the image generation method shown, please refer to [link / reference]. Figure 9 The image generation apparatus may include at least: an acquisition unit 901, an image feature extraction unit 902, a text feature extraction unit 903, and a generation unit 904, wherein:

[0180] The acquisition unit 901 is used to perform object recognition on the content text, obtain object description text, and generate object control images based on the object description text;

[0181] The image feature extraction unit 902 is used to extract object features from the object control image, obtain object features, and acquire style guidance information for indicating image style and content guidance information for indicating image content elements;

[0182] The text feature extraction unit 903 is used to extract text features from target text that has image requirements in the content text, and obtain the text features of the target text.

[0183] The generation unit 904 is used to perform reverse diffusion processing on the noisy image based on object features, style guidance information, content guidance information and text features of the target text to generate an image corresponding to the target text.

[0184] In one embodiment, when the text feature extraction unit 903 extracts text features from target text that has image requirements in the content text to obtain the text features of the target text, it can specifically perform the following: obtaining key content text in the target text; wherein, the key content text includes at least one of object description text and scene description text contained in the target text; performing feature extraction processing on the key content text to obtain the text features of the target text.

[0185] In another embodiment, the text feature extraction unit 903 can also be used to: acquire a style control image for generating style guidance information; classify the style control image to obtain style description text of the style to which the style control image belongs; and concatenate the style description text and the key content text to obtain concatenated text. Specifically, when performing feature extraction processing on the key content text to obtain the text features of the target text, the text feature extraction unit 903 can be used to: encode the text features of the concatenated text to obtain the text features of the target text.

[0186] In another embodiment, when the image feature extraction unit 902 acquires style guidance information for indicating image style and content guidance information for indicating image content elements, it can also specifically perform: acquiring a style control image and extracting style features from the style control image to obtain style guidance information; acquiring a content control image and extracting content features from the content control image to obtain content guidance information.

[0187] In another embodiment, when the image feature extraction unit 902 acquires style guidance information for indicating image style and content guidance information for indicating image content elements, it can also specifically perform: style feature extraction on the object control image to obtain style guidance information; and content feature extraction on the object control image to obtain content guidance information.

[0188] In another embodiment, when the image feature extraction unit 902 extracts object features from the object control image to obtain object features, it can also specifically perform the following: extracting face features from the face control image included in the object control image to obtain face features; acquiring a pose control image and extracting pose features from the pose control image to obtain pose features; and obtaining object features based on the pose features and face features.

[0189] In another embodiment, when generating an image corresponding to the target text by performing reverse diffusion processing on a noisy image based on object features, style guidance information, content guidance information, and text features of the target text, the generation unit 904 may also perform the following: calling a trained image generation model to encode the noisy image to obtain an initial feature map; performing feature sampling on the initial feature map based on object features, style guidance information, content guidance information, and text features to obtain a target feature map; and performing decoding processing on the target feature map to obtain an image corresponding to the target text.

[0190] In another embodiment, the trained image generation model includes a feature processing module, which comprises a downsampling network, a transition network, and an upsampling network connected in sequence. A skip connection is established between the signal output of the downsampling network and the signal input of the upsampling network. When the generation unit 904 performs feature sampling on the initial feature map based on object features, style guidance information, content guidance information, and text features to obtain the target feature map, it can specifically perform the following: calling the downsampling network to downsample the initial feature map based on style guidance information, content guidance information, and text features to obtain a downsampled feature map; calling the transition network to perform feature encoding on the downsampled feature map based on object features to obtain a transitional feature map; and calling the upsampling network to upsample the transitional feature map based on the feature map output from the signal output of the downsampling network, style guidance information, content guidance information, object features, and text features to obtain the target feature map.

[0191] In another embodiment, the downsampling network includes multiple downsampling sub-networks connected in sequence, and the upsampling network includes multiple upsampling sub-networks connected in sequence, with each downsampling sub-network corresponding to each upsampling sub-network. When the generation unit 904 calls the upsampling network and performs upsampling processing on the transition feature map based on the feature map, style guidance information, content guidance information, object features, and text features output from the signal output terminal of the downsampling network to obtain the target feature map, it can also be used to perform the following: for any upsampling sub-network in the upsampling network, obtain the downsampling corresponding to any upsampling sub-network. The specified feature map is output by the signal output terminal of the sub-network; the specified feature map, object features, and the feature map output by the previous network connected to any upsampled sub-network are concatenated to obtain a concatenated feature map; wherein, when any upsampled sub-network is the first upsampled sub-network in the upsampled network, the feature map output by the previous network is a transitional feature map; based on the concatenated feature map, style guidance information, content guidance information, and text features, the feature map output by the signal output terminal of any upsampled sub-network is obtained; the feature map output by the signal output terminal of the last upsampled sub-network in the upsampled network is taken as the target feature map.

[0192] In another embodiment, any upsampling sub-network includes a feature fusion network layer and an upsampling convolutional network layer connected in sequence. The feature fusion network layer includes a residual network and a cross-attention network connected in sequence. Based on the aforementioned scheme, when the generation unit 904 obtains the feature map output by the signal output terminal of any upsampling sub-network based on the spliced ​​feature map, style guidance information, content guidance information, and text features, it can specifically perform the following: calling the residual network to perform residual encoding on the spliced ​​feature map to obtain a residual feature map; calling the cross-attention network to fuse the residual feature map, style guidance information, content guidance information, and text features to obtain a fused feature map; and calling the upsampling convolutional network layer to perform convolution processing on the fused feature map to obtain the feature map output by the signal output terminal of any upsampling sub-network.

[0193] According to one embodiment of this application, Figure 2 and Figure 5 The steps involved in the method shown can be derived from... Figure 9 This is performed by the individual units in the image generation apparatus shown. For example, Figure 5 The step S501 shown can be performed by Figure 9 The acquisition unit 901 in the image generation apparatus shown is responsible for executing step S502; step S502 can be performed by... Figure 9 The image feature extraction unit 902 in the image generation apparatus shown is used to perform step S503; step S503 can be performed by... Figure 9 The text feature extraction unit 903 in the image generation apparatus shown is used to perform the steps; steps S504 to S506 can be performed by... Figure 9 The generation unit 904 in the image generation apparatus shown performs this operation.

[0194] According to another embodiment of this application, Figure 9 The units in the illustrated image generation apparatus are divided based on logical functions. These units can be individually or entirely merged into one or more other units, or some of these units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. In other embodiments of this application, the aforementioned image generation apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0195] According to another embodiment of this application, a general-purpose computing device, such as a computer device, including processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), can be used to run an application capable of performing tasks such as... Figure 2 or Figure 5The computer program (including program code) involved in each step of the method shown is used to construct, as... Figure 9 The image generation apparatus shown is illustrated, as well as the image generation method for implementing the embodiments of this application. A computer program may be recorded on, for example, a computer storage medium, loaded onto the aforementioned computer device via the computer storage medium, and run therein.

[0196] In this embodiment, a reverse diffusion process is used to process noisy images based on object features, style guidance information, content guidance information, and text features of the target text. This ensures that the final generated image is influenced not only by text features but also by object features, style guidance information, and content guidance information. Under the influence of the same object features, similar style guidance information, and similar content guidance information, the objects, styles, and content elements in the generated image maintain a high degree of consistency. Compared to existing solutions that require attention interactions between different images and simultaneous generation of multiple scenes, this embodiment only requires interaction and fusion between object features, style guidance information, content guidance information, and text features, consuming relatively fewer computational resources and achieving faster image generation. Therefore, this embodiment can effectively improve image generation speed while ensuring image content consistency.

[0197] Based on the above method and apparatus embodiments, this application also provides an electronic device. See also Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 10 The electronic device shown may include at least a processor 1001, an input interface 1002, an output interface 1003, and a computer storage medium 1004. The processor 1001, the input interface 1002, the output interface 1003, and the computer storage medium 1004 may be connected via a bus or other means.

[0198] The computer storage medium 1004 can be stored in the memory of the electronic device. The computer storage medium 1004 is used to store computer programs, which include program instructions. The processor 1001 is used to execute the program instructions stored in the computer storage medium 1004. The processor 1001 (or CPU (Central Processing Unit)) is the computing and control core of the electronic device. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to realize the above-mentioned image generation method flow or corresponding functions.

[0199] This application embodiment also provides a computer storage medium (Memory), which is a memory device in an electronic device used to store programs and data. It is understood that the computer storage medium here can include the built-in storage medium in a terminal, or it can include an extended storage medium supported by the terminal. The computer storage medium provides storage space, which stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 1001. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium here can be a high-speed random access memory (RAM), or it can be non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor.

[0200] In one embodiment, the processor 1001 may load and execute one or more instructions stored in the computer storage medium to implement the aforementioned related... Figure 2 and Figure 5 In the corresponding steps of the image generation method embodiment, in a specific implementation, one or more instructions in the computer storage medium are loaded and executed by the processor 1001, such as... Figure 2 and Figure 5 The steps in the illustrated image generation method embodiment are shown.

[0201] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned actions. Figure 2 and Figure 5 The method embodiment shown. The computer-readable storage medium may be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0202] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims. Furthermore, this application does not limit the execution order of the various steps in the specific embodiments.

Claims

1. An image generation method, characterized in that, The method includes: Object recognition is performed on the content text to obtain object description text, and an object control image is generated based on the object description text; The object control image is subjected to object feature extraction to obtain object features, and style guidance information for indicating image style and content guidance information for indicating image content elements are obtained. Text features are extracted from the target text that contains images, and the text features of the target text are obtained. Based on the object features, style guidance information, content guidance information, and text features of the target text, the noisy image is subjected to reverse diffusion processing to generate an image corresponding to the target text.

2. The method according to claim 1, characterized in that, The step of extracting text features from the target text that requires accompanying images, to obtain the text features of the target text, includes: Obtain key content text from the target text; wherein, the key content text includes at least one of object description text and scene description text contained in the target text; Feature extraction processing is performed on the key content text to obtain the text features of the target text.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the style control image used to generate the style guidance information; The style control image is classified to obtain the style description text of the style to which the style control image belongs; The style description text and the key content text are concatenated to obtain the concatenated text; The step of performing feature extraction processing on the key content text to obtain the text features of the target text includes: Text feature encoding is performed on the concatenated text to obtain the text features of the target text.

4. The method according to claim 1, characterized in that, The acquisition of style guidance information for indicating image style and content guidance information for indicating image content elements includes: A style control image is acquired, and style features are extracted from the style control image to obtain the style guidance information; A content control image is acquired, and content features are extracted from the content control image to obtain the content guidance information.

5. The method according to claim 1, characterized in that, The acquisition of style guidance information for indicating image style and content guidance information for indicating image content elements includes: Style features are extracted from the object control image to obtain the style guidance information; Content features are extracted from the object control image to obtain the content guidance information.

6. The method according to claim 1, characterized in that, The step of extracting object features from the object control image to obtain object features includes: Facial features are extracted from the face control image included in the object control image to obtain facial features; Acquire an attitude control image and extract attitude features from the attitude control image to obtain attitude features; The object features are obtained based on the pose features and the face features.

7. The method according to any one of claims 1 to 6, characterized in that, The method of performing reverse diffusion processing on the noisy image based on the object features, the style guidance information, the content guidance information, and the text features of the target text to generate an image corresponding to the target text includes: The trained image generation model is invoked to encode the noisy image and obtain an initial feature map; Based on the object features, the style guidance information, the content guidance information, and the text features, feature sampling is performed on the initial feature map to obtain the target feature map; The target feature map is decoded to obtain the corresponding image for the target text.

8. The method according to claim 7, characterized in that, The trained image generation model includes a feature processing module, which includes a downsampling network, a transition network and an upsampling network connected in sequence. The signal output terminal of the downsampling network and the signal input terminal of the upsampling network are connected by a skip connection. The step of sampling features from the initial feature map based on the object features, the style guidance information, the content guidance information, and the text features to obtain the target feature map includes: The downsampling network is invoked to downsample the initial feature map based on the style guidance information, content guidance information, and text features, to obtain a downsampled feature map; The transition network is invoked to encode the downsampled feature map based on the object features, thereby obtaining the transition feature map. The upsampling network is invoked, and based on the feature map output from the signal output terminal of the downsampling network, the style guidance information, the content guidance information, the object features, and the text features, the transition feature map is upsampled to obtain the target feature map.

9. The method according to claim 8, characterized in that, The downsampling network includes multiple downsampling sub-networks connected in sequence, and the upsampling network includes multiple upsampling sub-networks connected in sequence, with each downsampling sub-network corresponding to each upsampling sub-network. The step of calling the upsampling network, based on the feature map output from the signal output terminal of the downsampling network, the style guidance information, the content guidance information, the object features, and the text features, upsamples the transition feature map to obtain the target feature map, including: For any upsampling subnetwork in the upsampling network, obtain the specified feature map output by the signal output terminal of the downsampling subnetwork corresponding to any upsampling subnetwork; The specified feature map, the object feature, and the feature map output by the previous network connected to any upsampling sub-network are concatenated to obtain a concatenated feature map; wherein, when any upsampling sub-network is the first upsampling sub-network in the upsampling network, the feature map output by the previous network is the transition feature map; Based on the spliced ​​feature map, the style guidance information, the content guidance information, and the text features, the feature map output by the signal output terminal of any upsampling sub-network is obtained; The feature map output by the signal output terminal of the last upsampling sub-network in the upsampling network is used as the target feature map.

10. The method according to claim 9, characterized in that, Each upsampling sub-network includes a feature fusion network layer and an upsampling convolutional network layer connected in sequence, and the feature fusion network layer includes a residual network and a cross attention network connected in sequence. The step of obtaining the feature map output by the signal output terminal of any upsampled sub-network based on the spliced ​​feature map, the style guidance information, the content guidance information, and the text features includes: The residual network is invoked to perform residual encoding on the spliced ​​feature map to obtain a residual feature map; The cross-attention network is invoked to fuse the residual feature map, the style guidance information, the content guidance information, and the text features to obtain a fused feature map; The upsampling convolutional network layer is invoked to perform convolution processing on the fused feature map, thereby obtaining the feature map output by the signal output terminal of any upsampling sub-network.

11. An image generation apparatus, characterized in that, The device includes an acquisition unit, an image feature extraction unit, a text feature extraction unit, and a generation unit, wherein: The acquisition unit is used to perform object recognition on the content text to obtain object description text, and generate an object control image based on the object description text; The image feature extraction unit is used to extract object features from the object control image, obtain object features, and acquire style guidance information for indicating image style and content guidance information for indicating image content elements. The text feature extraction unit is used to extract text features from target text that has image requirements in the content text, and obtain the text features of the target text. The generation unit is used to perform reverse diffusion processing on the noisy image based on the object features, the style guidance information, the content guidance information, and the text features of the target text, to generate an image corresponding to the target text.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image generation method as described in any one of claims 1 to 10.

13. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the image generation method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program adapted to be loaded by a processor and executed as any one of claims 1 to 10.

Images