Image processing method and apparatus, electronic device, computer-readable storage medium, and computer program product

Abstract

Embodiments of the present application provide an image processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which are at least applied to the field of artificial intelligence field or the field of image generation. The method comprises: acquiring a description text, and a raw image having a target object; encoding the raw image, to obtain an image encoding feature; encoding the description text, to obtain a text encoding feature; performing multi-level image residual mapping on the image encoding feature, to obtain an image semantic identification feature of the target object under each level among multiple levels; on the basis of the image semantic identification feature under each level, performing multi-level text residual mapping on the text encoding feature, to obtain a fused feature, the number of levels of the multi-level text residual mapping being the same as the number of stages of the multi-level image residual mapping; and decoding the fused feature, to obtain a generated image, the generated image comprising an object feature of the target object and a description feature of the description text.

Description

Image processing methods, apparatuses, electronic devices, computer-readable storage media, and computer program products

[0001] Cross-reference to related applications

[0002] This application is based on and claims priority to Chinese Patent Application No. 202410660678.6, filed on May 27, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the Internet field, and includes, but is not limited to, an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0004] In the field of image generation, to generate personalized posters and photos in the desired style for users, or to generate game-style images or game character images with the user's facial features or appearance, related technologies typically use general text-to-image (TGI) models. For example, a stable diffusion model can be fine-tuned using several or dozens of personal images of the user to refine the basic TGI model, thus incorporating the user's personal image conceptually into the model, enabling it to generate personalized images. Alternatively, a generic style image can be generated first, without any specific facial features. Then, a face-swapping model can be used to swap the face from the user's input image onto the generated style image, resulting in a personalized image.

[0005] However, the image generation methods in related technologies all suffer from problems such as long generation time, high consumption of computing resources of electronic devices, poor image fusion effect of generated images, and overall facial disharmony and incoordination, which affect the final image quality. Summary of the Invention

[0006] This application provides an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can be applied to at least the fields of artificial intelligence or image generation. It can not only improve image generation efficiency and save computing resources, but also improve the feature fusion effect and image quality of the generated image.

[0007] The technical solution of this application embodiment is implemented as follows:

[0008] This application provides an image processing method executed by an electronic device. The method includes: acquiring descriptive text and an original image containing a target object; encoding the original image to obtain image encoding features; encoding the descriptive text to obtain text encoding features; performing multi-level image residual mapping on the image encoding features to obtain image semantic identifier features of the target object at each of the multiple levels; performing multi-level text residual mapping on the text encoding features based on the image semantic identifier features at each level to obtain fusion features; wherein the number of levels of the multi-level text residual mapping is the same as the number of levels of the multi-level image residual mapping; and decoding the fusion features to obtain a generated image; the generated image includes object features of the target object and descriptive features of the descriptive text.

[0009] This application provides an image processing apparatus, comprising: an acquisition module configured to acquire descriptive text and an original image having a target object; an encoding module configured to encode the original image to obtain image encoding features; and to encode the descriptive text to obtain text encoding features; an image residual mapping module configured to perform multi-level image residual mapping on the image encoding features to obtain image semantic identifier features of the target object at each of the multiple levels; a text residual mapping module configured to perform multi-level text residual mapping on the text encoding features based on the image semantic identifier features at each level to obtain fused features; wherein the number of levels of the multi-level text residual mapping is the same as the number of levels of the multi-level image residual mapping; and a decoding module configured to decode the fused features to obtain a generated image; the generated image includes object features of the target object and descriptive features of the descriptive text.

[0010] This application provides an electronic device, including: a memory for storing executable instructions; and a processor for executing the executable instructions stored in the memory to implement the above-described image processing method.

[0011] This application provides a computer program product including executable instructions stored in a computer-readable storage medium; wherein, when the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the above-described image processing method.

[0012] This application provides a computer-readable storage medium storing executable instructions, which are used to cause a processor to execute the executable instructions to implement the above-described image processing method.

[0013] The embodiments of this application have the following beneficial effects:

[0014] During image processing, after encoding the original image and descriptive text separately, the electronic device performs multi-level image residual mapping on the image encoding features to obtain the image semantic identifier features of the target object in the original image at each of the multiple levels. Based on the image semantic identifier features at each level, the text encoding features are then sequentially subjected to multi-level text residual mapping to obtain fused features. Decoding these fused features yields a generated image that simultaneously carries the object features of the target object and the descriptive features of the descriptive text. Furthermore, using these fused features, which have undergone multi-level text residual mapping, to generate the final image, improves the image fusion effect. Since the multi-level text residual mapping process integrates the image semantic identifier features at each level, the resulting fused feature is a fusion feature that considers the image semantic identifier features at multiple levels. In other words, the fused feature incorporates the object features of the target object in the original image at multiple levels. Therefore, using this fused feature to generate the image improves the overall image fusion effect, making the overall face of the target object more harmonious in the generated image, thereby improving the image quality of the generated image. Furthermore, the method in this application embodiment can directly process the input original image and descriptive text. The input information consists only of the original image and descriptive text, without the need for too many complex model training and generation steps, thereby improving image generation efficiency and greatly saving computational resources in the image generation process. Attached Figure Description

[0015] Figure 1 is a schematic diagram of an optional architecture of the image processing system provided in an embodiment of this application;

[0016] Figure 2 is a schematic diagram of the structure of the electronic device provided in an embodiment of this application;

[0017] Figure 3 is a schematic flowchart of an optional image processing method provided in an embodiment of this application;

[0018] Figure 4 is another optional flowchart of the image processing method provided in the embodiments of this application;

[0019] Figure 5 is a schematic diagram of the image processing model provided in an embodiment of this application;

[0020] Figure 6 is a flowchart illustrating the image processing method implemented through an image processing model according to an embodiment of this application;

[0021] Figure 7 is a flowchart illustrating the training method of the image processing model provided in an embodiment of this application;

[0022] Figure 8 is a schematic diagram of the implementation process of loss calculation for image processing model provided in the embodiment of this application;

[0023] Figure 9 is a product schematic diagram of the image processing method provided in an embodiment of this application;

[0024] Figure 10 is a system flowchart of the training phase provided in an embodiment of this application;

[0025] Figure 11 is a system flowchart of the inference stage provided in an embodiment of this application;

[0026] Figure 12 is a schematic diagram of the basic architecture of the stable diffusion model provided in the embodiments of this application;

[0027] Figure 13 is a schematic diagram of the spatial attention module provided in an embodiment of this application;

[0028] Figure 14 is a schematic diagram of the structure of the multi-level ID injection mechanism network provided in the embodiment of this application;

[0029] Figure 15 is a schematic diagram of the structure of the loss calculation module provided in an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.

[0032] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0033] Before explaining the image processing method provided in the embodiments of this application, the technical terms involved in the embodiments of this application will be explained first.

[0034] (1) Convolutional Neural Network (CNN): A type of feedforward neural network. The artificial neurons in a CNN can respond to a portion of the surrounding units within their coverage area, making it excellent for large-scale image processing. A CNN consists of one or more convolutional layers and fully connected layers, as well as associated weights and pooling layers.

[0035] (2) Latent Diffusion Model (LDM): This includes the Stable Diffusion Model (a variant of the Diffusion Model). The purpose of the Stable Diffusion Model is to eliminate continuous Gaussian noise on the training image. It can be viewed as a series of denoising autoencoders. The Stable Diffusion Model consists of three parts: a Variational Autoencoder (VAE), a U-Net, and a text encoder. The VAE is trained to convert the image into a low-dimensional latent space (i.e., a low-dimensional representation space). The Stable Diffusion Model applies this low-dimensional latent representation during both adding and removing Gaussian noise, and then decodes the final denoised output into the pixel space. During forward diffusion, Gaussian noise is iteratively applied to the compressed latent representation features. Each denoising step is performed by a U-Net architecture containing a residual neural network, obtaining the latent representation by denoising from forward diffusion to backward diffusion. Finally, the decoder in the VAE generates the final output image by transforming the latent representation features back into the pixel space.

[0036] (3) U-Net: U-Net is one of the algorithms for semantic segmentation using fully convolutional networks. The U-Net network is a convolutional neural network architecture that uses a symmetrical U-shaped structure containing compression and expansion paths. The U-Net network has influenced the design of several subsequent segmentation networks to some extent, and its name is derived from its U-shaped form. In the U-Net network, the hidden feature output of each layer represents the feature representation of the input image at different levels of abstraction. These feature representations correspond to feature maps that depict the abstract texture and structural information gradually extracted from the original image. Specifically, the encoder part of the U-Net network gradually reduces the size of the feature map through a series of convolutional layers, max-pooling layers, and batch normalization layers, while increasing the feature depth, i.e., the number of channels in the feature map. This process achieves feature downsampling.

[0037] (4) Contrastive Language-Image Pretraining (CLIP) Model: This is a deep learning model that can understand text and images. In this embodiment, the CLIP model is used as a text encoder. It is a multimodal model that has been trained to match text and images, thereby learning to recognize the content in the image and the language describing the image. The main goal of the CLIP model is to learn to match images and text through contrastive learning, predicting which image belongs to a given text and vice versa. During training, the CLIP model learns to encode images and text into a unified vector space, enabling it to understand the relationship between images and text in both linguistic and visual terms. In this way, the CLIP model can recognize elements such as objects, scenes, and actions in images, and can also understand text related to images, such as labels, descriptions, or titles.

[0038] (5) InsightFace: An open-source 2D and 3D deep face analysis tool, and also an open-source face recognition tool based on deep learning technology. InsightFace provides a complete set of face-related solutions, including face detection, face recognition, and face attribute analysis, with high performance and accuracy. InsightFace's biggest feature is its support for end-to-end face recognition, enabling efficient implementation of a wide variety of face recognition, face detection, and face alignment algorithms, optimized for training and deployment. InsightFace can also be a deep learning-based face recognition system, characterized by high efficiency, accuracy, and scalability. InsightFace utilizes neural network technology to extract facial features from various complex environments, thereby achieving high-precision face matching and recognition.

[0039] (6) Spatial Similarity: Also known as spatial distance, spatial similarity measures the similarity between two vectors by measuring the cosine of the angle between them. The cosine of a 0-degree angle in space is 1, while the cosine of any other angle is no greater than 1, with a minimum value of -1. Therefore, the spatial similarity of two vectors is determined by the cosine of the angle between them, which represents the degree of overlap between their spatial angle and direction. When two vectors have the same direction and high similarity, the cosine similarity value is 1; when the spatial angle between two vectors is 90° and low similarity, the cosine similarity value is 0; when two vectors point in completely opposite directions and are not similar at all, the cosine similarity value is -1. This result is independent of the length of the vectors, only related to their direction. Cosine similarity is usually used in positive space, so the given value is between 0 and 1.

[0040] In related technologies, a common approach is to fine-tune a general stable diffusion model. This involves using a few or dozens of user photos to fine-tune the backbone generative graph model, thus injecting the user's personal image as a concept into the generative graph model. This allows the model to generate personalized user images. The implementation first generates a generic style image without any specified facial features. Then, a faceswap model is used to transform facial features from the user's input image into the generated image (i.e., performing a face swap between the two images) to obtain a personalized image. Furthermore, the stable diffusion model is modified. During the inference phase, the uploaded user image is encoded using a CLIP model to obtain an embedding vector. This embedding vector is then merged with the embedding vector calculated from language units (tokens). The merged vector is then injected as a concept into the generative graph model to generate the final personalized image.

[0041] While using user-uploaded personal images to fine-tune the model can generate personalized images that resemble the user to some extent, this fine-tuning training method requires a certain amount of training data, increasing the difficulty for users and easily leading to a loss of patience. Furthermore, this fine-tuning training method results in a very long training time for the entire model (generally over 20 minutes), wasting computational resources and negatively impacting user experience, thus affecting the effectiveness of practical applications. Using face-swapping to integrate the user's facial features into the final personalized image is a real-time generation method that does not require pre-training. However, this method heavily relies on the performance of the face-swapping model. Also, because it generates personalized images, the faces may contain unrealistic elements (such as monkey hair on a monkey's face). These elements affect the face-swapping effect, resulting in poor integration, an overall unnatural and uncoordinated appearance, and ultimately affecting the quality of the final image. By encoding the face into an embedding vector and injecting it together with the embedding vector calculated from the language unit into the generative graph model as a concept, it is possible to generate a personalized image that is relatively similar to the user's face to a certain extent. However, this method of directly embedding the face into the embedding vector has several drawbacks. First, the concept injection strength is too weak, resulting in a low similarity between the generated personalized image and the user's uploaded image. Second, because it is directly injected as a concept along with other embedding vectors, the semantic understanding weights of these concepts in the generative graph model are equal during the generation process. This leads to mutual interference between the semantic representations of the face embedding vector and other embedding vectors, thus affecting the final result.

[0042] Based on at least one of the aforementioned problems existing in the methods of related technologies, embodiments of this application provide an image processing method. First, an electronic device acquires descriptive text and an original image containing a target object. The original image is encoded to obtain image encoding features, and the descriptive text is encoded to obtain text encoding features. Then, multi-level image residual mapping is performed on the image encoding features to obtain image semantic identification features of the target object in the original image at multiple levels. Based on the image semantic identification features at each level, multi-level text residual mapping is performed on the text encoding features to obtain fusion features. Finally, the fusion features are decoded to obtain a generated image, which includes object features of the target object and descriptive features of the descriptive text. Thus, since the multi-level text residual mapping process integrates the image semantic identifier features at each level, the fused feature obtained after multi-level text residual mapping is a fused feature that considers the image semantic identifier features at multiple levels. In other words, the fused feature incorporates the object features of the target object in the original image at multiple levels. Therefore, using this fused feature to generate an image can improve the image fusion effect, making the overall face of the target object more harmonious in the generated image, thereby improving the image quality of the generated image. Furthermore, the method in this embodiment can directly process the input original image and descriptive text. The input information only includes the original image and descriptive text, eliminating the need for extensive model training and generation steps, thereby reducing the time consumption of the image generation process and greatly saving computational resources.

[0043] Here, we first describe an exemplary application of the image processing device according to the embodiments of this application. This image processing device is an electronic device used to implement an image processing method. In one implementation, the image processing device (i.e., electronic device) provided in the embodiments of this application can be implemented as a terminal or as a server. In one implementation, the image processing device provided in the embodiments of this application can be implemented as any terminal with image display or image data processing functions, such as a laptop, tablet, desktop computer, mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device, intelligent robot, smart home appliance, and smart vehicle device. In another implementation, the image processing device provided in the embodiments of this application can also be implemented as a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of this application. The following will describe an exemplary application of the image processing device as a server.

[0044] Referring to Figure 1, which is a schematic diagram of an optional architecture of the image processing system provided in this application embodiment, the image processing system 10 of this application embodiment includes at least a terminal 100, a network 200, and a server 300. An image processing application is installed on the terminal 100, which can provide image processing functions, such as generating an animated avatar carrying the user's facial features. Alternatively, a game application is installed on the terminal 100, which provides a character generation function, which can generate a character carrying the user's facial features, allowing the user to use the newly generated character in the game.

[0045] The following description uses the installation of an image processing application on a terminal as an example. In this embodiment, server 300 can be a server for the image processing application. Server 300 can constitute the image processing device of this embodiment, that is, the image processing method of this embodiment is implemented through server 300. Terminal 100 is connected to server 300 through network 200, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0046] Referring to Figure 1, when a user wants to generate an animated avatar carrying their facial features, they can input the original image and the descriptive text of the animated avatar into the client of the image processing application. The terminal 100 encapsulates the original image and the descriptive text into an image processing request and sends the image processing request to the server 300 via the network 200. Then, in response to the image processing request, the server 300 encodes the original image and the descriptive text respectively, obtaining corresponding image encoding features and text encoding features. Next, the server 300 performs multi-level image residual mapping on the image encoding features to obtain the image semantic identifier features of the target object in the original image at each level. Based on the image semantic identifier features at each level, the server 300 performs multi-level text residual mapping on the text encoding features to obtain fused features. Finally, the server 300 decodes the fused features to obtain the generated image, which includes the object features of the target object and the descriptive features of the descriptive text; that is, the generated image simultaneously carries the user's object features and the descriptive features of the descriptive text. After obtaining the generated image, the server 300 sends the generated image to the terminal 100, which can then display the generated image on the current interface of the image processing application.

[0047] In some embodiments, the steps in the image processing method can also be executed by the terminal 100. That is, after the terminal 100 acquires the original image and the descriptive text containing the target object, the terminal 100 encodes the original image and the descriptive text respectively to obtain image encoding features and text encoding features; and the terminal 100 performs multi-level image residual mapping on the image encoding features and multi-level text residual mapping on the text encoding features; finally, the terminal 100 decodes the fusion features obtained after multi-level text residual mapping to obtain a generated image that simultaneously carries the object features of the target object and the descriptive features of the descriptive text, and displays the generated image.

[0048] The image processing method provided in this application embodiment can also be implemented based on a cloud platform and through cloud technology. For example, the server 300 mentioned above can be a cloud server. The original image and descriptive text are encoded separately by the cloud server. Alternatively, the image encoding features can be mapped using a multi-level image residual mapping method by the cloud server. Alternatively, the text encoding features can be mapped using a multi-level text residual mapping method by the cloud server. Alternatively, the fused features obtained after multi-level text residual mapping can be decoded by the cloud server to obtain a generated image that simultaneously carries the object features of the target object and the descriptive features of the descriptive text.

[0049] In some embodiments, a cloud storage device may also be included, which can store the original image and descriptive text, or a generated image that carries both object features of the target object and descriptive features of the descriptive text, in the cloud storage. In this way, the required original image or descriptive text can be retrieved from the cloud storage during the next image processing.

[0050] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device shown in Figure 2 can be an image processing device, which includes at least one processor 310, a memory 350, at least one network interface 320, and a user interface 330. The various components in the image processing device are coupled together through a bus system 340. It is understood that the bus system 340 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 340 also includes a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus system 340 in Figure 2.

[0051] Processor 310 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor can be a microprocessor or any conventional processor, etc. User interface 330 includes one or more output devices 331 and one or more input devices 332 that enable the presentation of media content. Memory 350 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Memory 350 may optionally include one or more storage devices physically located remote from processor 310. Memory 350 includes volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), and volatile memory can be random access memory (RAM). The memory 350 described in this application embodiment is intended to include any suitable type of memory. In some embodiments, memory 350 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0052] Operating system 351 includes system programs for handling various basic system services and performing hardware-related tasks, such as framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks; network communication module 352 is used to reach other computing devices via one or more (wired or wireless) network interfaces 320, exemplary network interfaces 320 include: Bluetooth, WiFi, and Universal Serial Bus (USB); input processing module 353 is used to detect and translate user input or interaction from one or more input devices 332.

[0053] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 shows an image processing apparatus 354 stored in memory 350. This image processing apparatus 354 can be an image processing device in an electronic device, and it can be software in the form of programs and plug-ins, including the following software modules: acquisition module 3541, encoding module 3542, image residual mapping module 3543, text residual mapping module 3544, and decoding module 3545. These modules are logically related, and therefore can be arbitrarily combined or further divided according to the functions they implement. The functions of each module will be described below.

[0054] In some embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the image processing method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0055] The image processing methods provided in the embodiments of this application can be executed by an electronic device, which can be a server or a terminal. That is, the image processing methods in the embodiments of this application can be executed by a server, by a terminal, or by interaction between a server and a terminal.

[0056] The image processing method provided in this application can be applied to various business scenarios, such as personalized social media content creation, virtual avatar customization, creative advertising and marketing, education and learning, game development, entertainment and film and television production, medical and health, artistic creation, news and media, virtual makeup and clothing design, etc. Examples are given below.

[0057] In the realm of personalized social media content creation, users can generate personalized photos or illustrations by inputting their facial images and personalized text descriptions (such as "on vacation at the beach" or "wearing retro-style clothing"). These generated images can be directly used for sharing on social media, enhancing the appeal and personalization of the content.

[0058] In the field of virtual avatar customization, within virtual reality (VR) or augmented reality (AR) applications, users can combine their facial images with text descriptions (such as "anime style," "anime character," etc.) to generate personalized virtual avatars. This technology can provide users with highly customized virtual characters, enhancing immersion.

[0059] In the field of creative advertising and marketing, advertising agencies can generate targeted advertising images by combining the client's brand image and product description with facial images of the target audience. For example, combining the faces of the target audience with product usage scenarios can generate more persuasive and attractive advertising content.

[0060] In the field of education and learning, teachers can use this technology to combine students' photos with descriptions of historical figures, scientific roles, etc., to generate educational images. For example, combining students' faces with "ancient scientists" or "historical figures" can help students better understand and remember relevant knowledge.

[0061] In the field of game development, game developers can use this technology to generate personalized game characters based on players' facial images and game scene descriptions (such as "medieval knight" or "sci-fi warrior"). This not only enhances player immersion but also saves time and costs for game development.

[0062] In the entertainment and film production fields, this technology can be used to generate concept art for characters or special effects shots. For example, combining an actor's face with a fantasy character description can generate a character image with fantasy effects, providing a reference for special effects production.

[0063] In the medical and health field, this technology can be used to generate medical images that match the patient's condition description. For example, based on the patient's facial image and condition description (such as "skin lesions," "facial fractures," etc.), corresponding medical images can be generated to assist doctors in diagnosis.

[0064] In the field of artistic creation, artists can use this technology to combine their facial images with creative descriptions to generate unique artworks. For example, combining an artist's face with a "surrealist style" or a "futuristic scene" can create imaginative works of art.

[0065] In the news and media field, news outlets can utilize this technology to generate relevant news images based on the content and descriptions of people in news reports. For example, based on descriptions of people and event scenes in news articles, images that meet the reporting requirements can be generated, enhancing the visual impact of the news.

[0066] In the fields of virtual makeup and clothing design, users can combine their facial images with descriptions of clothing or makeup to generate images showing the effect of trying on or wearing the clothes. For example, inputting "retro style clothing" or "natural makeup" can generate corresponding try-on or makeup effects, helping users better choose products that suit them.

[0067] Figure 3 is an optional flowchart of the image processing method provided in an embodiment of this application. The steps shown in Figure 3 will be described below. As shown in Figure 3, taking a server as the execution subject of the image processing method as an example, the method includes the following steps S101 to S105:

[0068] Step S101: Obtain the descriptive text and the original image containing the target object.

[0069] Here, the target object can be a human face or an animal face, etc. Acquiring the original image containing the target object can be either acquiring an original image input by the user through a terminal, or acquiring an original image containing the target object in real time through an image acquisition device. The original image may include one or more target objects, and the multiple target objects included in the original image may be from the same user or different users. In this embodiment, the number of original images is one or more; when there are multiple original images, all of them include the target object.

[0070] In this embodiment, after acquiring the original image, the target object to be processed can be determined from the original image. When determining the target object, if the original image contains multiple objects, one or more target objects can be identified from these multiple objects; if the original image contains only one object, that single object can be identified as the target object. When determining the target object from multiple objects, the user can select the object selected by the user, or the description text can be parsed to determine the specific object from the content described in the description text. Alternatively, the object with the highest percentage among multiple original images can be identified as the target object. Here, percentage refers to the ratio between the number of times the same object appears in multiple original images and the number of images in the multiple original images.

[0071] For example, embodiments of this application can be applied to the following scenario: A user inputs multiple original images through a terminal, all of which contain the face of person A. The server can perform identity recognition on the multiple original images to determine the identity of each person in the multiple original images. Then, person A, which is included in all the multiple original images, is determined as the final target object. Furthermore, based on the facial features of the target object and the input descriptive text, the image processing method provided in this application is used to generate generated text containing the facial features of person A and conforming to the descriptive text.

[0072] For example, embodiments of this application can be applied to the following scenario: A user inputs an original image through a terminal. The server recognizes the original image and identifies the faces of multiple people within it. Therefore, a reminder message can be sent to the terminal, prompting the user to select a target object from the identified faces. Upon receiving the user's selection of a target object, the server can confirm the selected person as the target object and further obtain descriptive text. Then, based on the facial features of the target object and the input descriptive text, the image processing method provided in this application is used to generate text containing the facial features of person A and conforming to the descriptive text.

[0073] Descriptive text refers to the textual description of the overall content of the image to be generated and the imagery of the objects within it. In other words, the imagery of the objects in the final generated image is determined based on the descriptive text. Descriptive text can include, but is not limited to, the following information: facial feature descriptions, scene and background descriptions, character states and actions descriptions, style and artistic effects descriptions, and other personalized descriptions.

[0074] Here, facial feature descriptions can include biometric features, non-biometric features, and expressions and emotions. Biometric features include the shape, size, and location of facial contours, eyebrows, eyes, nose, lips, chin, etc.; non-biometric features include, for example, hair color, length, hairstyle, and facial skin texture, moles, scars, etc.; expressions and emotions can be, for example, smiling, laughing, angry, and sad. Scene and background descriptions can include environmental background, temporal background, and location background. Environmental background describes the scene in which the person is, such as "on vacation at the beach" or "working in the office"; temporal background can be, for example, "at sunset," "under the starry night sky," or "in the morning sunlight"; location background can be, for example, "on the street in city A" or "in front of a skyscraper in region B." Descriptions of a person's state and actions can include action descriptions, posture descriptions, and clothing and accessories. Action descriptions can be, for example, "running," "playing the piano," or "talking with a friend"; posture descriptions can be, for example, "sitting," "standing," or "lying down"; clothing and accessories can be, for example, "wearing retro-style clothing" or "wearing sunglasses." Descriptions of style and artistic effects can include artistic style, color tone and lighting, and special effects. For example, artistic style could be "oil painting style," "comic book style," or "watercolor painting style"; color tone and lighting could be "warm colors," "cool colors," or "strong light and shadow contrast"; and special effects could be "with science fiction elements" or "with a dreamlike effect." Other personalized descriptions can include age and gender, cultural background, and creative and abstract descriptions. For example, age and gender could be "young woman" or "middle-aged man"; cultural background could be "wearing traditional clothing"; and creative and abstract descriptions could be "a blend of human face and natural elements" or "a combination of human face and technological elements."

[0075] The descriptive text can be text information entered by the user through the terminal, or it can be text information obtained after the user enters voice information through the terminal and performs a speech-to-text operation. The descriptive text can also be the preset description information of the selected image after the user selects one image from multiple preset images through the terminal. The descriptive text may include attribute information of the image to be generated, which includes, but is not limited to, at least one of the following: the name, type, and characteristics of the objects in the image to be generated.

[0076] For example, the descriptive text can include the name of the game character, the type of the game character, or the characteristics of the game character such as skills. In this way, the server can generate a new game character that incorporates the facial features of the target object based on the description of the game character in the descriptive text.

[0077] Step S102: Encode the original image to obtain image encoding features; encode the descriptive text to obtain text encoding features.

[0078] Here, encoding the original image refers to image encoding, which is the process of converting the image data of the original image into a data format suitable for further processing and image residual mapping. In other words, it's the process of converting the original image into a form that a computer can understand and process. In this process, image encoding can be performed using an image encoder to obtain the image encoding features of the original image. Image encoding features refer to a feature vector corresponding to the original image and having a first form. This first form refers to a data format that a computer can understand and process; for example, this first form of data can be binary data. Image encoding features include the feature values ​​obtained after converting the pixel values ​​of each pixel in the original image. All the feature values ​​obtained after converting the pixel values ​​of all pixels in the original image constitute the feature vector corresponding to the image encoding features.

[0079] In this embodiment, the encoder in a Variational Autoencoder (VAE) can be used as the image encoder to encode the original image. A VAE is a deep learning model designed to learn the probability distribution of data. Unlike traditional autoencoders, VAEs learn not only the encoded representation of data but also its probability distribution, giving them greater capabilities in generative models and unsupervised learning. A VAE consists of two parts: an encoder and a decoder. The encoder transforms the input image into a representation in a latent space, which in a VAE typically has a much lower dimensionality than the input data. The decoder generates data from the latent space, accepting the latent space representation as input and outputting data similar to the input data. Through the cooperation of the encoder and decoder in the VAE, the image encoding process of the original image is achieved, thereby outputting image-encoded features.

[0080] Encoding the descriptive text refers to text encoding, which is the process of converting the descriptive text into a feature vector of a second form. This second form is one that a computer can understand and further process. The second form can be the same as the first form; for example, the data in the second form can also be in binary form. In implementation, text encoding can be performed using a text encoder to obtain the text-encoded features of the descriptive text. Text-encoded features are feature vectors corresponding to the descriptive text and possessing this second form. These features include the feature values ​​obtained after converting each character in the descriptive text, and all the feature values ​​obtained after converting all the characters in the descriptive text, which together constitute the feature vector corresponding to the text-encoded features.

[0081] In this embodiment, the CLIP model can be used as a text encoder to encode descriptive text. The CLIP model works by using contrastive learning to distinguish between positive and negative samples; essentially, it achieves this through classification. When used as a text encoder, the CLIP model is a transformer-based model that extracts text features.

[0082] Step S103: Perform multi-level image residual mapping on the image coding features to obtain the image semantic identification features of the target object at each level in multiple levels.

[0083] Here, multi-level image residual mapping refers to taking image encoded features as input and performing image residual mapping multiple times sequentially. The multiple image residual mapping processes are connected sequentially. "Multi-level" in multi-level image residual mapping means multiple times; the total number of levels equals the cumulative number of image residual mapping processes. The level i of any given level of image residual mapping equals the number i of the image residual mapping processes. In other words, the i-th level of image residual mapping is the i-th image residual mapping process.

[0084] Image residual mapping is implemented through a feature mapping process, which includes multiple steps, each with corresponding processing parameters. The feature mapping process for the input features remains the same across different levels of image residual mapping; that is, each step is identical. The difference lies in the processing parameters involved in each step of the feature mapping process. These processing parameters refer to the weight parameters involved in feature calculation at each corresponding step.

[0085] The input feature is different for different levels of image residual mapping. The input feature for the first level of image residual mapping is the image coding feature. After performing the first level of image residual mapping on the image coding feature, the result of the first level of image residual mapping is obtained. The result of the first level of image residual mapping is then used as the input feature for the second level of image residual mapping, and thus the result of the second level of image residual mapping is obtained. That is, for the process of two adjacent levels of image residual mapping, the result of the previous level of image residual mapping will be used as the input feature of the next level of image residual mapping.

[0086] Image residual mapping refers to the residual mapping of input features. Residual mapping is a concept in deep learning, referring to the calculation of the difference between input and output data; that is, the difference between the input data after passing through the network and the data processed by directly applying a function (such as the identity function). In deep learning, as the number of network layers increases, the vanishing or exploding gradient problem is often encountered, making the network difficult to train. To solve this problem, residual learning emerged. The idea behind residual learning is that the network needs to learn not only the direct mapping from input data to output data, but also the residual mapping from input data to output data. Specifically, if input data x is mapped to output data y through the network, then the actual learning goal of the network is to learn the residual function f(x) = yx, rather than directly learning y.

[0087] In this embodiment, the residual mapping process can be implemented using a ResNet module. Within ResNet, this idea is specifically implemented as a Residual Block, where an Identity Shortcut Connection is added after each convolutional layer, directly adding the input data to the output of the convolutional layer. In this way, the network learns f(x) = yx instead of y, simplifying the learning task and allowing for deeper networks. The concept of residual mapping provides a method for constructing very deep networks while maintaining model training efficiency and performance.

[0088] In implementation, the image residual mapping in this embodiment can be implemented through a first residual mapping module (a ResNet block). This first residual mapping module includes residual mapping functions. Through the residual mapping functions in each first residual mapping module, multi-level image residual mapping is performed on the input image encoding features. That is, different first residual mapping modules are called for different levels of image residual mapping, and each first residual mapping module can implement one feature mapping process. Each first residual mapping module includes multiple residual mapping sub-modules, such as convolutional layers used to implement convolutional processing steps. Each residual mapping sub-module implements a step in the feature mapping process through a residual mapping function, and this residual mapping function includes weight parameters.

[0089] Here's an example illustrating that "the processing parameters involved in each step of the feature mapping process differ when performing image residual mapping at different levels": For any two image residual mappings S1 and S2 at different levels, where image residual mapping S1 corresponds to feature mapping process S11 and image residual mapping S2 corresponds to feature mapping process S21. Image residual mapping S1 is implemented through the first residual mapping module R1, and image residual mapping S2 is implemented through the first residual mapping module R2. The first residual mapping modules R1 and R2 have the same structure, both having three convolutional layers, each used to implement one step in the feature mapping process. The difference between image residual mapping S1 and image residual mapping S2 is that the weight parameters of the convolutional layer Conv11 in the first residual mapping module R1 are different from the weight parameters of the convolutional layer Conv21 in the first residual mapping module R2, the weight parameters of the convolutional layer Conv12 in the first residual mapping module R1 are different from the weight parameters of the convolutional layer Conv22 in the first residual mapping module R2, and the weight parameters of the convolutional layer Conv13 in the first residual mapping module R1 are different from the weight parameters of the convolutional layer Conv23 in the first residual mapping module R2.

[0090] In this embodiment, each level of image residual mapping outputs an image semantic identifier feature. That is, during each level of image residual mapping, after performing residual mapping on the input features through the residual mapping function in the first residual mapping module, an image semantic identifier feature is obtained.

[0091] Image semantic identifiers are identifiers that can identify facial feature information of a target object in an original image. In other words, image semantic identifiers include image features representing local facial locations of a target object or image features representing specific facial types of a target object. For example, local facial locations may be areas around the eyes, mouth, and nose; specific facial types may be the location and shape of facial features, face shape, hair length, hair color, and skin color. In this embodiment, image semantic identifiers are represented in the form of feature vectors. Image semantic identifiers at different levels have different feature sizes, where the feature size can be a vector dimension. That is, image semantic identifiers at different levels are image features with different vector dimensions used to represent the facial features of the target object. Image semantic identifiers at different vector dimensions represent different local facial regions of the target object, or image semantic identifiers at different vector dimensions represent different specific facial types of the target object.

[0092] Image semantic labeling features at multiple levels include: image semantic labeling features at each level, where each level of image residual mapping yields one image semantic labeling feature, and the level of this image semantic labeling feature is the same as the level at which the corresponding image residual mapping was performed. For example, the first level of image residual mapping yields an image semantic labeling feature with level one (i.e., the first-level image semantic labeling feature), the second level of image residual mapping yields an image semantic labeling feature with level two (i.e., the second-level image semantic labeling feature), and so on.

[0093] For example, image semantic identifier features at one level include image features used to characterize the area around the eyes of the target object, while image semantic identifier features at another level include image features used to characterize the area around the mouth of the target object; or, image semantic identifier features at one level include image features used to characterize the face shape of the target object, while image semantic identifier features at another level include image features used to characterize the hair color of the target object.

[0094] Step S104: Perform multi-level text residual mapping on the text encoding features sequentially.

[0095] Here, multi-level text residual mapping refers to performing multiple text residual mappings sequentially using text encoding features as input. The multiple text residual mapping processes are connected sequentially. "Multi-level" in multi-level text residual mapping also refers to the number of times the text residual mapping process is performed. The input to the first-level text residual mapping is the text encoding features. After performing the first-level text residual mapping on the text encoding features, the result of the first-level text residual mapping is obtained. This result is then used as the input to the second-level text residual mapping, and so on. In other words, for two adjacent levels of text residual mapping, the result of the previous level serves as the input to the next level.

[0096] Text residual mapping refers to performing residual mapping on the encoded features of input text. In this embodiment, the residual mapping process can be implemented using a ResNet residual mapping module. In implementation, the text residual mapping in this embodiment can be implemented using a second residual mapping module (also a ResNet block). This second residual mapping module includes residual mapping functions, and through the residual mapping functions in each second residual mapping module, multi-level text residual mapping is achieved on the encoded features of the input text.

[0097] In this embodiment, the number of levels in the multi-level text residual mapping is the same as the number of levels in the multi-level image residual mapping. Between each two adjacent levels of text residual mapping, the text residual mapping result obtained from the previous level of text residual mapping is fused with the image semantic identifier features of the same level. Furthermore, when performing the next level of text residual mapping, text residual mapping is performed on the text encoding features and the feature fusion result after feature fusion. Here, the image semantic identifier features of the corresponding level refer to the current feature fusion level being the same as the level of the image residual mapping that generated the image semantic identifier features. That is, a feature fusion is performed after each level of text residual mapping, and the input of this feature fusion includes the image semantic identifier features obtained after image residual mapping of the same level and the text encoding features.

[0098] For example, both multi-level image residual mapping and multi-level text residual mapping have 6 levels. The 6 levels of image residual mapping are connected sequentially. The input to the first level image residual mapping is the image encoding feature; the input to the second level image residual mapping is the image semantic identifier feature output from the first level; the input to the third level image residual mapping is the image semantic identifier feature output from the second level, and so on, thus achieving multi-level image residual mapping. For multi-level text residual mapping, a feature fusion process is connected between each of the 6 sequentially connected text residual mappings. That is, a feature fusion process is performed between the first and second level text residual mappings; a feature fusion process is performed between the second and third level text residual mappings, and so on, thus achieving multi-level text residual mapping. For each feature fusion process, the input information for the first feature fusion is the text residual mapping result output after the first-level text residual mapping and the image semantic labeling features output after the first-level image residual mapping; the input information for the second feature fusion is the text residual mapping result output after the second-level text residual mapping and the image semantic labeling features output after the second-level image residual mapping, and so on, achieving multiple feature fusions. Furthermore, for multi-level text residual mapping, the input for the first text residual mapping is the text encoding feature; the input for the second text residual mapping is the fused feature output after the first feature fusion and the text encoding feature; the input for the third text residual mapping is the fused feature output after the second feature fusion and the text encoding feature, and so on, achieving multi-level text residual mapping.

[0099] In this embodiment, image semantic identifier features are identifier features that can identify the facial feature information of the target object in the original image. That is, image semantic identifier features include image features used to characterize the facial features of the target object. Multi-level image residual mapping can obtain image features used to characterize the facial features of the target object in multiple dimensions. Furthermore, in multi-level text residual mapping, the image semantic identifier features obtained after multi-level image residual mapping are fused into the text encoding features. In other words, image features used to characterize the facial features of the target object in multiple dimensions are injected into the text encoding features. This allows for comprehensive and multi-faceted injection of the facial features of the target object into the text encoding features, enabling sufficient feature fusion between the facial features of the target object and the input descriptive text. Consequently, in the subsequent image generation process, a generated image that fully integrates the object features of the target object and the descriptive features of the descriptive text can be generated. This means injecting the facial semantics of the target object into the image generation process, guiding the generation of the final image's image through this facial semantics. This maximizes the preservation of the target object's style and ensures the granularity of the generated details, improving the overall quality of the final generated image.

[0100] Step S105: Decode the fused features to obtain the generated image.

[0101] In this embodiment, a decoder can be used to decode the fusion features obtained after multi-level text residual mapping. For example, the decoder can be a decoder in a VAE (Visual Encryption Object). The decoder in the VAE transforms the latent representation of the fusion features back into the original data space, thereby generating a new data sample similar to the original data. In other words, the decoder in the VAE converts the fusion features into image-like data to obtain the final generated image. Meanwhile, since the fusion features are image features that combine multiple dimensions of the original image to characterize the overall appearance of the target object, as well as text encoding features describing the text, the generated image obtained by decoding based on the fusion features will simultaneously carry the object features of the target object and the descriptive features of the descriptive text.

[0102] In this embodiment, the generated image is an image that includes both the object features of the target object and the descriptive features of the descriptive text. That is, the generated image obtained after decoding the fused features simultaneously carries both the object features of the target object and the descriptive features of the descriptive text. The object features of the target object refer to the overall facial features of the target object in the original image. These overall facial features include both image features characterizing the complete facial region of the target object and image features characterizing various specific facial types of the target object. The complete facial region refers to the entire face of the target object within the original image; specific facial types can include features such as facial feature position, facial feature shape, face shape, hair length, hair color, and skin color.

[0103] To facilitate understanding, the differences between object features, image coding features, and image semantic identifier features are explained below: Image coding features refer to the features obtained by encoding the original image, which represent the original image. Image coding features include all features in the original image. Image semantic identifier features refer to the identifier features that can identify the facial feature information of the target object in the original image. Image semantic identifier features include image features used to represent the facial features of the target object at a local location or image features representing specific types of facial features of the target object. Object features refer to the overall facial features of the target object in the original image. Object features include image features used to represent the facial features of the complete facial region of the target object, as well as image features used to represent specific types of facial features of the target object. Therefore, object features, image coding features, and image semantic identifier features are three completely different types of features.

[0104] The descriptive features of descriptive text refer to the features obtained after extracting the text content from the descriptive text and applying them to the final generated image. These features correspond to a portion of the text content, specifically the portion of the text content that can be applied to the final generated image. For example, if the descriptive text is "Please generate an image of a boy with multicolored, ink-wash-like skin against a dark background," then the descriptive features of the descriptive text are the features corresponding to the portion of the text content "dark background, multicolored, ink-wash-like skin, boy." In other words, the descriptive features of the descriptive text are the feature vector obtained after feature extraction from the portion of the text content "dark background, multicolored, ink-wash-like skin, boy."

[0105] To facilitate understanding, the difference between text-encoded features and descriptive features of the descriptive text is explained here: Text-encoded features refer to feature vectors corresponding to the descriptive text in a form that computers can understand and further process (i.e., the second form mentioned above). In other words, they are feature vectors obtained by encoding the entire text content of the descriptive text. Descriptive features of the descriptive text, on the other hand, are feature vectors obtained by first extracting the text content of the descriptive text to obtain a portion of the text content used in the final generated image, and then encoding that portion. Therefore, descriptive features of the descriptive text contain some of the text features of the descriptive text, while text-encoded features contain all of the text features of the descriptive text.

[0106] The image processing method provided in this application involves an electronic device encoding the original image and descriptive text separately. Then, it performs multi-level image residual mapping on the image encoding features to obtain the image semantic identifier features of the target object in the original image at each of the multiple levels. Based on the image semantic identifier features at each level, it sequentially performs multi-level text residual mapping on the text encoding features to obtain fused features. Decoding these fused features yields a generated image that simultaneously carries the object features of the target object and the descriptive features of the descriptive text. Furthermore, using these fused features after multi-level text residual mapping to generate the final image, since the multi-level text residual mapping process fuses the image semantic identifier features at each level, the resulting fused features are fused features that consider the image semantic identifier features at multiple levels. In other words, the fused features incorporate the object features of the target object in the original image at multiple levels. Therefore, using these fused features to generate the image improves the image fusion effect, making the overall face of the target object more harmonious in the generated image, thereby improving the image quality of the generated image. Furthermore, the method in this application embodiment can directly process the input original image and descriptive text. The input information consists only of the original image and descriptive text, without the need for too many complex model training and generation steps, thereby improving image generation efficiency and greatly saving computational resources in the image generation process.

[0107] The following examples illustrate the application scenarios of the image processing method provided in this application, using image generation and game scenarios as examples respectively.

[0108] Scenario 1: In an image generation scenario, the image processing system includes at least a terminal and a server. The terminal has an image processing application installed, which provides image processing functions, including generating animated avatars that incorporate the user's facial features. Users can use the image processing function in this application to create vivid animated images from uploaded images containing their faces. In this scenario, the server acts as the backend server for the image processing application. Users input an original image containing their face through the application's client, or by acquiring the original image in real-time through the terminal. Simultaneously, they input descriptive text about the animated avatar they want to generate through the image processing function's interface. This descriptive text can include the animation's shape, name, and characteristics. The terminal can encapsulate the original image and descriptive text in an image processing request and send it to the server. The server can respond to the image processing request and, using the image processing method provided in this embodiment, accurately and efficiently generate an image carrying the user's object features and the descriptive text.

[0109] Scenario 2: In a game scenario, the image processing system includes at least a terminal and a server. The terminal has a game application installed, which provides a character generation function. This function can generate characters carrying the user's facial features, allowing the user to use these newly generated characters in the game. Users can use the character generation function in the game application to create new game characters from uploaded images containing their own faces. In this scenario, the server is the backend server of the game application. The user inputs an original image containing their face through the game application's client, or the terminal captures the user's original image in real time. Simultaneously, the user inputs descriptive text for the game character they want to generate on the character generation function's interface. For example, the descriptive text can include the game character's appearance, name, and characteristics. The terminal can encapsulate the original image and descriptive text in an image processing request and send the request to the server. The server can respond to the image processing request and, using the image processing method provided in this embodiment, accurately and efficiently generate a game character carrying the user's object features and the descriptive text.

[0110] The image processing method of this application embodiment will be described below with reference to the above scenario one.

[0111] Figure 4 is another optional flowchart of the image processing method provided in the embodiment of this application. As shown in Figure 4, the method includes the following steps S201 to S210:

[0112] Step S201: The terminal receives the original image and descriptive text of the target object input by the user.

[0113] In this embodiment, the image processing application client provides an image input component and a text input component. Users can input raw images using the image input component provided by the client. For example, they can select an image from locally stored images on the terminal as the raw image input, or they can call the terminal's image acquisition module to acquire an image in real time as the raw image input. This image acquisition module can be, for example, the terminal's camera. Simultaneously, users can also input descriptive text using the text input component provided by the client. The input methods for the text input component can include: text editing input, text-to-speech input, or text copying input, etc. This embodiment does not limit the input methods for the text input component.

[0114] In step S202, the terminal generates an image processing request based on the original image and the descriptive text.

[0115] In this embodiment of the application, the terminal can encapsulate the original image and descriptive text into an image processing request.

[0116] In step S203, the terminal sends an image processing request to the server.

[0117] In this embodiment, after generating an image processing request, the terminal can send the image processing request to the server. The server responds to the image processing request sent by the terminal and, based on the original image and descriptive text carried in the image processing request, generates a final image that includes the object features of the target object and the descriptive features of the descriptive text. In some embodiments, the terminal can send the image processing request using protocols such as HTTP or WebSocket.

[0118] In step S204, the server calls the text encoder to encode the description text and obtain the text encoding features.

[0119] In this embodiment, a text encoder can be selected before text encoding. For example, the text encoder in the CLIP model can be used. The text encoder in the CLIP model is based on the Transformer architecture and can convert descriptive text into high-dimensional embedding vectors, capturing the semantic information of the descriptive text. Alternatively, in addition to the CLIP model, other pre-trained text encoders can be selected, such as Bidirectional Encoder Representations from Transformers (BERT) or Generative Pre-trained Transformer (GPT) models, but adjustments and optimizations are required based on the specific task.

[0120] When a text encoder encodes descriptive text, its workflow includes tokenization, embedding, and encoding. Tokenization involves breaking down the input descriptive text into words or phrases and converting them into a format that the model can understand. For example, the word "dog" might be broken down into a series of tokens. Embedding involves converting each token into a vector in a high-dimensional space, called an embedding vector. These embedding vectors capture the semantic information of the word. Encoding uses a Transformer-based architecture to process these embedding vectors and employs a self-attention mechanism to understand the context of the descriptive text. The encoding process generates a series of text feature vectors (i.e., text-encoded features) that capture the semantic information implied by the entire descriptive text.

[0121] In step S205, the server calls the image encoder to encode the original image and obtain the image encoding features.

[0122] In this embodiment, before image encoding, an image encoder can be selected. The role of the image encoder is to convert the input raw image into feature vectors so that subsequent image generation models can use these feature vectors to generate new images. The image encoders in this embodiment include: Transformer-based encoders, convolutional neural networks, or CLIP image encoders. Transformer-based encoders can be, for example, MAETok, which uses a Vision Transformer (ViT) architecture to process image patches in the raw image and generate latent representations. Convolutional neural networks can be, for example, Residual Neural Networks (ResNet) or Visual Geometry Groups (VGG), which can extract high-level features from the raw image. The CLIP image encoder is the image encoder in the CLIP model; the CLIP image encoder can convert the raw image into feature vectors aligned with text features.

[0123] In step S206, the server performs multi-level image residual mapping on the image coding features through multiple hidden layers in the pre-trained image identifier extractor to obtain the image semantic identifier features of the target object in the original image at multiple levels.

[0124] In this embodiment, when performing first-level image residual mapping on the image coding features, the first hidden layer among multiple hidden layers is used to perform first-level image residual mapping on the image coding features to obtain the first-level image semantic identifier features of the target object in the original image. When performing N-level image residual mapping on the image coding features, the Nth hidden layer among multiple hidden layers is used to perform N-level image residual mapping on the (N-1)th-level image semantic identifier features to obtain the N-level image semantic identifier features of the target object in the original image. N is an integer greater than 1. That is, for each two adjacent levels of image residual mapping in the entire image residual mapping process, the image coding features are sequentially image residual mapped by two adjacent hidden layers in the image identifier extractor. Furthermore, in each two adjacent levels of image residual mapping, the output features of the hidden layer during the previous level of image residual mapping are used as the input features of the hidden layer during the next level of image residual mapping.

[0125] In some embodiments, the image identifier extractor includes multiple hidden layers connected in sequence. Each hidden layer is a first residual mapping module, which includes a residual mapping function. In step S206, the image encoding features are subjected to multi-level image residual mapping through multiple hidden layers in the pre-trained image identifier extractor to obtain the image semantic identifier features of the target object in the original image at multiple levels. This can be achieved by calling the residual mapping function in each of the first residual mapping modules in the multiple hidden layers to perform multi-level image residual mapping on the image encoding features, thereby obtaining the image semantic identifier features of the target object in the original image at multiple levels. That is, when performing N-level image residual mapping on the (N-1)th level image semantic identifier features through the Nth hidden layer of the image identifier extractor to obtain the image semantic identifier features of the target object at the Nth level, this is done by calling the residual mapping function in the first residual mapping module of the Nth hidden layer to perform N-level image residual mapping on the (N-1)th level image semantic identifier features, thus obtaining the image semantic identifier features of the target object at the Nth level.

[0126] For example, the image identifier extractor can be implemented as a U-Net network, a convolutional neural network for image segmentation, comprising multiple residual mapping modules (i.e., the first residual mapping module) connected sequentially. The U-Net network in this embodiment has an encoder-decoder structure, meaning it has a symmetrical contracting path. This symmetrical contracting path is similar to a convolutional neural network, progressively reducing the feature map size while increasing the number of channels through convolutional layers and max-pooling layers. This process helps extract contextual information from the image. Furthermore, the U-Net network has a symmetrical expanding path, a unique design feature of the U-Net network. This expanding path progressively increases the feature map size while decreasing the number of channels through upsampling (such as interpolation or transposed convolution). This process helps combine contextual information with image details.

[0127] In this embodiment, after the U-Net network performs multi-level image residual mapping on the image coding features, the sizes of the image semantic labeling features of the target object in the original image at different levels are not the same. Furthermore, due to the symmetrical contraction and expansion paths of the U-Net network, the sizes of the obtained image semantic labeling features at multiple levels exhibit a trend of first decreasing and then increasing. Specifically, the size of the image semantic labeling feature at the last level among the multiple levels is the same as the size of the input image coding features.

[0128] In step S207, the server uses a pre-trained feature segmentation network to perform multi-level text residual mapping on the text encoding features based on the image semantic labeling features at each level, thereby obtaining fused features.

[0129] Here, the number of levels in the multi-level text residual mapping is the same as the number of levels in the multi-level image residual mapping. Between each two adjacent levels of text residual mapping, a feature fusion is performed on the text residual mapping result obtained from the previous level of text residual mapping and the corresponding level of image semantic labeling features. Furthermore, when performing the next level of text residual mapping, text residual mapping is performed on the text encoding features and the feature fusion result. In other words, for each two adjacent levels of text residual mapping in the multi-level text residual mapping, firstly, the text residual mapping result obtained from the previous level of text residual mapping and the image semantic labeling features at the previous level are obtained; then, feature fusion is performed on the text residual mapping result and the image semantic labeling features at the previous level to obtain the feature fusion result; finally, text residual mapping is performed on the feature fusion result to obtain the text residual mapping result of the next level of text residual mapping in the two levels of text residual mapping.

[0130] In this embodiment of the application, the feature segmentation network includes a plurality of second residual mapping modules connected in series, and a spatial attention module is connected between each pair of adjacent second residual mapping modules; the number of second residual mapping modules is the same as the number of first residual mapping modules in the image identifier extractor.

[0131] In some embodiments, when performing multi-level text residual mapping on text encoded features through a pre-trained feature segmentation network, the text encoded features may be mapped at the first level using the first second residual mapping module among multiple second residual mapping modules to obtain the first-level representation features describing the text after the first-level text residual mapping. Then, the first spatial attention module in the feature segmentation network fuses the first-level representation features after the first-level text residual mapping with the image semantic label features output by the first first residual mapping module in the image label extractor to obtain the first feature fusion result. Then, the second second residual mapping module sequentially performs second-level text residual mapping on the first feature fusion result, and the second spatial attention module performs feature fusion on the second-level representation features after the second-level text residual mapping with the image semantic label features output by the second first residual mapping module in the image label extractor to obtain the second feature fusion result, and so on, until the processing of each second residual mapping module and spatial attention module in the feature segmentation network is completed, resulting in the final fused feature.

[0132] Therefore, after performing the M-th level text residual mapping on the text encoding features, the M-th spatial attention module in the feature segmentation network fuses the M-th level representation features after the M-th level text residual mapping with the image semantic label features output by the M-th first residual mapping module in the image label extractor to obtain the M-th feature fusion result; M is an integer greater than 0. When performing the K-th level text residual mapping on the text encoding features, the K-th second residual mapping module in multiple second residual mapping modules performs the K-th level text residual mapping on the fusion result of the text encoding features and the K-1-th feature output by the K-1-th spatial attention module to obtain the K-th level representation features describing the text after the K-th level text residual mapping; K is an integer greater than 1. Finally, the representation features obtained after the last level text residual mapping of the multi-level text residual mapping are determined as the fused features obtained after the multi-level text residual mapping.

[0133] In some embodiments, the dimension of the Mth level representation feature is the same as the dimension of the image semantic label feature output by the Mth first residual mapping module. That is, the size of the image semantic label feature output by the Mth first residual mapping module in the image label extractor is the same as the size of the Mth level representation feature output by the Mth second residual mapping module in the feature segmentation network. In other words, the structure of the sequentially connected first residual mapping modules in the image label extractor is the same as the structure of the multiple second residual mapping modules in the feature segmentation network. The only difference between the image label extractor and the feature segmentation network is that in the feature segmentation network, a spatial attention module is connected between every two adjacent second residual mapping modules. The multiple first residual mapping modules in the image label extractor correspond one-to-one with the multiple second residual mapping modules in the feature segmentation network. It should be noted that the network parameters of the first residual mapping module and the corresponding second residual mapping module can be the same or different.

[0134] When performing feature fusion on the M-th level representation feature after the M-th level text residual mapping and the image semantic label feature output by the M-th first residual mapping module in the image label extractor through the M-th spatial attention module in the feature segmentation network, it can be achieved in the following way: First, the M-th level representation feature and the image semantic label feature output by the M-th first residual mapping module are concatenated along the channel dimension through the M-th spatial attention module in the feature segmentation network to obtain concatenated features; then, the concatenated features are pooled to obtain pooled features; then, the pooled features are convolved to obtain the weight matrix between the M-th level representation feature and the image semantic label feature output by the M-th first residual mapping module; finally, feature fusion calculation is performed on the concatenated features based on the weight matrix to obtain the M-th feature fusion result.

[0135] The spatial attention module takes as input the representation features and the image semantic labeling features output by the first residual mapping module. The spatial attention module comprises a feature concatenation layer, a pooling layer, a convolutional layer, and a feature fusion layer, connected sequentially. Multiple spatial attention models in the feature segmentation network have the same structure, but their model parameters can be the same or different. For each spatial attention module, the input representation features and the image semantic labeling features output by the first residual mapping module are sequentially processed through feature concatenation, pooling, convolution, and fusion.

[0136] Here, the M-th level representation feature and the image semantic identifier feature output by the M-th first residual mapping module are concatenated along the channel dimension to obtain a concatenated feature. This can be achieved by performing a feature concatenation operation on the M-th level representation feature and the image semantic identifier feature output by the M-th first residual mapping module. Feature concatenation refers to combining two different features together for further analysis and processing. Feature concatenation can be performed across multiple dimensions; in this embodiment, it can be performed on the last dimension (usually the channel dimension). Consider two tensors of shape (B, H, W, C1) and (B, H, W, C2), where B is the batch size, H is the height, W is the width, and C1 and C2 are the number of channels. In this case, through the feature concatenation operation, these two tensors can be concatenated along the channel dimension to form a new tensor of shape (B, H, W, C1+C2).

[0137] It's important to note that the channel dimension mentioned above refers to the number of feature channels in a Convolutional Neural Network (CNN) that represent features and semantic image representations. For example, for semantic image representations, each feature channel represents a specific feature in the original image, such as edges, textures, or shapes. Generally, in CNNs, since feature maps are output by convolutional layers, they represent different regions of the image in spatial dimensions (width and height) and different features in channel dimensions. For instance, a common CNN architecture might start with an input image containing three channels, each representing a color (red, green, blue) in the RGB color space, and then pass through multiple convolutional and pooling layers to ultimately output a feature map with hundreds or even thousands of channels. The channel dimension of the feature map is crucial for the network's learning and performance; a high channel dimension means the network can learn more complex feature representations.

[0138] Pooling can be performed on the concatenated features by applying max pooling and average pooling respectively, resulting in max-pooled and average-pooled features. These max-pooled and average-pooled features are then designated as the pooled features. Max pooling and average pooling are two commonly used pooling operations used to reduce the size of the feature map while retaining key information. Max pooling selects the maximum value from each local region (such as a 2x2 or 3x3 window) of the input feature map as the output for that region. This operation helps preserve salient features in the image, such as edges and corners, because these features typically have high pixel values. Max pooling can improve the localization of features because it only focuses on the highest pixel value in each region. Average pooling, on the other hand, averages all pixel values ​​in each local region to obtain the output for that region. Average pooling helps smooth feature maps, reduce noise, and preserve the overall structural information of the image. It can also improve the generalization ability of features because it focuses on the average trend of pixel values ​​in each region.

[0139] In some embodiments, the feature segmentation network includes a plurality of second residual mapping modules connected in series. The plurality of second residual mapping modules include: at least one second residual mapping module for downsampling text-encoded features, and at least one second residual mapping module for upsampling text-encoded features; wherein the number of second residual mapping modules for downsampling text-encoded features is the same as the number of second residual mapping modules for upsampling text-encoded features; and at least one second residual mapping module for upsampling text-encoded features is connected after at least one second residual mapping module for downsampling text-encoded features.

[0140] Downsampling reduces the resolution and data volume of an image by decreasing the number of pixels, thereby saving storage space and improving processing speed. It is typically used to remove redundant pixels, reducing data transmission and storage costs while preserving key image information. During downsampling, a certain number of pixels can be selectively retained based on specific requirements, usually achieved through image segmentation or thresholding. Upsampling (also known as resampling or upscaling), on the other hand, increases the resolution and data volume of an image by increasing the number of pixels. It is typically used to enlarge smaller images or repair damaged images, making them clearer and more realistic. During upsampling, interpolation algorithms can be used to estimate the color or features between adjacent pixels, thus generating more pixels.

[0141] In some embodiments, before performing the first-level text residual mapping on the text encoding features, a preset random noise generation algorithm can be used to generate random representation features for the feature segmentation network. Then, the random representation features and the text encoding features are used as input features and input into the feature segmentation network. Thus, when performing the first-level text residual mapping on the text encoding features through the first second residual mapping module among multiple second residual mapping modules, it can be achieved as follows: the first-level text residual mapping is performed on the text encoding features and the random representation features through the first second residual mapping module to obtain the first-level representation features.

[0142] Random noise generation algorithms are used to generate random and unpredictable data. They are often used to simulate random phenomena in the real world or to increase the diversity and complexity of data. This application's embodiments can employ any of the following random noise generation algorithms to generate random representation features for the feature segmentation network: uniformly distributed random number generation algorithm, Gaussian distributed random number generation algorithm, white noise generation algorithm, gradient noise generation algorithm, and fractal noise generation algorithm.

[0143] Here, the generation principle of the uniformly distributed random number generation algorithm is that the probability of each value appearing within a specified interval is equal. The random numbers generated by the uniformly distributed random number generation algorithm are uniformly distributed within the specified interval, without obvious clustering or regularity. The uniformly distributed random number generation algorithm is suitable for scenarios that require random selection, random sampling, or simulation of uniform random events, such as random sorting or random sampling. It can be implemented by the linear congruential generator (LCG), and the formula of the linear congruential generator is as follows (1): X n+1 =(aX n +c) mod m (1)

[0144] Among them, X n X is the current random number. n+1 This is the next random number; a, c, and m are constants. By choosing appropriate parameters, a uniformly distributed pseudo-random number sequence can be generated.

[0145] The random numbers generated by the Gaussian distribution random number generation algorithm follow a normal distribution, meaning that most values ​​are concentrated around the mean, and the further away from the mean, the lower the probability of occurrence. The random numbers generated by the Gaussian distribution random number generation algorithm have obvious clustering, concentrated around the mean, which is in stark contrast to the uniformity of uniform distribution random numbers. The Gaussian distribution random number generation algorithm can be applied to fields such as simulation of natural phenomena, statistical analysis, and signal processing, such as generating noise signals and simulating measurement errors. The Gaussian distribution random number generation algorithm can be implemented through the Box-Muller transform and the central limit theorem. Among them, the Box-Muller transform generates a Gaussian distribution random number by using two independent uniformly distributed random numbers. The formula for the Box-Muller transform is as follows (2):

[0146] U1 and U2 are two independent uniformly distributed random numbers.

[0147] The central limit theorem states that by adding multiple independent uniformly distributed random numbers and then scaling and shifting them appropriately, a Gaussian distributed random number can be approximately generated.

[0148] White noise is an ideal random signal whose power spectral density is uniform across all frequencies, meaning it has the same energy at every frequency. White noise generation algorithms can approximate white noise by generating uniformly or Gaussian distributed random number sequences, because these sequences have a flat power spectral density in the frequency domain. White noise focuses on frequency domain characteristics, i.e., the same energy distribution across all frequencies, while uniformly and Gaussian distributed random numbers focus on numerical distribution characteristics.

[0149] Gradualized noise (such as Perlin noise) is a smoothed random noise generated by assigning random gradient vectors to grid points and interpolating between these points. The specific steps of a gradualized noise generation algorithm include: first, generating random gradient vectors at the grid points; then, calculating the vector from the input point to the grid point; calculating the noise value through the dot product; and finally, smoothing the noise value through interpolation. The noise generated by the gradualized noise generation algorithm has spatial smoothness and continuity, unlike the completely random nature of white noise, and also differs from the numerical distribution characteristics of uniformly and Gaussian distributed random numbers.

[0150] Fractal noise is generated by superimposing noise signals of different frequencies and amplitudes to produce noise with fractal characteristics. For example, multiple Perlin noise signals of different frequencies can be superimposed, with the amplitude of each noise signal decreasing by a certain proportion, thereby generating noise with fractal characteristics. The noise generated by fractal noise generation algorithms has fractal characteristics, that is, it has a similar structure at different scales, which is different from the characteristics of white noise, uniformly distributed and Gaussian distributed random numbers, and also different from the smoothness of gradually changing noise.

[0151] In step S208, the server decodes the fusion features obtained after multi-level text residual mapping to obtain the generated image.

[0152] Here, the generated image includes object features of the target object and descriptive features of the descriptive text. For the process of decoding the fused features, please refer to the relevant explanation of step S105 above; this embodiment will not repeat it further.

[0153] In step S209, the server generates an image and sends it to the terminal.

[0154] In step S210, the terminal displays the generated image on the current interface.

[0155] The image generation method provided in this application can directly process the input original image and descriptive text. The input information consists only of the original image and descriptive text, eliminating the need for extensive model training and generation steps. This reduces the time consumption of the image generation process and significantly saves computational resources. Furthermore, the multi-level text residual mapping process also integrates the image semantic identifier features of the target object at multiple levels. In other words, the text encoding features of the descriptive text are fully integrated with relevant image features representing the facial features of the target object across multiple dimensions. This allows for a more comprehensive fusion of the target object's facial features, improving the image fusion effect and image quality of the generated image.

[0156] In some embodiments, the above image processing method can be implemented by a pre-trained image processing model; as shown in FIG5, the image processing model 50 includes a backbone generation network 51 and a feature label injection network 52; the backbone generation network 51 includes at least a text encoder 511 and a feature segmentation network 512, and the feature label injection network 52 includes an image encoder 521 and an image label extractor 522. The input of the backbone generation network 51 is the descriptive text 501, and the input of the feature label injection network 52 is the original image 502.

[0157] Referring to Figure 6, the above image processing method can be implemented through the following steps S301 to S307:

[0158] In step S301, the electronic device inputs the original image of the target object into the feature identification injection network.

[0159] Here, the electronic device can be either a server or a terminal, and this application embodiment does not limit it.

[0160] In step S302, the electronic device encodes the original image by injecting feature identifiers into the image encoder in the network to obtain image encoding features.

[0161] In step S303, the electronic device performs multi-level image residual mapping on the image coding features through the image identifier extractor in the feature identifier injection network, thereby obtaining the image semantic identifier features of the target object in the original image at multiple levels.

[0162] Here, the image identifier extractor includes multiple hidden layers, each of which is a first residual mapping module, and the first residual mapping module includes a residual mapping function. Therefore, the image encoding features can be subjected to multi-level image residual mapping through multiple first residual mapping modules to obtain the image semantic identifier features output by each first residual mapping module.

[0163] In step S304, the electronic device inputs the descriptive text into the backbone generation network.

[0164] In step S305, the electronic device performs text encoding on the descriptive text through the text encoder in the backbone generator network to obtain text encoding features.

[0165] Here, the text encoder in the backbone generative network can be the text encoder in the CLIP model, or it can be other pre-trained text encoders.

[0166] In step S306, the electronic device performs multi-level text residual mapping on the text encoding features through the feature segmentation network in the backbone generation network to obtain fused features.

[0167] Here, the feature segmentation network comprises multiple serially connected second residual mapping modules, with a spatial attention module connected between each pair of adjacent second residual mapping modules; the number of second residual mapping modules is the same as the number of first residual mapping modules in the image identifier extractor. Therefore, the number of levels in the multi-level text residual mapping is the same as the number of levels in the multi-level image residual mapping. The input of each spatial attention module includes the output of the preceding second residual mapping module connected to it and the output of the corresponding first residual mapping module. Except for the first second residual mapping module, whose input is text-encoded features, the inputs of the remaining second residual mapping modules all include text-encoded features and the output of the preceding spatial attention module connected to it. The output of the last second residual mapping module constitutes the fused features obtained after multi-level text residual mapping.

[0168] In step S307, the electronic device decodes the fusion features obtained after multi-level text residual mapping through a decoder to obtain the generated image.

[0169] In some embodiments, the image processing model may further include a decoder (not shown in Figure 5). This decoder may belong to the backbone generation network of the image processing model or exist independently of the backbone generation network. That is, the decoder may be included in the backbone generation network or located outside of the backbone generation network; this application embodiment does not impose any limitations. In the implementation process, the decoder may be a decoder in a VAE. The decoder decodes the fused features obtained after multi-level text residual mapping to obtain a generated image that simultaneously carries the object features of the target object and the descriptive features of the descriptive text.

[0170] Based on the above-described image processing model implementation scheme, this application further provides a training method for the image processing model. This training method can be implemented through a model training module, which can be a module in an electronic device. That is, the execution entity of the image processing model training method can be a server or a terminal. This application embodiment uses a server as an example to illustrate the method, as shown in Figure 7. The image processing model training method includes the following steps S401 to S406:

[0171] Step S401: Input the sample image from the sample data into the feature identifier injection network, and encode the sample image through the image encoder in the feature identifier injection network to obtain the sample image encoded features.

[0172] Step S402: The image identifier extractor in the feature identifier injection network performs multi-level image residual mapping on the encoded features of the sample image to obtain the semantic identifier features of the sample target object in the sample image at each level in multiple levels.

[0173] Step S403: Input the sample description text from the sample data into the backbone generation network, and use the text encoder in the backbone generation network to encode the sample description text to obtain the sample text encoding features.

[0174] Step S404: The sample text encoding features are mapped using a multi-level text residual mapping through the feature segmentation network in the backbone generation network to obtain the sample fusion features.

[0175] In this embodiment of the application, the process of processing the input sample image and sample description text by the image processing model during training is the same as the process of processing the input original image and description text by the trained image processing model when using the image processing model. Therefore, the specific implementation process of steps S401 to S404 will not be described again in this embodiment of the application.

[0176] Step S405: Based on the semantic identifier features and sample fusion features of the sample images, the loss of the image processing model is calculated to obtain the loss result of the image processing model.

[0177] In some embodiments, referring to Figure 8, the loss calculation of the image processing model in step S405 can be achieved through the following steps S4051 to S4053:

[0178] Step S4051: Based on the sample fusion features and sample images, perform basic loss calculation on the image processing model to obtain the basic loss result.

[0179] In this embodiment of the application, the basic loss calculation of the image processing model can be achieved in the following way: First, the sample fusion features are decoded to obtain a sample generated image that carries both the object features of the sample target object and the descriptive features of the sample description text; then, the mean square error between the sample generated image and the sample image is determined as the basic loss result.

[0180] Here, the mean squared error (MSE) between the generated image and the sample image is used to measure the difference between the two images. It can be calculated as follows: Assume there are two images A and B, with the same size (H×W), where the value of each pixel in the image ranges from [0, 255]. The pixel value of image A is a... ij The pixel value of image B is b ij , where i and j represent the pixel positions. Then, the formula for calculating the MSE loss is as follows (3):

[0181] The smaller the MSE loss value, the more similar the generated image and the sample image are; the larger the MSE loss value, the greater the difference between the generated image and the sample image.

[0182] Step S4052: Based on the semantic labeling features of the sample images and the sample fusion features, calculate the image semantic labeling loss of the image processing model to obtain the image semantic labeling loss result.

[0183] In this embodiment of the application, the image semantic labeling loss calculation of the image processing model can be achieved in the following way: First, the semantic labeling features of the sample image are decoded to obtain a decoded image; then, the sample fusion features are decoded to obtain a sample generated image that carries both the object features of the sample target object and the descriptive features of the sample description text; then, the decoded image is face detection encoded to obtain a first encoding vector, and the sample generated image is face detection encoded to obtain a second encoding vector; finally, the cosine distance between the first encoding vector and the second encoding vector is determined as the image semantic labeling loss result.

[0184] Here, the first encoding vector is the embedding vector obtained after face detection encoding of the decoded image, and the second encoding vector is the embedding vector obtained after face detection encoding of the sample generated image. Face detection encoding refers to performing face detection encoding on both the decoded image and the sample generated image using a face encoding feature model. This face encoding feature model can be a face detection network in face analysis networks (such as the RetinaFace network). Using this face detection network, facial information in the image can be directly identified, and the positions of facial features can be located. Simultaneously, based on the located face positions, the entire face region can be encoded into a face embedding vector, resulting in the first and second encoding vectors.

[0185] Step S4053: The sum of the basic loss result and the image semantic labeling loss result is determined as the loss result of the image processing model.

[0186] Step S406: Update the model parameters in the image processing model based on the loss results to obtain the trained image processing model.

[0187] In this embodiment of the application, when updating the model parameters in the image processing model based on the loss result, the parameters in both the backbone generation network and the feature labeling injection network are updated. That is, the model parameters in the backbone generation network and the feature labeling injection network are updated synchronously based on the obtained total loss result, thereby realizing the training of the entire image processing model.

[0188] In this embodiment, when training the image processing model, both the basic loss result and the image semantic labeling loss result are considered simultaneously. Thus, features are extracted based on the face detection network to calculate the image semantic labeling loss result, which, together with the basic loss result in the backbone generation network, forms the comprehensive loss result of the entire system. The sample images predicted by the backbone generation network during training are input into the face detection network. Simultaneously, the user's sample images are processed through multi-level image residual mapping to extract the sample image semantic labeling features. Then, the decoded images are input into the face detection network to calculate the embedding vectors of the facial features in the images. These two embedding vectors are used to calculate the difference between the predicted image and the user's real appearance during model training. This increases the image processing model's perception of the user's real facial features, thereby improving the model's ability to reconstruct the user's facial features and increasing the similarity between the final generated image and the user's real appearance.

[0189] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0190] This application provides an image processing method for real-time generation of personalized avatar images based on a multi-level face ID injection mechanism and face ID loss. The multi-level face ID injection mechanism involves injecting image semantic features at multiple levels into a multi-level text residual mapping process. In actual inference, this method can generate a personalized avatar image with high similarity to a specified real face in real time, based on the user-input facial features (i.e., descriptive text). This application improves the similarity between the face and real facial features in the generated personalized avatar, and also maintains the final personalized avatar style representation without affecting the embedding of facial information, thereby improving the overall quality of the personalized avatar. This application embodiment constructs a multi-layered face identifier injection mechanism (i.e., image semantic identifier feature injection). It calculates the features of the face information in the original image at multiple levels, extracts the semantic information of facial features (i.e., the aforementioned image semantic identifier features) from multiple dimensions, and then inputs this information into the backbone generative network at multiple levels. Using the fusion mechanism described in this application embodiment, namely the spatial attention mechanism, the extracted semantic information of facial features is adaptively fused with the target image style information in the backbone generative network. This balances the style of the personalized image and the user's facial similarity. After fusion, the backbone generative network can generate a stylized image with the user's facial features. This application embodiment requires pre-training the model using a batch of image data to obtain real-time generation capabilities during the inference stage. In the pre-training stage of this application embodiment, in addition to calculating the loss function between the stable diffusion model and the original training image, the generated image and the input original image are also input into a face loss calculation module for separate calculation of the loss function for face similarity. This weighted combination of the two loss functions can gradually improve the backbone generation network's ability to restore facial features in the input image and its ability to represent the desired personalized image style during the pre-training process, thereby improving the quality of the final personalized image and the face similarity.

[0191] The method described in this application can be implemented in the field of personalized image generation. Users provide a portrait image to the system, and based on the user's specified style, a highly similar and stylized personalized image can be generated in real time. Firstly, this method can be applied to cameras that generate personalized images. Users can take a selfie or upload a facial image, and then, according to the style features selected by the user, a personalized portrait poster photo in the desired style can be automatically generated in real time, providing more functionality for such camera applications. Secondly, this method can also be applied to game operation and promotion scenarios. By pre-training a stable diffusion model for game-style text-to-image generation and replacing the backbone generation network in this application, this method can generate game-style character images with the user's facial features in real time. This provides users with sharing and game poster creation functions, offering more gameplay and methods for game operation and promotion. Furthermore, the embodiments of this application can also be applied to the creation of user-generated content (UGC) on the theme of film and television dramas. By utilizing the model in the embodiments of this application, film and television drama character images with user appearance features can be directly generated for users, helping users to have more ways to play in the UGC creation platform.

[0192] Therefore, the problems to be solved and the technical effects to be achieved by the embodiments of this application include, but are not limited to, at least one of the following: First, the embodiments of this application provide a real-time personalized image generation technology solution. The model has been fully pre-trained on a large dataset. In actual application deployment, users only need to upload their portrait photo. Without additional model fine-tuning, a personalized image photo with high similarity to the portrait photo and high style and texture can be generated for the user in real time. Therefore, the embodiments of this application require less user data (only one photo), reducing the difficulty of system operation. Furthermore, without the need for additional fine-tuning, the overall time consumption is greatly reduced, and resource costs are saved, significantly improving the user experience. In addition, in order to improve the analysis capability of user faces and to balance the style of the image to be generated, the embodiments of this application construct a multi-layer face ID injection mechanism, which can inject the user's personal image into the final personalized image generation model (i.e., the backbone generation network) at multiple semantic levels. This process fully analyzes the facial features of the user-input images, and the multi-layered injection mechanism allows the model to deeply understand the real characteristics of the user's image, thereby improving the similarity between the final generated personalized image and the real facial features of the input user image. Furthermore, injecting the semantics of the user image into the generation model to guide image generation maximizes style preservation and ensures the granularity of generated details, improving the overall quality of the final image. In addition, to improve facial similarity in image generation, this embodiment also constructs a FaceID Loss (corresponding to the image semantic labeling loss result mentioned above) based on features extracted from a face analysis network, which, together with the loss function in the backbone generation network, forms the overall system loss update. By inputting the images predicted by the generated graph during training into the face analysis network, and simultaneously inputting the user image after it has been decoded using the ID mechanism, the embedding vectors of the facial features are calculated. These two embedding vectors are used to calculate the difference between the predicted image and the user's real appearance during training. This increases the model's perception of the user's real facial features, thereby improving the model's ability to reconstruct the user's facial features and increasing the similarity between the final generated image and the original user portrait.Finally, in order to balance the adaptive compatibility issues between face information injection and other semantic features in the model in this embodiment, a spatial attention mechanism is used in the multi-level face ID injection process to adaptively fuse face ID information and generated graph features. This method ensures that the user's face ID information is passed to the generated graph model while also adaptively fusing with the target generated image style information, without causing conflicts in information interaction, thus affecting the generation of the target image style representation, and further improving the similarity of the person and the overall image quality in the final personalized image.

[0193] Figure 9 is a schematic diagram of the image processing method provided in this application embodiment. As shown in Figure 9, the original image 901 uploaded by the user is processed by the image processing method of this application embodiment to obtain a personalized generated image, such as the first generated image 902, the second generated image 903, the third generated image 904, and the fourth generated image 905 in Figure 9. The first generated image 902, the second generated image 903, the third generated image 904, and the fourth generated image 905 have different personalized images. In particular, this application embodiment constructs a real-time generation method for personalized image text-based images based on a multi-level face ID injection mechanism and face ID loss during personalized image generation. This method uses a multi-level information injection mechanism to extract face ID information (i.e., image semantic identifier features) and injects the extracted face ID information into the backbone generation network. At the same time, during the training process, loss is calculated by combining the face ID loss function under large-scale face data, so that the model can learn to generate images that are highly similar to a specified face. In practical applications, users only need to upload a portrait photo, and then, based on the selected style, a high-quality, highly similar personalized image can be generated in real time.

[0194] As shown in Figure 10, the system flowchart for the training phase, this embodiment of the application consists of three parts in terms of the overall system construction: a backbone generation network 101 based on a stable diffusion model, a multi-level ID injection mechanism network 102 (i.e., a feature identifier injection network), and a loss calculation module 103 (i.e., a FaceID Loss calculation module). First, the backbone generation network 101 serves as the base model for the entire system, and all innovative modules and mechanisms revolve around the calculation of this base model. The backbone generation network 101 incorporates the computational information of the multi-level ID injection mechanism network, so a spatial attention module 1012 is added to the U-Net network 1011 within the backbone generation network. As shown in Figure 10, the U-Net network 1011 includes multiple second residual mapping modules 1013. The second residual mapping modules 1013 are used to perform multi-level text residual mapping on the text encoding features, where the text encoding features are obtained by the text encoder 1010 after encoding the descriptive text. Meanwhile, the input to the U-Net network 1011 may also include random representation features 1014, which are random noise generated by a preset random noise generation algorithm. The U-Net network 1011 in the backbone generative network is used to adaptively fuse face ID information and generated map features to achieve a balanced selection between face ID information and original style information.

[0195] The input to the multi-level ID injection mechanism network 102 is the original image 1021 (a portrait image of a face) uploaded by the user. In order to ensure that the dimensionality of the intermediate structural features is the same as that of the U-Net network 1011 in the backbone generation network, the U-Net network architecture 1022, which is the same as the stable diffusion model, is directly used when constructing the multi-level ID injection mechanism network 102. The U-Net network architecture 1022 includes multiple second residual mapping modules 1023, which are used to perform multi-level image residual mapping on the image encoding features to obtain the image semantic identification features of the target object in the original image 1021 at each level (i.e., the above-mentioned multiple levels). The image encoding features are obtained by encoding the original image 1021 through the decoder 1024 in the VAE. After the image encoding features of the original image 1021 are input into the U-Net network architecture 1022, the hidden feature output of each layer in the U-Net network architecture 1022 can be calculated and then passed to the spatial attention module 1012 in the backbone generative network 101 for feature semantic fusion, thereby improving the compatibility between face and style.

[0196] As shown in Figure 11, the system flowchart of the reasoning stage shows that in the actual reasoning stage (i.e., in actual application), users only need to upload a photo 1101 of the required portrait, and then, based on the selected input style, a high-quality, highly similar personalized style image 1102 can be generated in real time.

[0197] Please refer to Figure 10. After the loss calculation module is deployed in the multi-level ID injection mechanism network 102, the final network output of the U-Net network architecture 1022 in the multi-level ID injection mechanism network 102 is a latent representation. This latent representation is the hidden space feature obtained by the model after calculating the original input image 1021. Then, the decoder 104 in the VAE is used to decode this latent representation to generate an image that is highly similar to the original input image. The image and the generated image 1036 from the backbone generation network (here, the generated image is obtained by decoding the fusion features output by the U-Net network 1011 in the backbone generation network through the decoder 105 in the VAE) are respectively input into the face analysis network 1031 and the face analysis network 1032 (both face analysis networks 1031 and 1032 can be InsightFace networks) for face localization and encoding. This allows the face encoding features 1033 injected by the multi-level ID injection mechanism network 102 and the face encoding features 1034 in the generated image to be calculated. Then, the spatial distance between the face encoding features 1033 and the face encoding features 1034 can be calculated. This spatial distance is used as the face ID loss 1035 (corresponding to the above image semantic labeling loss result), and together with the calculated MSE loss 1037 of the backbone generation network (corresponding to the above basic loss result), they form the loss function of the entire model, thereby improving the face similarity in the generated image. Among them, the MSE loss 1037 is the loss value obtained by performing basic MSE loss calculation on the generated image 1036 and the pre-provided training target image 1038.

[0198] The structure and data processing of the backbone generation network 101, the multi-level ID injection mechanism network 102, and the loss calculation module 103 in this embodiment will be described below with reference to the system flowcharts shown in Figures 10 and 11.

[0199] First, the backbone generation network 101 will be described. The backbone generation network 101 is constructed based on the stable diffusion model. It adds a spatial attention fusion mechanism, specifically a spatial attention module 1012, to the U-Net network 1011. The basic architecture of the stable diffusion model is shown in Figure 12. The stable diffusion model consists of an encoder 121 in a VAE, a U-Net network 122, and a decoder 123 in a VAE. The U-Net network 122 includes multiple sequentially connected residual mapping modules 1221.

[0200] In the stable diffusion model, the encoder 121 and decoder 123 in the VAE need to be trained first. This allows the encoder 121 to compress the input image 120, followed by a diffusion operation in the latent space (i.e., using the U-Net network 122 as the diffusion model to diffuse the latent representation obtained after compression by the encoder 121). Finally, the decoder 123 in the VAE restores the feature vector obtained after the diffusion operation to the original pixel space, thus obtaining the final image 124. This method is called perceptual compression in the stable diffusion model. This method of compressing high-dimensional features to low dimensions and then performing diffusion operations in the low-dimensional space has universality. In this embodiment, the U-Net network 122 in the stable diffusion model is input with random representation features 1211 obtained after compressing the image 120 and text-encoded features obtained after text encoding the descriptive text. The text encoder 125 can be used to encode the input descriptive text. The random representation features are obtained after compressing the image 120. In some embodiments, random noise can be generated using a pre-defined random noise generation algorithm, and this random noise can be determined as a random representation feature. A diffusion operation is performed in the latent space. The main process of the diffusion operation is not significantly different from the standard diffusion model, and the diffusion model used can be implemented as a time-conditional U-Net network 122. However, an important aspect is that the diffusion model introduces conditional mechanisms for the diffusion operation, achieving multimodal training through cross-attention, thus enabling conditional image generation tasks to be implemented as well.

[0201] Based on the above-mentioned stable diffusion model architecture, the backbone generation network 101 provided in this application embodiment can be constructed based on the above-mentioned stable diffusion model. The backbone generation network 101 constructed based on the stable diffusion model can also be called a latent diffusion model (LDM). The LDM model has three main components: VAE, U-Net network (i.e., the above-mentioned feature segmentation network), and text encoder.

[0202] The VAE (Variational Autoencoder) in the LDM model consists of two main parts: an encoder and a decoder. In the LDM model (i.e., a backbone generative network), the encoder in the VAE transforms the input image into a low-dimensional latent representation (i.e., random representation features), which serves as the input to the U-Net network in the LDM model. The decoder in the VAE does the opposite, transforming the latent representation back into the image.

[0203] In some embodiments, the LDM model may include two structures: first, having both an encoder and a decoder in the VAE; second, having an encoder in the VAE but no decoder. That is, the decoder in the VAE can be separated from the LDM model and connected after it. In this way, the encoder in the VAE within the LDM model can convert the input image into a low-dimensional latent representation, and this latent representation can be input into the U-Net network in the LDM model. The U-Net network then outputs the fused features obtained after multi-level text residual mapping.

[0204] The U-Net network in the LDM model also consists of an encoder and a decoder. Both the encoder and decoder are composed of residual mapping blocks (ResNet) (wherein, the ResNet in the U-Net network of the LDM model is the second residual mapping module mentioned above). The encoder in the U-Net network compresses the image representation into a low-resolution image, and the decoder in the U-Net network decodes the low-resolution image back into a high-resolution image. To prevent the U-Net network from losing important information during downsampling, a shortcut connection is usually added between the downsampling ResNet of the encoder and the upsampling ResNet of the decoder in the U-Net network. That is, a cross-attention layer can be added to the U-Net network of the stable diffusion model to adjust the output of the text encoder. Here, the cross-attention layer (i.e., the spatial attention module mentioned above) is added between the ResNet of the encoder and the decoder in the U-Net network.

[0205] The text encoder in the LDM model transforms the input text prompt (i.e., the descriptive text mentioned above) into an embedding space that the U-Net network can understand. This text encoder is a transformer-based encoder that maps the labeled sequence of the text prompt to a latent text embedding sequence (i.e., text-encoded features), thereby using the text prompt to obtain a better expected output. The text-encoded features output by the text encoder, along with the random representation features output by the encoder in the VAE, are fed into the U-Net network. The U-Net network outputs a fused feature obtained after multi-level text residual mapping.

[0206] When the LDM model is the first structure described above (having both an encoder and a decoder in the VAE), the decoder in the VAE is connected after the U-Net network, and is used to convert the latent representation of fused features back into an image. When the LDM model is the second structure described above (having an encoder in the VAE but not a decoder in the VAE), the image processing model can have a decoder in the VAE, and the decoder in the VAE is connected after the LDM model, and is used to convert the latent representation of fused features output by the U-Net network in the LDM model back into an image.

[0207] During training, the stable diffusion model uses the mean squared error (MSE) loss function as the update function mechanism for the entire system. Furthermore, this embodiment embeds a spatial attention module as a feature fusion module into the backbone generative network built on the stable diffusion model. This spatial attention module is primarily designed to ensure that the injection of multi-level face IDs does not conflict with the original style concept semantic information, thus achieving compatibility between face similarity and style representation.

[0208] Figure 13 is a schematic diagram of the spatial attention module provided in this application embodiment. As shown in Figure 13, the spatial attention module receives the image semantic identification feature 131 (faceid feature) output by the multi-level ID injection mechanism network and the basic features 132 of each layer in the U-Net network in the backbone generation network. Since the multi-level ID injection mechanism network and the U-Net network in the backbone generation network are of the same dimension, the generated feature sizes are the same. After the image semantic identification feature 131 and the basic features 132 are concatenated in the channel dimension, the concatenated features are generated. Then, the max pooling feature and the average pooling feature are calculated by passing the max pooling layer 134 and the average pooling layer 135 respectively. After passing the max pooling feature and the average pooling feature through a convolutional layer 136, the weight matrix 137 between the image semantic identification feature 131 and the basic features 132 can be calculated. Based on the weight matrix 137, the feature fusion method in the concatenated features can be calculated. In this way, the final feature fusion calculation 138 is performed, and a fused feature of the same dimension as the original basic features can be generated.

[0209] The following section explains the multi-level ID injection mechanism network.

[0210] Figure 14 shows a schematic diagram of the multi-level ID injection mechanism network 102. The purpose of the multi-level ID injection mechanism network 102 is to enable the model to perceive the facial features of the target human face at multiple levels during the training and generation process, providing guidance for the generation of the image, thereby improving the facial similarity and texture of the final generated image.

[0211] The multi-level ID injection mechanism network consists of a U-Net network architecture 1022, which is formed by downsampling and upsampling. Structurally, this U-Net network architecture 1022 is exactly the same as the U-Net network 1011 in the stable diffusion model. Therefore, during the pre-training phase, this module is initialized using the weight parameters of the stable diffusion model. This direct reuse of the U-Net network in the stable diffusion model serves two purposes: first, it makes the calculated image semantic label features more compatible with the stable diffusion model; second, it allows for the injection of multi-level semantic information into the stable diffusion model, ensuring that the injected semantic features at each layer have the same dimension.

[0212] The multi-level ID injection mechanism network 102 takes facial photos of the target portrait as input during training and inference. The implementation process of the multi-level ID injection mechanism network 102 involves first encoding the facial photo 141 through the encoder 1024 in the VAE to generate feature dimensions that the U-Net network architecture 1022 can process. After computation by the U-Net network architecture 1022, the image is then decoded by the decoder in the VAE to reconstruct the original input facial photo. During computation, the output of each layer in the U-Net network architecture 1022 is passed to the spatial attention mechanism in the U-Net network 1011 of the backbone generator network 101. This multi-level transmission of semantic information of the target face to the backbone generator network 101 improves the facial similarity of the final generated image.

[0213] The loss calculation module 103 will be described next.

[0214] The loss calculation module 103 is mainly used to detect faces in the generated images during training. It then uses an encoding network to detect faces in the generated images and calculates the face features into a face embedding vector. The same method is used to calculate the face embedding vector of the input face photo in the multi-level ID injection mechanism network 102. The spatial distance between the two face embedding vectors is then calculated and used as the loss for each face. Essentially, this loss is designed to compare the faces in the generated images from the backbone generation network 101 with the faces in the input face photos, thereby improving the similarity between the faces in the generated images and the faces in the input face photos during model training.

[0215] In this embodiment, the loss calculation module 103 uses face analysis networks 1031 and 1032 (e.g., RetinaFace network) as face encoding feature models. This embodiment can directly reuse the parameter weights of these face analysis networks. Using these face analysis networks, facial information in images can be directly identified, and the positions of facial features can be located. Simultaneously, the entire face region can be encoded into a face embedding vector based on the located facial feature positions.

[0216] As shown in the structural diagram of the loss calculation module in Figure 15, the output of the backbone generation network 101 is first decoded by the decoder 105 in the VAE to generate a generated image 1036 that is very similar to the training target image 1038. Then, the generated image 1036 and the training target image 1038 are used to calculate the basic MSE loss 1037. The loss function obtained after the calculation of the MSE loss 1037 is the basic loss result in the training process of the stable diffusion model. The output calculated by the multi-level ID injection mechanism network 102 is decoded by the decoder 104 in the VAE to obtain the output of the decoder 104 in the VAE. The output of the decoder 104 in the VAE is input into the face analysis network 1031, which performs face detection and encoding to obtain face encoding features 1033. Using the same method, the output of the backbone generation network 101 can also be encoded into face encoding features 1034 by the face analysis network 1032. Then, the spatial cosine distance is calculated for these two face encoding features. This allows us to calculate the distance between the face generated by the backbone generative network (i.e., the face in generated image 1036) and the guiding face, thus generating the face ID loss 1035 (i.e., the image semantic labeling loss result). The image semantic labeling loss result is then added to the base loss result, and the sum is used as the final loss function 151 to guide the gradient update of the entire model.

[0217] In this embodiment, after training on a large dataset, the entire image processing model enables the multi-layered ID injection mechanism network to fully grasp the facial features analysis of the input face photo and the semantic multi-layered representation ability of the face ID. Simultaneously, the entire image processing model learns the ability to generate faces that are compatible with various styles. During the actual inference phase, the loss calculation module is temporarily unnecessary; only the backbone generation network and the trained multi-layered ID injection mechanism network are required. Throughout the inference process, a face photo containing the user's face is input to the multi-layered ID injection mechanism network, and then descriptive text representing the style is input to the backbone generation network. After internal calculation, a personalized portrait image with the user's facial features (e.g., a boy with colorful, watercolor-like skin against a dark background and a specific facial appearance) can be generated.

[0218] The beneficial effects of the embodiments of this application are summarized below: First, the embodiments of this application provide a real-time personalized image generation solution. The image processing model has been fully pre-trained on a large dataset. In actual application deployment, users only need to upload their portrait photo. Without additional model fine-tuning, a personalized image photo with high similarity and high style and texture to the portrait photo can be generated for the user in real time. Therefore, the embodiments of this application require less user data (only one photo), reducing the difficulty of system operation. Furthermore, without additional fine-tuning, the overall time consumption is greatly reduced, and resource costs are saved, significantly improving the user experience. Second, in order to improve the analysis capability of user portrait faces and to balance the style of the generated image, the embodiments of this application construct a multi-layered ID injection mechanism, which can inject the user's face photo into the final backbone generation model at multiple semantic levels. This process allows for thorough analysis of the facial features in a user's photo, and the multi-layered injection mechanism enables the backbone generation model to deeply understand the real characteristics of the user's image, thereby improving the similarity between the final generated personalized image and the user's real facial features in the photo. Furthermore, injecting the semantics of the user's image into the backbone generation model to guide the image generation maximizes style preservation and ensures the granularity of generated details, improving the overall quality of the final image. Finally, to further enhance facial similarity in the generated image, this embodiment also constructs a process for calculating the face ID loss based on features extracted from the face analysis network, which, together with the loss function in the backbone generation model, constitutes the overall system loss. By inputting the images predicted by the generated graph during training into the face analysis network, and simultaneously inputting the decoded image of the user's personal image after ID calculation, the embedding vectors of facial features are calculated. These two embedding vectors are used to calculate the difference between the predicted image and the user's real appearance during training. This increases the image processing model's perception of the user's real facial features, thereby improving the model's ability to reconstruct the user's facial features and increasing the similarity between the final generated image and the original user portrait. Finally, to balance the adaptive compatibility between face information injection and other semantic features in the model, this embodiment uses a spatial attention module to adaptively fuse ID and generated graph features during multi-level face ID injection. This approach ensures that the user's face ID information is passed to the backbone generation model while adaptively fusing with the style information of the target generated image, avoiding information interaction conflicts that could affect the generation of the target image's style representation. This further improves the similarity of the person and the overall image quality in the final personalized image.

[0219] It should be noted that in the embodiments of this application, the multi-level ID injection mechanism network uses a complete U-Net network structure, which has large model parameters. Therefore, it has high requirements for the video memory capacity of the graphics card during training. Thus, in some embodiments, the U-Net network in the multi-level ID injection mechanism network can be distilled or simplified, but it is still necessary to ensure that the feature dimensions of each layer in the backbone generation network are consistent, so as to reduce the hardware burden during training, improve the training speed, and reduce the convergence time of the entire image processing model.

[0220] It is understood that in the embodiments of this application, if the content involves user information, such as a user's facial image, and if it involves data related to user information or enterprise information, when the embodiments of this application are applied to specific products or technologies, it is necessary to obtain the user's permission or consent, or to obfuscate this information to eliminate the correspondence between this information and the user; and the collection and processing of related data should strictly comply with the requirements of relevant national laws and regulations when applied in practice, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0221] The following continues to describe an exemplary structure of the image processing device 354 provided in this application embodiment as a software module. In some embodiments, as shown in FIG2, the image processing device 354 includes: an acquisition module 3541 configured to acquire descriptive text and an original image having a target object; an encoding module 3542 configured to encode the original image to obtain image encoding features; and to encode the descriptive text to obtain text encoding features; an image residual mapping module 3543 configured to perform multi-level image residual mapping on the image encoding features to obtain image semantic identification features of the target object at each of the multiple levels; a text residual mapping module 3544 configured to perform multi-level text residual mapping on the text encoding features based on the image semantic identification features at each level to obtain fusion features; wherein the number of levels of the multi-level text residual mapping is the same as the number of levels of the multi-level image residual mapping; and a decoding module 3545 configured to decode the fusion features to obtain a generated image; the generated image includes object features of the target object and descriptive features of the descriptive text.

[0222] In some embodiments, the text residual mapping module 3544 is further configured to: for each adjacent two-level text residual mapping in the multi-level text residual mapping, obtain the text residual mapping result obtained from the previous level text residual mapping in the two-level text residual mapping, and obtain the image semantic identification features under the previous level; perform feature fusion on the text residual mapping result and the image semantic identification features under the previous level to obtain the feature fusion result; perform text residual mapping on the feature fusion result to obtain the text residual mapping result of the next level text residual mapping in the two-level text residual mapping.

[0223] In some embodiments, the image residual mapping module 3543 is further configured to: perform a first-level image residual mapping on the image coding features through the first hidden layer in the pre-trained image identifier extractor to obtain the first-level image semantic identifier features of the target object; and perform an Nth-level image residual mapping on the (N-1)th-level image semantic identifier features through the Nth hidden layer in the image identifier extractor to obtain the Nth-level image semantic identifier features of the target object; where N is an integer greater than 1.

[0224] In some embodiments, the hidden layer is a first residual mapping module, which includes a residual mapping function; the image residual mapping module 3543 is further configured to: perform N-level image residual mapping on the image semantic identifier features at the (N-1)th level by calling the residual mapping function in the first residual mapping module of the Nth hidden layer, thereby obtaining the image semantic identifier features of the target object at the Nth level.

[0225] In some embodiments, the text residual mapping module 3544 is further configured to: perform multi-level text residual mapping on the text encoding features based on the image semantic label features at each level through a pre-trained feature segmentation network to obtain fused features; wherein, the feature segmentation network includes a plurality of serially connected second residual mapping modules, and a spatial attention module is connected between each pair of adjacent second residual mapping modules; the number of second residual mapping modules is the same as the number of first residual mapping modules in the image label extractor.

[0226] In some embodiments, the text residual mapping module 3544 is further configured to: perform a first-level text residual mapping on the text encoding features through the first second residual mapping module among the plurality of second residual mapping modules to obtain the first-level representation features of the descriptive text; after performing the M-th level text residual mapping on the text encoding features, perform the feature fusion on the M-th level representation features after the M-th level text residual mapping and the image semantic identifier features output by the M-th first residual mapping module through the M-th spatial attention module in the feature segmentation network to obtain the M-th feature fusion result; M is an integer greater than 0; perform the K-th level text residual mapping on the text encoding features and the K-1-th feature fusion result output by the K-th spatial attention module through the K-th second residual mapping module among the plurality of second residual mapping modules to obtain the K-th level representation features of the descriptive text; K is an integer greater than 1; and determine the representation features obtained after the last level text residual mapping as the fused features.

[0227] In some embodiments, the plurality of second residual mapping modules include: at least one second residual mapping module for downsampling the text encoding features, and at least one second residual mapping module for upsampling the text encoding features; wherein the number of second residual mapping modules for downsampling the text encoding features is the same as the number of second residual mapping modules for upsampling the text encoding features; the at least one second residual mapping module for upsampling the text encoding features is connected after the at least one second residual mapping module for downsampling the text encoding features.

[0228] In some embodiments, the text residual mapping module 3544 is further configured to: acquire the input features of the feature segmentation network; the input features include the text encoding features and the random representation features; and perform a first-level text residual mapping on the text encoding features and the random representation features through the first second residual mapping module to obtain the first-level representation features.

[0229] In some embodiments, the text residual mapping module 3544 is further configured to: generate random representation features of the feature segmentation network using a preset random noise generation algorithm; and determine the random representation features and the text encoding features as input features of the feature segmentation network.

[0230] In some embodiments, the dimension of the Mth level representation feature is the same as the dimension of the image semantic identifier feature output by the Mth first residual mapping module; the text residual mapping module 3544 is further configured to: concatenate the Mth level representation feature and the image semantic identifier feature output by the Mth first residual mapping module through the Mth spatial attention module in the feature segmentation network to obtain a concatenated feature; perform pooling processing on the concatenated feature to obtain a pooled feature; perform convolution processing on the pooled feature to obtain a weight matrix between the Mth level representation feature and the image semantic identifier feature output by the Mth first residual mapping module; and perform feature fusion calculation on the concatenated feature based on the weight matrix to obtain the Mth feature fusion result.

[0231] In some embodiments, the text residual mapping module 3544 is further configured to: determine the first number of channels of the M-th level representation feature and the second number of channels of the image semantic identifier feature; and concatenate the M-th level representation feature and the image semantic identifier feature in the channel dimension to obtain the concatenated feature; wherein the number of channels of the concatenated feature is equal to the sum of the first number of channels and the second number of channels.

[0232] In some embodiments, the text residual mapping module 3544 is further configured to: perform max pooling on the spliced ​​features to obtain max pooled features; perform average pooling on the spliced ​​features to obtain average pooled features; and determine the max pooled features and the average pooled features as the pooled features.

[0233] In some embodiments, the encoding module 3542 is further configured to: call an image encoder to perform image encoding on the original image to obtain the image encoding features.

[0234] In some embodiments, the encoding module 3542 is further configured to: call a text encoder to perform text encoding on the description text to obtain the text encoding features.

[0235] In some embodiments, the image semantic identification features include facial features for characterizing the target object; the facial features include at least one of the following: facial feature position, facial feature shape, and face shape.

[0236] In some embodiments, the image processing method is implemented by a pre-trained image processing model; the image processing model includes a backbone generation network and a feature injection network; the backbone generation network includes at least a text encoder and a feature segmentation network, and the feature injection network includes an image encoder and an image identifier extractor.

[0237] In some embodiments, the apparatus further includes a model training module configured to train the image processing model through the following steps: inputting a sample image into the feature identifier injection network, encoding the sample image using an image encoder in the feature identifier injection network to obtain sample image encoded features; performing multi-level image residual mapping on the sample image encoded features using an image identifier extractor in the feature identifier injection network to obtain sample image semantic identifier features for each level of the sample target object in the sample image; inputting sample description text into the backbone generation network, encoding the sample description text using a text encoder in the backbone generation network to obtain sample text encoded features; performing multi-level text residual mapping on the sample text encoded features using a feature segmentation network in the backbone generation network to obtain sample fusion features; calculating the loss of the image processing model based on the sample image semantic identifier features and the sample fusion features to obtain the loss result of the image processing model; and updating the model parameters in the image processing model based on the loss result to obtain the trained image processing model.

[0238] In some embodiments, the model training module is further configured to: perform basic loss calculation on the image processing model based on the sample fusion features and the sample images to obtain a basic loss result; perform image semantic labeling loss calculation on the image processing model based on the sample image semantic labeling features and the sample fusion features to obtain an image semantic labeling loss result; and determine the sum of the basic loss result and the image semantic labeling loss result as the loss result of the image processing model.

[0239] In some embodiments, the model training module is further configured to: decode the sample fusion features to obtain a sample generated image; the sample generated image includes the object features of the sample target object and the descriptive features of the sample description text; and determine the mean square error between the sample generated image and the sample image as the basic loss result.

[0240] In some embodiments, the model training module is further configured to: decode the semantic labeling features of the sample image to obtain a decoded image; decode the sample fusion features to obtain a sample generated image; perform face detection encoding on the decoded image to obtain a first encoding vector; perform face detection encoding on the sample generated image to obtain a second encoding vector; and determine the cosine distance between the first encoding vector and the second encoding vector as the image semantic labeling loss result.

[0241] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0242] This application provides a computer program product including executable instructions, which are computer instructions stored in a computer-readable storage medium. When the processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the electronic device performs the method described in this application.

[0243] This application provides a storage medium storing executable instructions. When the executable instructions are executed by a processor, they cause the processor to execute the method provided in this application, such as the method shown in FIG3. In some embodiments, the storage medium may be a computer-readable storage medium, such as ferromagnetic random access memory (FRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disk-read-only memory (CD-ROM); or it may be various devices including one or any combination of the above-mentioned memories.

[0244] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0245] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file containing other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0246] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. An image processing method, the method being performed by an electronic device, the method comprising: Obtain the descriptive text and the original image containing the target object; The original image is encoded to obtain image encoding features; The descriptive text is encoded to obtain text encoding features; Multi-level image residual mapping is performed on the image encoding features to obtain the image semantic identification features of the target object at each of the multiple levels; Based on the image semantic identifier features at each level, multi-level text residual mapping is performed on the text encoding features to obtain fused features; wherein, the number of levels of the multi-level text residual mapping is the same as the number of levels of the multi-level image residual mapping. The fused features are decoded to obtain a generated image; the generated image includes the object features of the target object and the descriptive features of the descriptive text.

2. The method according to claim 1, wherein, The step of performing multi-level text residual mapping on the text encoding features based on the image semantic identifier features at each level includes: For each two adjacent levels of text residual mapping in the multi-level text residual mapping, obtain the text residual mapping result obtained from the previous level of text residual mapping in the two levels of text residual mapping, and obtain the image semantic labeling features under the previous level. The text residual mapping result and the image semantic labeling features of the previous level are fused to obtain the feature fusion result; The feature fusion result is subjected to text residual mapping to obtain the text residual mapping result of the second level of text residual mapping in the two-level text residual mapping.

3. The method according to claim 1 or 2, wherein, The step of performing multi-level image residual mapping on the image encoding features to obtain the image semantic identification features of the target object at each level in multiple levels includes: By using the first hidden layer in the pre-trained image identifier extractor, the image encoding features are subjected to first-level image residual mapping to obtain the first-level image semantic identifier features of the target object; The image semantic identifier features at level N are obtained by performing image residual mapping on the image semantic identifier features at level N-1 through the Nth hidden layer in the image identifier extractor; N is an integer greater than 1.

4. The method according to any one of claims 1 to 3, wherein, The hidden layer is a first residual mapping module, which includes a residual mapping function; The step of performing Nth-level image residual mapping on the (N-1)th level image semantic identifier features through the Nth hidden layer of the image identifier extractor to obtain the Nth level image semantic identifier features of the target object includes: By calling the residual mapping function in the first residual mapping module of the Nth hidden layer, the Nth level image semantic identifier features of the (N-1)th level are subjected to Nth level image residual mapping to obtain the Nth level image semantic identifier features of the target object.

5. The method according to any one of claims 1 to 4, wherein, The text encoding features are mapped using multi-level text residual mapping based on the image semantic identifier features at each level to obtain fused features, including: By using a pre-trained feature segmentation network, based on the image semantic labeling features at each level, multi-level text residual mapping is performed on the text encoding features to obtain fused features; The feature segmentation network includes multiple second residual mapping modules connected in series, with a spatial attention module connected between each pair of adjacent second residual mapping modules; the number of second residual mapping modules is the same as the number of first residual mapping modules in the image identifier extractor.

6. The method according to any one of claims 1 to 5, wherein, The pre-trained feature segmentation network performs multi-level text residual mapping on the text encoding features based on the image semantic labeling features at each level to obtain fused features, including: The first second residual mapping module among the plurality of second residual mapping modules performs first-level text residual mapping on the text encoding features to obtain the first-level representation features of the descriptive text; After performing the Mth level text residual mapping on the text encoding features, the Mth level representation features after the Mth level text residual mapping and the image semantic labeling features output by the Mth first residual mapping module are fused through the Mth spatial attention module in the feature segmentation network to obtain the Mth feature fusion result; M is an integer greater than 0. The Kth second residual mapping module among the plurality of second residual mapping modules performs Kth-level text residual mapping on the fusion result of the text encoding feature and the K-1th feature output by the (K-1)th spatial attention module to obtain the Kth-level representation feature of the descriptive text; K is an integer greater than 1. The representation features obtained after mapping the last-level text residuals are determined as the fusion features.

7. The method according to any one of claims 1 to 6, wherein, The plurality of second residual mapping modules include: at least one second residual mapping module for downsampling the text encoding features, and at least one second residual mapping module for upsampling the text encoding features; The number of second residual mapping modules used to downsample the text encoding features is the same as the number of second residual mapping modules used to upsample the text encoding features; the at least one second residual mapping module used to upsample the text encoding features is connected after the at least one second residual mapping module used to downsample the text encoding features.

8. The method according to any one of claims 1 to 7, wherein, The first-level text residual mapping of the text encoding features, performed through the first of the plurality of second residual mapping modules, to obtain the first-level representation features of the descriptive text includes: Obtain the input features of the feature segmentation network; the input features include the text encoding features and random representation features; The first-level text residual mapping is performed on the text encoding features and the random representation features through the first-level second residual mapping module to obtain the first-level representation features.

9. The method according to any one of claims 1 to 8, wherein, The step of obtaining the input features of the feature segmentation network includes: The random representation features of the feature segmentation network are generated using a preset random noise generation algorithm. The random representation features and the text encoding features are determined as the input features of the feature segmentation network.

10. The method according to any one of claims 1 to 9, wherein, The dimension of the Mth level representation feature is the same as the dimension of the image semantic identifier feature output by the Mth first residual mapping module; The feature fusion is performed on the Mth-level representation features after the Mth-level text residual mapping and the image semantic labeling features output by the Mth first residual mapping module through the Mth spatial attention module in the feature segmentation network to obtain the Mth feature fusion result, including: The Mth spatial attention module in the feature segmentation network is used to concatenate the Mth level representation feature and the image semantic identifier feature output by the Mth first residual mapping module to obtain the concatenated feature. The spliced ​​features are subjected to pooling processing to obtain pooled features; The pooling features are convolved to obtain the weight matrix between the Mth level representation features and the image semantic identifier features output by the Mth first residual mapping module; Based on the weight matrix, feature fusion calculation is performed on the spliced ​​features to obtain the Mth feature fusion result.

11. The method according to any one of claims 1 to 10, wherein, The concatenation process of the Mth level representation feature and the image semantic identifier feature output by the Mth first residual mapping module to obtain the concatenated feature includes: Determine the number of the first channels of the M-th level representation feature and the number of the second channels of the image semantic identifier feature; In the channel dimension, the M-th level representation feature and the image semantic identifier feature are concatenated to obtain the concatenated feature; wherein, the number of channels of the concatenated feature is equal to the sum of the number of the first channel and the number of the second channel.

12. The method according to any one of claims 1 to 11, wherein, The process of pooling the spliced ​​features to obtain pooled features includes: The spliced ​​features are subjected to max pooling to obtain max pooled features; The spliced ​​features are subjected to average pooling to obtain average pooled features; The maximum pooling feature and the average pooling feature are determined as the pooling feature.

13. The method according to any one of claims 1 to 12, wherein, The process of encoding the original image to obtain image encoding features includes: The original image is encoded by calling an image encoder to obtain the image encoded features.

14. The method according to any one of claims 1 to 13, wherein, The process of encoding the descriptive text to obtain text encoding features includes: The text encoding feature is obtained by calling a text encoder to encode the description text.

15. The method according to any one of claims 1 to 14, wherein, The image semantic identifier features include facial features used to characterize the target object; the facial features include at least one of the following: facial feature position, facial feature shape, and face shape.

16. The method according to any one of claims 1 to 15, wherein, The image processing method is implemented through a pre-trained image processing model; the image processing model includes a backbone generation network and a feature injection network; the backbone generation network includes at least a text encoder and a feature segmentation network, and the feature injection network includes an image encoder and an image identifier extractor.

17. The method according to any one of claims 1 to 16, wherein, The method further includes training the image processing model through the following steps: The sample image is input into the feature identifier injection network, and the image encoder in the feature identifier injection network performs image encoding on the sample image to obtain the sample image encoded features. The image identifier extractor in the feature identifier injection network performs multi-level image residual mapping on the encoded features of the sample image to obtain the semantic identifier features of the sample target object in the sample image at each level in multiple levels. The sample description text is input into the backbone generation network, and the text encoder in the backbone generation network performs text encoding on the sample description text to obtain the sample text encoding features. The sample text encoding features are obtained by performing multi-level text residual mapping on the feature segmentation network in the backbone generation network; Based on the semantic identifier features of the sample images and the sample fusion features, the loss of the image processing model is calculated to obtain the loss result of the image processing model; The model parameters in the image processing model are updated based on the loss result to obtain the trained image processing model.

18. The method according to any one of claims 1 to 17, wherein, The loss calculation of the image processing model based on the semantic identifier features of the sample images and the sample fusion features, to obtain the loss result of the image processing model, includes: Based on the sample fusion features and the sample images, the basic loss of the image processing model is calculated to obtain the basic loss result; Based on the semantic labeling features of the sample images and the sample fusion features, the image processing model is subjected to image semantic labeling loss calculation to obtain the image semantic labeling loss result; The sum of the basic loss result and the image semantic labeling loss result is determined as the loss result of the image processing model.

19. The method according to any one of claims 1 to 18, wherein, The basic loss calculation of the image processing model based on the sample fusion features and the sample images, to obtain the basic loss result, includes: The sample fusion features are decoded to obtain a sample generated image; the sample generated image includes the object features of the sample target object and the descriptive features of the sample description text; The mean square error between the generated image and the sample image is determined as the base loss result.

20. The method according to any one of claims 1 to 19, wherein, The step of calculating the image semantic labeling loss based on the semantic labeling features of the sample images and the sample fusion features, and obtaining the image semantic labeling loss result, includes: The semantic identifier features of the sample image are decoded to obtain a decoded image; The sample fusion features are decoded to obtain the sample generated image; The decoded image is subjected to facial detection encoding to obtain a first encoding vector; The sample generated image is subjected to facial detection encoding to obtain a second encoding vector; The cosine distance between the first encoding vector and the second encoding vector is determined as the image semantic labeling loss result.

21. An image processing apparatus, the apparatus comprising: The acquisition module is configured to acquire descriptive text and the original image containing the target object; The encoding module is configured to encode the original image to obtain image encoding features; The descriptive text is encoded to obtain text encoding features; The image residual mapping module is configured to perform multi-level image residual mapping on the image coding features to obtain the image semantic identification features of the target object at each of the multiple levels. The text residual mapping module is configured to perform multi-level text residual mapping on the text encoding features based on the image semantic identifier features at each level to obtain fused features; wherein, the number of levels of the multi-level text residual mapping is the same as the number of levels of the multi-level image residual mapping. The decoding module is configured to decode the fused features to obtain a generated image; the generated image includes the object features of the target object and the descriptive features of the descriptive text.

22. An electronic device, comprising: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the image processing method according to any one of claims 1 to 20.

23. A computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the image processing method according to any one of claims 1 to 20.

24. A computer program product comprising executable instructions stored in a computer-readable storage medium; When the processor of the electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, it implements the image processing method according to any one of claims 1 to 20.

Images