Method and apparatus for image processing based on artificial intelligence, electronic device, computer readable storage medium, and computer program product

Abstract

The present application provides a method and apparatus for image processing based on artificial intelligence, an electronic device, a computer readable storage medium, and a computer program product. The method comprises: performing noise addition processing on an object image, to obtain a noise image coding vector, and performing text coding processing on an action instruction text, to obtain a first action text coding vector; performing denoising processing on the noise image coding vector based on the first action text coding vector, to obtain a first action image, and updating the first action text coding vector on the basis of the difference between the first action image and the object image, to obtain a second action text coding vector; fusing the first action text coding vector and the second action text coding vector, to obtain a fused action text coding vector; and performing denoising processing on the noise image coding vector based on the fused action text coding vector, to obtain a second action image.

Description

Artificial intelligence-based image processing method, device, electronic device, computer-readable storage medium, and computer program product

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202310969776.3 and application date of August 3, 2023, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present application relates to artificial intelligence technology, and in particular to an image processing method, device, electronic device, computer-readable storage medium, and computer program product based on artificial intelligence. Background Art

[0004] Artificial Intelligence (AI) is a comprehensive field of computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.

[0005] Artificial intelligence-based image editing technology has been widely used in the image creation process, especially image motion editing technology. However, although the image motion editing technology in related technologies can realize motion editing, it is difficult to ensure the consistency of the image before and after editing, which affects the image editing effect.

[0006] Summary of the Invention

[0007] The embodiments of the present application provide an artificial intelligence-based image processing method, device, electronic device, computer-readable storage medium, and computer program product, which can realize motion editing while retaining the main content of the object image.

[0008] The technical solution of the embodiment of the present application is implemented as follows:

[0009] The present invention provides an artificial intelligence-based image processing method, including:

[0010] Performing noise processing on the object image to obtain a noise image coding vector, and performing text coding processing on the action instruction text to obtain a first action text coding vector;

[0011] performing denoising processing on the noise image coding vector based on the first action text coding vector to obtain a first action image, and performing updating processing on the first action text coding vector based on a difference between the first action image and the object image to obtain a second action text coding vector;

[0012] Fusing the first action text encoding vector and the second action text encoding vector to obtain a fused action text encoding vector;

[0013] The noise image encoding vector is denoised based on the fused action text encoding vector to obtain a second action image, wherein the second action image is the result of applying the action corresponding to the action instruction text to the object included in the object image.

[0014] The present invention provides an artificial intelligence-based image processing device, comprising:

[0015] An encoding module, configured to perform noise processing on the object image to obtain a noise image encoding vector, and perform text encoding processing on the action instruction text to obtain a first action text encoding vector;

[0016] a fine-tuning module, configured to perform a denoising process on the noise image coding vector based on the first action text coding vector to obtain a first action image, and to update the first action text coding vector based on a difference between the first action image and the object image to obtain a second action text coding vector;

[0017] a fusion module, configured to fuse the first action text encoding vector and the second action text encoding vector to obtain a fused action text encoding vector;

[0018] A generation module is used to perform denoising processing on the noise image coding vector based on the fused action text coding vector to obtain a second action image, wherein the second action image is the result of applying the action corresponding to the action instruction text to the object included in the object image.

[0019] The present invention provides an artificial intelligence-based image processing method, including:

[0020] Obtain an image editing request, wherein the image editing request includes any one of the following: an image rendering request, an action editing request;

[0021] When the image editing request is an image rendering request, calling an image rendering model to perform image rendering processing on the object image carried in the image editing request to obtain a rendered image;

[0022] When the image editing request is an action editing request, the action editing model is called to execute the method described in the embodiment of the present application on the object image carried in the image editing request to obtain a second action image.

[0023] The present invention provides an artificial intelligence-based image processing device, comprising:

[0024] An acquisition module, configured to acquire an image editing request, wherein the image editing request includes any one of the following: an image rendering request and an action editing request;

[0025] a rendering module, configured to, when the image editing request is an image rendering request, call an image rendering model to perform image rendering processing on the object image carried in the image editing request to obtain a rendered image;

[0026] The action module is used to call the action editing model to execute the method described in the embodiment of the present application on the object image carried by the image editing request when the image editing request is an action editing request, so as to obtain a second action image.

[0027] An embodiment of the present application provides an artificial intelligence-based image processing method, which is executed by an electronic device and includes:

[0028] Display image editing entrance;

[0029] In response to an information input operation at the image editing portal, the input image editing information is displayed, wherein the image editing information includes a base image and editing information, and the editing information includes at least one of the following: editing text and a guide image, the editing text being an action instruction text or a rendering text, and the guide image and the rendering text both representing a rendering direction;

[0030] In response to an image processing operation based on the image editing information, a target image obtained by editing the base image based on the editing information is displayed.

[0031] The present invention provides an artificial intelligence-based image processing device, comprising:

[0032] Display module, used for displaying image editing entrance;

[0033] an input module, configured to display input image editing information in response to an information input operation at the image editing portal, wherein the image editing information includes a base image and editing information, and the editing information includes at least one of the following: editing text and a guide image, wherein the editing text is action instruction text or rendering text, and the guide image and the rendering text both represent a rendering direction;

[0034] The editing module is configured to, in response to an image processing operation based on the image editing information, display a target image obtained by editing the base image based on the editing information.

[0035] An embodiment of the present application provides an electronic device, comprising:

[0036] a memory for storing computer-executable instructions;

[0037] The processor is used to implement the artificial intelligence-based image processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0038] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the artificial intelligence-based image processing method provided in an embodiment of the present application when executed by a processor.

[0039] An embodiment of the present application provides a computer program product, including computer-executable instructions. When the computer-executable instructions are executed by a processor, the artificial intelligence-based image processing method provided in the embodiment of the present application is implemented.

[0040] The embodiments of the present application have the following beneficial effects:

[0041] Through the embodiment of the present application, the object image is denoised to obtain a noise image coding vector, and the action instruction text is text-encoded to obtain a first action text coding vector; the noise image coding vector is denoised based on the first action text coding vector to obtain a first action image, and the first action text coding vector is updated based on the difference between the first action image and the object image to obtain a second action text coding vector, which is equivalent to fine-tuning the representation of the action instruction text, thereby ensuring the recognition of the original object image during the image processing process and controlling the consistency during the image editing process, the first action text coding vector and the second action text coding vector are fused to obtain a fused action text coding vector, the noise image coding vector is denoised based on the fused action text coding vector, and the result of applying the action corresponding to the action instruction text to the object included in the object image is obtained. Since the result is generated based on the control of the fused action text coding vector, action editing can be achieved while ensuring image consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1 is a schematic diagram of the structure of an image processing system provided in an embodiment of the present application;

[0043] FIG2 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0044] FIG3A is a schematic diagram of a first flow chart of an image processing method provided in an embodiment of the present application;

[0045] FIG. 3B is a second flow chart of the image processing method provided in an embodiment of the present application.

[0046] FIG3C is a schematic diagram of a third flow chart of the image processing method provided in an embodiment of the present application;

[0047] FIG3D is a schematic diagram of a fourth flow chart of the image processing method provided in an embodiment of the present application;

[0048] FIG3E is a schematic diagram of a fifth flow chart of the image processing method provided in an embodiment of the present application;

[0049] FIG4A is a first schematic diagram of an image creation interface provided by the related art;

[0050] FIG4B is a second schematic diagram of an image creation interface provided by the related art;

[0051] FIG5A is a schematic diagram of a first interface of an image processing method provided in an embodiment of the present application;

[0052] FIG5B is a schematic diagram of a second interface of the image processing method provided in an embodiment of the present application;

[0053] FIG5C is a schematic diagram of a third interface of the image processing method provided in an embodiment of the present application;

[0054] FIG5D is a schematic diagram of a fourth interface of the image processing method provided in an embodiment of the present application;

[0055] FIG5E is a schematic diagram of a fifth interface of the image processing method provided in an embodiment of the present application;

[0056] FIG5F is a schematic diagram of a sixth interface of the image processing method provided in an embodiment of the present application;

[0057] FIG6 is a system diagram of an image processing method provided in an embodiment of the present application;

[0058] FIG7 is a schematic diagram of a rendering process of an image processing method provided in an embodiment of the present application;

[0059] FIG8 is a schematic diagram of an action editing method of an image processing method provided by an embodiment of the present application;

[0060] FIG9 is a schematic diagram of fine-tuning of the image processing method provided in an embodiment of the present application;

[0061] FIG10 is a schematic diagram of action editing of the image processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0063] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0064] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0066] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0067] 1) Text-based Image Diffusion Model: A generative model based on the diffusion process. The input of the generative model is text. The generative model performs text-based image restoration on random noise images and generates predicted images related to the text.

[0068] 2) Image Editing: Image editing includes various scenarios: style change, motion editing, and scene-time atmosphere rendering. Style change is the transformation of the input image style. Motion editing is the change of the motion of the object in the image. Scene-time atmosphere rendering is the change of the overall image atmosphere, for example, changing a sunny image to a rainy image.

[0069] 3) Scene time atmosphere rendering is to perform operations on the scene in the image based on time, seasons, morning and evening, etc. For example, the image initially has a daytime atmosphere, but after rendering, it becomes a nighttime atmosphere; the image initially has a spring atmosphere, but after rendering, it becomes an autumn atmosphere. The image content before and after rendering remains unchanged, only the season-related content is changed.

[0070] 4) Image Noising: This refers to the process of artificially introducing noise into an image. This is a common image processing technique used to simulate real-world noise or improve image analysis in specific applications. Noising can simulate various noise types, such as Gaussian, salt-and-pepper, and uniform noise, to better study the effects of noise on image processing algorithms or train denoising algorithms.

[0071] 5) Image denoising: The process of removing or reducing noise from image data. Image noise can be caused by random interference introduced by the image acquisition device, transmission process, or storage process, resulting in degraded image quality and blurred details. The goal of denoising is to restore image clarity and reduce the impact of noise on image quality, making the image more suitable for subsequent analysis and processing.

[0072] 6) Downsampling: refers to the process of reducing the number of sampling points in the image data, lowering the resolution and detail of the image, thereby reducing the amount of data, reducing processing complexity or reducing storage requirements.

[0073] 7) Attention Mechanism: The process of applying varying degrees of attention to each piece of input data. In computer vision and natural language processing, attention processing refers to the process of filtering and focusing attention on data or information. This process typically involves classifying, clustering, or performing correlation analysis on the input data or information to identify key information or features.

[0074] In the field of image editing, although object motion editing can be implemented in related technologies, the motion editing solutions in related technologies cannot guarantee the consistency of images before and after editing when implementing motion editing.

[0075] Embodiments of the present application provide an image processing method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can implement motion editing while retaining the main content of an object image.

[0076] The following describes exemplary applications of the electronic device provided in the embodiments of the present application. The electronic device provided in the embodiments of the present application can be implemented as a terminal or a server.

[0077] Referring to FIG1 , FIG1 is a schematic diagram of an application mode of an image processing method provided in an embodiment of the present application; for example, FIG1 involves a server 200, a network 300, and a terminal 400. Terminal 400 is connected to server 200 via network 300, which can be a wide area network or a local area network, or a combination of the two.

[0078] In some embodiments, the server 200 may be a server corresponding to an application. For example, if the application is image processing software installed in the terminal 400 , the server 200 is an image processing background for executing the image processing method provided in the embodiment of the present application.

[0079] In some embodiments, the terminal 400 receives an image editing request, where the image editing request carries an image uploaded by the user and an action instruction text. The terminal 400 sends the image editing request to the server 200. The server 200 performs noise processing on the object image to obtain a noise image coding vector, and performs text coding processing on the action instruction text to obtain a first action text coding vector; performs denoising processing on the noise image coding vector based on the first action text coding vector to obtain a first action image, and updates the first action text coding vector based on the difference between the first action image and the object image to obtain a second action text coding vector; performs fusion processing on the first action text coding vector and the second action text coding vector to obtain a fused action text coding vector; performs denoising processing on the noise image coding vector based on the fused action text coding vector to obtain a second action image that applies the action corresponding to the action instruction text to the object image, and returns the second action image to the terminal 400.

[0080] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart TV, car terminal, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application. The database 500 can be set separately, can be integrated on the server 200, or the database 500 can be set on a machine independent of the server 200, which is not limited in the embodiments of the present application. The database 500 provided in the embodiments of the present application can be used to store the second action image generated by the server 200 for remote storage backup.

[0081] In some embodiments, the terminal 400 can implement the image processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as a photography APP; it can also be a mini-program, that is, a program that can be run by simply downloading it into a browser environment; it can also be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module, or plug-in in any form.

[0082] Referring to Figure 2, Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device is a terminal or a server. Taking the electronic device as a server as an example for explanation, the server shown in Figure 2 includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the terminal 400 are coupled together through a bus system 240. It can be understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as bus system 240 in Figure 2.

[0083] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0084] The user interface 230 includes one or more output devices 231 that enable presentation of media content, which may include one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232 that may include user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0085] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0086] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0087] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0088] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0089] A network communication module 252 for reaching other electronic devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, WiFi, and USB;

[0090] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0091] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0092] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in software. FIG2 shows an image processing device 255-1 stored in the memory 250, which can be software in the form of a program or plug-in, including the following software modules: encoding module 2551, fine-tuning module 2552, fusion module 2553, generation module 2554, and training module 2555. FIG2 shows an image processing device 255-2 stored in the memory 250, which can be software in the form of a program or plug-in, including the following software modules: acquisition module 2556, rendering module 2557, and action module 2558. These modules are logical and can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0093] The following describes the image processing method provided by the embodiments of the present application. As previously described, the electronic device implementing the image processing method of the embodiments of the present application can be a terminal or a server. The server is used as an example for the following description. Therefore, the execution entity of each step will not be repeated below. Referring to FIG. 3A , steps 101 to 104 shown in FIG. 3A will be described below.

[0094] In step 101, the object image is subjected to noise processing to obtain a noise image coding vector, and the action instruction text is subjected to text coding processing to obtain a first action text coding vector.

[0095] As an example, an object image can be an image uploaded by a user or obtained by taking a photo. The object can be a person, an animal, a physical object, or a virtual object. The object image here can be an image including a person and an animal, or an image including a certain object. The action instruction text here consists of a basic image description and a user-edited action text. The basic image description is the text used to describe the object image, such as "man" or "woman", and the user-edited action text is the action that the user wants the object in the image to perform, such as "smile", "raise hand", etc. Physical objects include objects in the real world, and virtual objects include virtual props in virtual game scenes, fictional objects in paintings, etc.

[0096] In some embodiments, the noise image coding vector is obtained by performing noise addition processing on the object image in step 101, which can be achieved by the following technical solutions: superimposing the object image and the noise image to obtain a superimposed image; performing image latent space coding processing on the superimposed image to obtain a noise image coding vector. The image latent space coding processing can be achieved by the following methods: inputting the superimposed image into the image encoder of the text image contrast model, and using the feature vector of the middle hidden layer of the image encoder to represent the superimposed image to obtain a noise image coding vector

[0097] Through the latent space encoding processing performed by the embodiments of the present application, the subsequent denoising processing can be performed in the latent space, thereby reducing the amount of calculation and improving the denoising effect.

[0098] As an example, randomly select a seed i to generate a noise image, superimpose the generated noise image with the original object image to generate a superimposed image x, and perform latent space encoding on the superimposed image x to obtain the noise image encoding vector Z as the latent space representation T The seed i in the noise generation algorithm is a parameter that serves as the initial value of the random number generator. Latent space encoding is to call the encoder to map the image data of the superimposed image x from a high-dimensional to a low-dimensional latent space.

[0099] In some embodiments, text encoding processing is implemented by calling a text model in a text-image contrast model, obtaining multiple first text samples and first image samples matching each first text sample; performing image encoding processing on each first image sample through the visual model of the text-image contrast model to obtain an image encoding vector for each first image sample; performing text encoding processing on each first text sample through the text model of the text-image contrast model to obtain a text encoding vector for each first text sample; determining a text-image contrast loss based on the text encoding vector of each first text sample, the image encoding vector of each first image sample, and the matching relationship between each first text sample and each first image sample; and updating the parameters of the text-image contrast model based on the text-image contrast loss. Through the embodiments of the present application, the visual model and the text model can be constrained to align in the semantic space, thereby improving the representation ability of the first action text encoding vector.

[0100] As an example, the first image sample matched by the first text sample refers to an image including a picture corresponding to the content described by the first text sample. The core idea of ​​the text-image contrast model is to use the semantic similarity between the first image sample and the first text sample to train the visual-language model. Specifically, the text-image contrast model uses a dual-tower neural network, where one tower is an image encoder and the other tower is a text encoder. The image encoder is responsible for converting the image into a vector representation (image encoding vector), and the text encoder is responsible for converting the text into a vector representation (text encoding vector). Then, the text-image contrast model uses a contrast loss function to optimize the inner product between the two vectors. In other words, the text-image contrast model hopes that the larger the inner product between the image and the text, the more similar they are; conversely, the smaller the inner product between the image and the text, the less similar they are.

[0101] In step 102, the noise image coding vector is denoised based on the first action text coding vector to obtain a first action image, and the first action text coding vector is updated based on the difference between the first action image and the object image to obtain a second action text coding vector.

[0102] For example, the denoising processing performed on the noise image coding vector based on the first action text coding vector includes a denoising operation and a decoding operation, including the following processing: inputting the first action text coding vector and the noise image coding vector into a denoising network of multiple levels, identifying and removing the noise in the noise image coding vector with the first action text coding vector as a reference, and decoding the denoised noise image coding vector to obtain a first action image.

[0103] In some embodiments, the denoising process based on the first action text encoding vector is implemented by an image generation model, which includes N cascaded denoising networks and a decoding network, where the value range of N is 2 ≤ N; referring to FIG. 3B, in step 102, the noise image encoding vector is denoised based on the first action text encoding vector to obtain the first action image, which can be realized through steps 1021 to 1022 shown in FIG. 3B.

[0104] In step 1021, through the nth denoising network in the N cascaded denoising networks, the input of the nth denoising network is denoised, and the nth denoising result output by the nth denoising network is transmitted to the (n + 1)th denoising network for continuous denoising to obtain the (n + 1)th denoising result corresponding to the (n + 1)th denoising network.

[0105] Exemplarily, the denoising process in the N cascaded denoising networks is sequentially executed by multiple denoising networks, and the output of each denoising network is the input of the next denoising network, that is, the iterative execution of the denoising process.

[0106] In step 1022, the denoising result output by the Nth denoising network is decoded through the decoding network to obtain the first action image.

[0107] As an example, there are N denoising networks cascaded and a decoding network in the image generation model, which is equivalent to performing N denoising processes and finally performing image decoding. Each time, the denoising is performed based on the latent space image encoding vector obtained from the previous denoising and then input into the next denoising network for denoising. n is an integer variable starting from 1, and the value range of n is 1 ≤ n < N. When n = 1, the input of the nth denoising network is the noise image encoding vector and the first action text encoding vector. When the value range of n is 2 ≤ n < N, the input of the nth denoising network is the (n - 1)th denoising result output by the (n - 1)th denoising network and the first action text encoding vector.

[0108] As an example, taking N = 3 for illustration, the first denoising network denoises the noise image encoding vector (latent space noise encoding) and the first action text encoding vector to obtain the first denoising result. The second denoising network denoises the first denoising result and the first action text encoding vector to obtain the second denoising result. The third denoising network denoises the second denoising result and the first action text encoding vector to obtain the third denoising result. Each denoising result obtained in the above manner is also a latent space encoding, and the denoising process performed by each denoising network is equivalent to the denoising process of one time step.

[0109] In some embodiments, in step 1021, denoising the input of the n-th denoising network through the n-th denoising network among N cascaded denoising networks can be achieved through the following technical solution: performing first attention processing on the input of the m-th attention layer and the first action text encoding vector through the m-th attention layer in the n-th denoising network to obtain first attention features as the m-th attention result of the m-th attention layer in the n-th denoising network; transmitting the m-th attention result of the m-th attention layer in the n-th denoising network to the (m + 1)-th attention layer to continue attention processing to obtain the (m + 1)-th attention result of the (m + 1)-th attention layer in the n-th denoising network; using the M-th attention result output by the M-th attention layer in the n-th denoising network as the n-th denoising result.

[0110] As an example, the value range of M is 2 ≤ M, m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the m-th attention layer is the (n - 1)-th fusion denoising result. When the value range of m is 2 ≤ m < M, the input of the m-th attention layer is the (m - 1)-th attention result output by the (m - 1)-th attention layer.

[0111] As an example, the n-th denoising network includes H cascaded downsampling networks and H cascaded upsampling networks. Here, the value of M is 2 * H, and the value range of H is 2 ≤ H; denoising the input of the n-th denoising network through the n-th denoising network among N cascaded denoising networks can be achieved through the following technical solution: performing downsampling on the (n - 1)-th denoising result and the first action text encoding vector through H cascaded downsampling networks to obtain the downsampling result of the n-th denoising network; performing upsampling on the downsampling result of the n-th denoising network through H cascaded upsampling networks to obtain the upsampling result of the n-th denoising network as the n-th denoising result corresponding to the n-th denoising network. By performing downsampling and upsampling in each denoising process, more detailed information can be retained during the denoising process.

[0112] Continuing with the above example, taking the 2nd denoising network as an example for illustration, the denoising network may include 3 downsampling networks and 3 upsampling networks. Performing downsampling on the 1st denoising result and the first action text encoding vector through 3 cascaded downsampling networks to obtain the downsampling result of the 2nd denoising network; performing upsampling on the downsampling result of the 2nd denoising network through 3 cascaded upsampling networks to obtain the upsampling result of the 2nd denoising network as the 2nd denoising result corresponding to the 2nd denoising network.

[0113] As an example, the n-th denoising result and the first action text encoding vector are downsampled through H cascaded downsampling networks to obtain the downsampling result of the n-th denoising network, which can be achieved through the following technical solution: Through the h-th downsampling network in the H cascaded downsampling networks, the input of the h-th downsampling network is downsampled to obtain the h-th downsampling result corresponding to the h-th downsampling network, and the h-th downsampling result corresponding to the h-th downsampling network is transmitted to the (h + 1)-th downsampling network to continue the downsampling process to obtain the (h + 1)-th downsampling result corresponding to the (h + 1)-th downsampling network; The downsampling result output by the H-th downsampling network is used as the n-th denoising result; where h is an integer variable that increases starting from 1, and the value range of h is 1 ≤ h ≤ H - 1. When h takes the value of 1, the input of the h-th sampling network is the (n - 1)-th denoising result and the first action text encoding vector. When the value range of h is 2 ≤ h < H, the input of the h-th downsampling network is the (h - 1)-th downsampling result output by the (h - 1)-th downsampling network and the first action text encoding vector. The processing process of the upsampling network is the same as that of the downsampling network.

[0114] Continuing with the above example, through the first downsampling network, the input of the first downsampling network is downsampled to obtain the downsampling result corresponding to the first downsampling network, and the downsampling result corresponding to the first downsampling network is transmitted to the second downsampling network to continue the downsampling process to obtain the second downsampling result corresponding to the second downsampling network; Through the second downsampling network, the input of the second downsampling network is downsampled to obtain the downsampling result corresponding to the second downsampling network, and the downsampling result corresponding to the second downsampling network is transmitted to the third downsampling network to continue the downsampling process to obtain the third downsampling result corresponding to the third downsampling network. The third downsampling result output by the third downsampling network is used as the second denoising result. Here, the input of each downsampling network includes the first action text encoding vector.

[0115] As an example, the m-th downsampling network includes an attention layer; Through the m-th downsampling network in the M cascaded downsampling networks, the input of the m-th downsampling network is downsampled to obtain the m-th downsampling result corresponding to the m-th downsampling network, which can be achieved through the following technical solution: Through the attention layer, the (m - 1)-th downsampling result corresponding to the (m - 1)-th downsampling network and the first action text encoding vector are subjected to the first attention processing to obtain the m-th downsampling result corresponding to the m-th downsampling network. Each downsampling network includes an attention layer. The input of the attention layer is the output of the previous cascaded downsampling network (that is, the output of the attention layer included in the previous cascaded downsampling network). Through the embodiments of the present application, more effective information can be retained through the residual layer, and the attention layer can model the spatial dimension based on the text encoding vector, thereby improving the denoising effect.

[0116] In some embodiments, the nth denoising network includes M cascaded attention layers; the above-mentioned first attention processing is performed on the input of the mth attention layer and the first action text encoding vector by the mth attention layer in the nth denoising network to obtain the first attention feature as the mth attention result of the mth attention layer in the nth denoising network, which can be achieved through the following technical solutions: mapping processing based on the query matrix is ​​performed on the input of the mth attention layer to obtain the attention query matrix; mapping processing based on the key matrix is ​​performed on the first action text encoding vector to obtain the attention key matrix; mapping processing based on the value matrix is ​​performed on the first action text encoding vector to obtain the attention value matrix; multiplying the attention query matrix and the transposed matrix of the attention key matrix to obtain the multiplication result, and obtaining the ratio between the multiplication result and the dimension of the attention key matrix; performing maximum likelihood processing on the above ratio, and multiplying the maximum likelihood result with the attention value matrix to obtain the first attention feature.

[0117] Maximum Likelihood Estimation (MLE) is an estimation method in statistics used to estimate parameters so that the probability (i.e., likelihood) of the model occurring is maximized given the observed data. In the embodiment of the present application, the maximum likelihood process is to estimate the maximum value of the ratio between the product of the transposed matrix of the attention query matrix and the attention key matrix and the dimension of the attention key matrix, and the estimated maximum ratio is used as the maximum likelihood result.

[0118] Through the embodiments of the present application, the action instruction text can be specifically integrated into the denoising network, thereby playing a constraint role in image generation and improving the training effect of the model.

[0119] As an example, the first action text encoding vector is introduced as a conditional signal through the cross attention mechanism. This cross attention mechanism can also integrate the conditional information of the action instruction text into the denoising process, see formula (1) to formula (3):

[0120] Among them, z represents the output of the m-1th attention layer, and is a projection matrix with learnable parameters, is the attention value matrix, is the attention query matrix, is the attention key matrix, x is the first action text encoding vector, Q is the attention query matrix, K is the attention key matrix, and V is the attention value matrix.

[0121] In some embodiments, referring to FIG3C , in step 102 , the first action text encoding vector is updated based on the difference between the first action image and the object image to obtain the second action text encoding vector, and steps 105 to 107 shown in FIG3C may be executed.

[0122] In step 105 , a first pixel value at each position in the first action image and a second pixel value at each position in the object image are acquired.

[0123] Here, the first pixel value and the second pixel value are used to distinguish pixel values ​​belonging to different images.

[0124] In step 106 , for each position, the difference between the first pixel value and the second pixel value is obtained.

[0125] For example, for a position in the first action image that is the same as the object image, the difference between the pixel values ​​corresponding to the position in the first action image and the object image is obtained. The difference between the two pixel values ​​can be the square of the difference between the two pixel values. For example: the first pixel value of position i in the first action image is The second pixel value at position i in the object image is y i The difference between the two is characterized by

[0126] In step 107 , the differences at multiple positions are fused to obtain the difference between the first action image and the object image.

[0127] As an example, see formula (4)

[0128] Among them, y i is the second pixel value at position i in the object image, is the first pixel value at position i in the first motion image, and MSE is the difference between the first motion image and the object image.

[0129] By constraining the consistency of the image before and after through the embodiment of the present application, it is possible to ensure that the main content of the image remains unchanged during the action editing process, thereby optimizing the action editing effect of the image.

[0130] In some embodiments, denoising processing based on the first action text encoding vector is implemented through an image generation model. When the first action text encoding vector is updated based on the difference between the first action image and the object image, the parameters of the image generation model are updated based on the difference between the first action image and the object image to obtain an updated image generation model.

[0131] As an example, only the first action text encoding vector can be updated based on the error, and the first action text encoding vector and the image generation model can also be updated simultaneously based on the error (all parameters in the image generation model can be updated here, or the parameters of the U-shaped network in the image generation model can be updated). Through the embodiments of the present application, the image generation model for performing denoising processing can be fine-tuned while fine-tuning the first action text encoding vector, thereby optimizing the denoising effect.

[0132] In step 103, the first action text encoding vector and the second action text encoding vector are fused to obtain a fused action text encoding vector.

[0133] In some embodiments, in step 103, the first action text encoding vector and the second action text encoding vector are fused to obtain a fused action text encoding vector, which can be achieved through steps 1031 to 1033 shown in Figure 3D.

[0134] In step 1031, the first action text encoding vector is subjected to a truncation process based on a first number to obtain a first truncation encoding vector, wherein the first truncation encoding vector includes a first number of vectors located at the front of the first action text encoding vector.

[0135] For example, the first number can be set according to actual application scenarios. The interception process is a process of extracting at least part of the content in the code.

[0136] In step 1032, the second action text encoding vector is subjected to a truncation process based on a second number to obtain a second truncation encoding vector, wherein the second truncation encoding vector includes a second number of vectors located at the front of the second action text encoding vector.

[0137] For example, the second number can be set according to the actual application scenario, and the first number and the second number can be the same or different.

[0138] In step 1033, the second truncated coding vector is spliced ​​to the end of the first truncated coding vector to obtain a fused action text coding vector.

[0139] As an example, first, the fine-tuned second action text encoding vector and the unfine-tuned first action text encoding vector are spliced ​​to generate the final fused action text encoding vector. The splicing principle is that the unfine-tuned first action text encoding vector is in front and the fine-tuned second action text encoding vector is in the back. Half of each is taken to splice into the final fused action text encoding vector. For example, for the diffusion model, 77 (first number) vectors are used as encoding, then the first 38 vectors of the unfine-tuned first action text encoding vector and the first 39 (second number) vectors of the fine-tuned second action text encoding vector are taken to splice. Here, the unfine-tuned first action text encoding vector must be placed in the front, because the unfine-tuned first action text encoding vector retains more editing capabilities, while the fine-tuned second action text encoding vector represents the original object image. The object image cannot provide editing control information, so the unfine-tuned first action text encoding vector must be placed in the front to ensure higher editing capabilities.

[0140] In step 104 , the noise image coding vector is subjected to denoising processing based on the fused action text coding vector to obtain a second action image, wherein the second action image is a result of applying the action corresponding to the action instruction text to the object included in the object image.

[0141] In some embodiments, the denoising processing based on the fused action text coding vector here and the denoising processing based on the first action text coding vector in step 102 can be performed using exactly the same model, that is, both are performed using a pre-trained image generation model, or, the denoising processing based on the fused action text coding vector here and the denoising processing based on the first action text coding vector in step 102 can be performed using models with the same structure but different parameters, that is, step 104 is performed using an image generation model obtained based on error update, and step 102 is performed only using a pre-trained image generation model, or, the denoising processing based on the fused action text coding vector here and the denoising processing based on the first action text coding vector in step 102 can be performed using a structurally related but structurally different action editing model, and the action editing model is composed of an image generation model (obtained based on the difference update in step 102 or the pre-trained image generation model used in step 102) and multiple image information networks.

[0142] In some embodiments, denoising processing based on action text coding vectors is implemented through an image generation model, and denoising processing based on fused action text coding vectors is implemented through an action editing model, which is composed of an image generation model and multiple image information networks; before performing denoising processing on the noise image coding vector based on the fused action text coding vector to obtain a second action image, the noise image coding vector and the action text coding vector are forward propagated in the action editing model to obtain a third action image, wherein the action text coding vector is the first action text coding vector or the fused action text coding vector; based on the difference between the third action image and the object image, the multiple image information networks in the action editing model are updated to obtain an updated action editing model.

[0143] As an example, the action editing model cascades multiple image information networks based on the image generation model. The action editing model can be trained before executing step 104. During training, only the multiple image information networks can be updated, while the parameters of the image generation model remain unchanged. The image generation model here can be the image generation model updated based on the differences obtained in step 102.

[0144] As an example, the image generation model included in the action editing model can also be a pre-trained image generation model, that is, it has not been updated based on the difference obtained in step 102. In this case, the training of the action editing model can be performed before the model is deployed on the server, that is, the update involved in step 102 occurs after receiving the user's image editing request, and the training of the action editing model can be performed after receiving the user's image editing request (after updating the action text encoding vector based on the difference obtained in step 102) or before receiving the user's image editing request.

[0145] In some embodiments, the image generation model includes N cascaded denoising networks and a decoding network, where the value range of N is 2 ≤ N. The action editing model is obtained by configuring an image information network for each denoising network based on the image generation model. Each denoising network and the corresponding image information network form a fusion denoising network, and the cascading relationship of multiple fusion denoising networks is the same as that of multiple denoising networks. The above-mentioned forward propagation of the noise image encoding vector and the action text encoding vector in the action editing model to obtain the third action image can be achieved through the following technical solution: Through the n-th fusion denoising network in N cascaded fusion denoising networks, perform fusion denoising processing on the input of the n-th fusion denoising network, and transmit the n-th fusion denoising result output by the n-th fusion denoising network to the (n + 1)-th fusion denoising network to continue the fusion denoising processing, obtaining the (n + 1)-th fusion denoising result corresponding to the (n + 1)-th fusion denoising network; perform decoding processing on the fusion denoising result output by the N-th fusion denoising network to obtain the third action image.

[0146] As an example, n is an integer variable that increases incrementally starting from 1, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th fusion denoising network is the noise image encoding vector and the action text encoding vector. When the value range of n is 2 ≤ n < N, the input of the n-th fusion denoising network is the (n - 1)-th fusion denoising result output by the (n - 1)-th fusion denoising network and the action text encoding vector.

[0147] As an example, there are N cascaded fusion denoising networks and a decoding network in the image generation model, which is equivalent to performing N times of fusion denoising processing and finally performing image decoding processing. Each time, denoising processing is performed based on the latent space image encoding vector obtained from the previous denoising and then input into the next fusion denoising network for denoising processing. n is an integer variable that increases incrementally starting from 1, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th fusion denoising network is the noise image encoding vector and the action text encoding vector. When the value range of n is 2 ≤ n < N, the input of the n-th fusion denoising network is the (n - 1)-th fusion denoising result output by the (n - 1)-th fusion denoising network and the action text encoding vector.

[0148] As an example, taking N as 3 for illustration, the noise image coding vector (latent space noise coding) and the action text coding vector are subjected to fusion denoising processing through the first fusion denoising network to obtain the first fusion denoising result. The first fusion denoising result and the action text coding vector are subjected to fusion denoising through the second fusion denoising network to obtain the second fusion denoising result. The second fusion denoising result and the action text coding vector are subjected to fusion denoising processing through the third fusion denoising network to obtain the third fusion denoising result. Each fusion denoising result obtained in the above manner is also a latent space coding, and the fusion denoising processing performed by each fusion denoising network is equivalent to the fusion denoising processing of one time step.

[0149] In some embodiments, the nth fusion denoising network includes a plurality of downsampling networks, a plurality of upsampling networks, and the nth image information network corresponding to the nth denoising network. For the nth fusion denoising network in the N cascaded fusion denoising networks, the input of the nth fusion denoising network can be subjected to fusion denoising processing through the following technical solution: the noise image coding vector and the action text coding vector are subjected to bypass control processing through the nth image information network to obtain a bypass control result; the action text coding vector and the bypass control result are subjected to downsampling processing through the downsampling network to obtain a downsampling result; the downsampling result is subjected to upsampling processing through the upsampling network to obtain the nth fusion denoising result. By introducing bypass control in the embodiments of the present application, the overall action editing model can be fine-tuned by using the bypass control, thereby avoiding the computational resources consumed by the update of the overall model.

[0150] In some embodiments, the nth image information network includes P cascaded attention layers, where the value range of P is 2 ≤ P. For the noise image coding vector and the action text coding vector to be subjected to bypass control processing through the nth image information network to obtain a bypass control result, the following technical solution can be adopted: the pth attention result of the pth attention layer in the nth image information network is transmitted to the (p + 1)th attention layer for further second attention processing to obtain the (p + 1)th attention result of the (p + 1)th attention layer in the nth image information network; the second attention results output by each attention layer are used as the bypass control result. The structure of the nth image information network is the same as that of the nth fusion denoising network, so that the overall action editing model can be fine-tuned by fine-tuning the parameters of the nth image information network.

[0151] As an example, p is an integer variable starting from 1 and increasing, and the value range of p is 1 ≤ p ≤ P - 1. When p takes the value of 1, the input of the pth attention layer is the (n - 1)th fusion denoising result, and when the value range of p is 2 ≤ p < P, the input of the pth attention layer is the (p - 1)th attention result output by the (p - 1)th attention layer in the nth image information network.

[0152] As an example, the input of each attention layer is the output of the previous cascaded attention layer. The above-mentioned second attention processing of the input of the p-th attention layer and the action text encoding vector by the p-th attention layer in the n-th image information network can be achieved through the following technical solutions: perform mapping processing on the input of the p-th attention layer based on a query matrix to obtain an attention query matrix; perform mapping processing on the action text encoding vector based on a key matrix to obtain an attention key matrix; perform mapping processing on the action text encoding vector based on a value matrix to obtain an attention value matrix; multiply the attention query matrix by the transposed matrix of the attention key matrix to obtain a multiplication result, and obtain the ratio between the multiplication result and the dimension of the attention key matrix; perform maximum likelihood processing on the comparison value, and multiply the maximum likelihood result by the attention value matrix to obtain a first attention feature.

[0153] Through the embodiments of the present application, the action instruction text can be incorporated into the image information network in a targeted manner, thereby playing a constraining role in image generation and improving the training effect of the model.

[0154] In some embodiments, the downsampling network includes P cascaded attention layers; the above-mentioned downsampling processing of the action text encoding vector and the bypass control result by the downsampling network to obtain a downsampling result can be achieved through the following technical solutions: perform first attention processing on the input of the p-th attention layer and the action text encoding vector by the p-th attention layer in the downsampling network to obtain a first attention feature; fuse the first attention feature with the p-th attention result output by the p-th attention layer in the n-th image information network to obtain the p-th attention result of the p-th attention layer in the downsampling network; transmit the p-th attention result of the p-th attention layer in the downsampling network to the (p + 1)-th attention layer in the downsampling network to obtain the (p + 1)-th attention result of the (p + 1)-th attention layer in the downsampling network; use the P-th attention result output by the P-th attention layer in the downsampling network as the downsampling result.

[0155] As an example, p is an integer variable that increases incrementally starting from 1, and the value range of p is 1 ≤ p ≤ P - 1. When p takes the value of 1, the input of the p-th attention layer is the (n - 1)-th fusion denoising result. When the value range of p is 2 ≤ p < P, the input of the p-th attention layer is the (p - 1)-th attention result output by the (p - 1)-th attention layer.

[0156] As an example, taking the value of P as 3, the first attention layer of the downsampling network performs the first attention processing on the input of the first attention layer (the output of the n-1th fusion denoising network of the previous cascade) and the action text encoding vector to obtain the first attention feature; the first attention feature is fused with the first attention result output by the first attention layer in the nth image information network to obtain the first attention result of the first attention layer in the downsampling network; the first attention result of the first attention layer in the downsampling network is transmitted to the second attention layer in the downsampling network to obtain the second attention result of the second attention layer in the downsampling network; the third attention result output by the third attention layer in the downsampling network is used as the downsampling result, that is, the first attention feature output by each attention layer of the downsampling network of the nth fusion denoising network will be fused with the output of the corresponding attention layer in the nth image information network, and the fusion result is transmitted to the next attention layer of the downsampling network for processing.

[0157] Next, we will continue to describe the image processing method provided by the embodiments of the present application. As previously described, the electronic device implementing the image processing method of the embodiments of the present application can be a terminal or a server. The server will be used as an example for the following description. Therefore, the execution entities of each step will not be repeated below. See Figure 3E for a description of steps 201 to 203 shown in Figure 3E.

[0158] In step 201, an image editing request is obtained, wherein the image editing request includes any one of the following: an image rendering request and an action editing request.

[0159] For example, the image editing request is generated according to the editing operation of the user on the terminal device.

[0160] Referring to FIG6 , a user's image editing request is received. The image editing request includes any of the following: an image rendering request, an action editing request, and an image rendering request can be a style rendering request or an atmosphere rendering request. The image editing request carries the image editing information input by the user, and the corresponding image editing branch is determined according to the image editing request. The server is deployed with a model library, including model series 1 (realistic diffusion model or anime-style anythingV5 model), model series 2, and model series 3 (open source instruct Pix2Pix model). Model series 2 includes a text model (open source instruct Pix2Pix model) and an image model (open source AdaIN model).

[0161] In step 202 , when the image editing request is an image rendering request, an image rendering model is called to perform image rendering processing on the object image carried in the image editing request to obtain a rendered image.

[0162] As an example, the image rendering model can perform at least one of the following processes: ray casting algorithm, scanline rendering, cube map, texture mapping, etc. The image rendering model can also handle various lighting and material effects, such as reflection, refraction, shadow, transparency, texture, etc.

[0163] The image rendering model performs text encoding processing on the image rendering request to obtain a text encoding vector, and performs semantic analysis based on the text encoding vector to obtain the rendering instruction represented by the image rendering request, and performs image rendering processing corresponding to the rendering instruction on the object image to obtain a rendered image.

[0164] In step 203, when the image editing request is an action editing request, the action editing model is called to execute the image processing method of the embodiment of the present application on the object image carried in the image editing request to obtain a second action image.

[0165] As an example, when the image editing request is a motion editing request, steps 101 to 104 are performed on the object image carried in the image editing request to obtain a second motion image to respond to the motion editing request.

[0166] For action editing requests, you need to first use a realistic model (diffusion model) or an animation model (anythingV5 model) from the series 1 model based on the basic style recorded in the image editing information. After selecting the model, perform fine-tuning on the object image based on the action instruction text (one-time image training) and generate the target image (second action image).

[0167] For style rendering requests, the specific branch to be executed is determined based on whether there is editing text and a style guide image in the image editing information. If the editing text is not empty, a model is selected from the text model included in model series 2 to generate a target image with the same style as the editing text. If the guide image is not empty, a model is selected from the image model included in model series 2 to generate a target image with the same style as the guide image. When both branches are executed, two target images are finally output for the user to choose.

[0168] For atmosphere rendering requests, a model from model series 3 is selected to generate a target image with the same atmosphere as the edited text.

[0169] This embodiment provides a model for both the style rendering request and the atmosphere rendering request branches. Alternatively, a realistic model and an anime model can be set for each branch, as can the action editing request. The corresponding model is selected based on the base image style. Since the original instruction Pix2Pix is ​​a realistic model, the actual task is to train the anime base model of the instruction Pix2Pix as the anime model selection in the model library.

[0170] Next, we will continue to explain the image processing method provided by the embodiment of the present application. As mentioned above, the electronic device that implements the image processing method of the embodiment of the present application may be a terminal or a server, and the terminal will be used as an example for explanation. Therefore, the execution subject of each step will not be repeated in the following. Display the image editing entrance; in response to the information input operation at the image editing entrance, display the input image editing information, wherein the image editing information includes a base image and editing information, and the editing information includes at least one of the following: editing text and a guide image, the editing text is an action instruction text or a rendering text, and the guide image and the rendering text both represent the rendering direction; in response to the image processing operation based on the image editing information, display the target image obtained by editing the base image based on the editing information.

[0171] As an example, when the edited text is an action instruction text, the basic image is used as the object image, and steps 101 to 104 are performed on the object image to obtain a second action image, and the second action image is used as the target image.

[0172] For example, when the edited text is rendered text, and the rendered text is style-rendered text, a model is selected from the text models included in model series 2 to generate a target image with the same style as the edited text. When the edit information includes a guide image, a model is selected from the image models included in model series 2 to generate a target image with the same style as the guide image. When the edited text is rendered text, and the rendered text is atmosphere-rendered text, a model is selected from model series 3 to generate a target image with the same atmosphere as the edited text.

[0173] Referring to FIG. 5A , FIG. 5A shows a human-computer interaction interface when no operation is performed, and the human-computer interaction interface includes an input control window and an image display window.

[0174] Referring to FIG5B , in response to the information input operation at the image editing entrance, the input image editing information is displayed. FIG5B shows the object image (base image) and the editing text. The editing text is “thumbs up”. FIG5B also shows the branch (object action editing) corresponding to the image editing request represented by the image editing information, the basic style selection is “realistic” (corresponding to the realistic model in the calling model series 1), and the basic image description selection is “man”. At this time, the image display window is displayed in grayscale, indicating that it is unavailable. In response to the image processing operation based on the image editing information, the target image obtained by editing the base image based on the editing information is displayed. Referring to FIG5C , FIG5C shows the output after executing the object action editing. The image display window cancels the grayscale display, indicating that the window is enabled, and the target image after the object action editing is displayed in the image display window.

[0175] See Figure 5D, which shows the result of style conversion of the object image based on the style mentioned in the edit text. See Figure 5E, which shows the editing result of outputting the style of the guide image when the style guide image is input (rather than the edit text). See Figure 5F, which shows the output of two style conversion results when the edit text representing the style and the style guide image are input, so that one of the two results can be selected and returned in the image display window.

[0176] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0177] In some embodiments, the terminal receives an image editing request, where the image editing request carries an image uploaded by the user and an action instruction text. The terminal sends the image editing request to the server, and the server performs noise processing on the object image to obtain a noise image coding vector, and performs text coding processing on the action instruction text to obtain a first action text coding vector; performs denoising processing on the noise image coding vector based on the first action text coding vector to obtain a first action image, and updates the first action text coding vector based on the difference between the first action image and the object image to obtain a second action text coding vector; performs fusion processing on the first action text coding vector and the second action text coding vector to obtain a fused action text coding vector; performs denoising processing on the noise image coding vector based on the fused action text coding vector to obtain a second action image that applies the action corresponding to the action instruction text to the object image, and returns the second action image to the terminal.

[0178] Image editing includes the following situations: 1. A scene needs to change from morning to afternoon to evening as a transition picture of time change; 2. The image needs to be converted to a specified style; 3. The hand movement of the character in the picture needs to be changed to a thumbs-up gesture, etc. The above image editing involves multiple input situations and multiple image part editing (such as the image style or atmosphere requires a global image change, and the action change requires a local image change), and different image editing capabilities require different model processing. For example, style editing needs to try to maintain the original image information under different styles without loss, time change requires the generation of a model that can maintain the main information of the entire picture unchanged, and the model needs to be fine-tuned and regenerated when the action changes. The image editing system of the related technology cannot support object action editing; at the same time, multiple editing capabilities are scattered throughout the system, resulting in different functions, and it is difficult for users to find the corresponding function buttons when they need to edit.

[0179] The embodiment of the present application designs a system that supports users' creative image editing capabilities, which can realize various creative editing through unified input. The system provided by the embodiment of the present application selects an editing model according to the user input, edits the image according to the selection of the editing model, and displays the final result after the image is edited.

[0180] The embodiments of this application introduce object motion editing (including training and inference to achieve effective editing) and scene time change editing, greatly expanding the editable range of images. At the same time, a unified entry is used to support the generation of multiple editing capabilities, including time change editing for full-image rendering, object motion editing, style conversion, and more. The service of the embodiments of this application integrates model training and image generation for the current best generation effect through functional module scheduling, ultimately enabling users to generate creative images.

[0181] Referring to Figures 4A and 4B , related art solutions only offer simple physical editing capabilities, such as style conversion and image filtering. They lack object motion editing, semantic image editing (e.g., rain, snow, or nightfall), or reference-guided style conversion. Furthermore, the software's functionality is spread across multiple pages, including animation style, image editing, and photo editing. Users must navigate multiple times to access the required editing features, wasting time, inefficiency, and frustration.

[0182] When implementing the embodiments of the present application, the applicant found that the object action editing scheme in the related art cannot ensure the consistency of the image; and the image editing APP in the related art is difficult to provide style rendering and semantic level image rendering (for example, time atmosphere rendering, etc.) at the same time. At the same time, the function selection of the image editing APP in the related art is distributed in multiple places on the page, which is not conducive to user use.

[0183] The embodiment of the present application introduces object action editing capabilities to achieve semantic-level action editing. It can fine-tune the text representation of the input image and then perform action editing based on the fine-tuned text representation. Internal service calls do not require user manual participation.

[0184] The embodiment of the present application can also realize various forms of editing such as semantic-level atmosphere rendering, style rendering with or without guide images, etc., thereby enriching the overall editing function.

[0185] The embodiment of the present application also calls the model through a unified image editing input interface. After the user specifies the input, the system determines which of the above-mentioned editing capabilities to call based on the unified input information. At the same time, it supports multiple functions such as input required for different editing (text and images), direct calling of the model for editing, and editing after fine-tuning the model, thereby reducing the time users spend looking for the entrance to each function and improving editing efficiency.

[0186] See Figure 7, which illustrates the model reasoning process for image atmosphere rendering and style editing in an embodiment of the present application. The edited text (representing the style) and the base image are input into the model of model series 2 to obtain a style-rendered output image. The edited text (representing the atmosphere) and the base image are input into the model of model series 3 to obtain an atmosphere-rendered output image.

[0187] The following introduces image atmosphere rendering. The atmosphere rendering model of the embodiment of the present application adopts the open source instruction Pix2Pix model. An image and edit text are input, and then the target image is generated through the model. The edit text needs to be translated into English first, for example, python edit_cli.py-input input_image.jpg--output output.jpg--edit "make it nighttime". The embodiment of the present application can replace the atmosphere rendering model. In addition to using this open source model, a newly trained model can also be used. For example, a batch of new rendering data is collected according to the open source model method (this model requires more training samples, so it is necessary to ensure that more than 100 training data are collected for each editing instruction), and the above-mentioned instruction Pix2Pix model is fine-tuned and trained to obtain a newly trained atmosphere rendering model.

[0188] The following introduces text-based style editing. This embodiment of the application uses the open source instruct Pix2Pix model, inputs an image and style editing instructions (Hayao Miyazaki style), and then generates the target graphics through the model. The editing text must first be translated into English, that is, the following editing instructions must be entered: "python edit_cli.py -input input_image.jpg --output output.jpg --edit "Hayao Miyazaki style"".

[0189] The following introduces image-based style editing. The embodiment of this application adopts the open source AdaIN model, which can support the input of object images and style guidance images, and generate images of target content under the guidance style. The generation instruction is "python test.py --content input_image.jpg -style reference_image.jpg".

[0190] See Figure 8, which illustrates the object action editing process. The base graph description and edit text are combined to form the action instruction text. The action instruction text and base graph are fed into the action editing model from Model Series 1 (the underlying structure of the generative model, i.e., the U-shaped network), resulting in an output graph. Based on the differences between the output graph and the base graph, the textual representation (embedding) of the action instruction text and the action editing model (U-shaped network) are fine-tuned. The original representation of the action instruction text is merged with the fine-tuned representation, and the merged result and the base graph are fed into the fine-tuned action editing model to obtain the output graph.

[0191] The following introduces object motion editing. The object motion editing of the embodiment of the present application is generated based on the diffusion model. For object motion editing, it is first necessary to ensure that the generated image must be sufficiently similar to the original image. The above-mentioned temporal rendering and style rendering models are better in ensuring image consistency (because their training samples are consistent samples, that is, the image content before and after editing is the same), but the object motion editing technology in the related art is more difficult to ensure image consistency before and after.

[0192] Therefore, the embodiment of the present application provides a mechanism for fine-tuning in the application based on a diffusion model (as an action editing model) with better generation effect. By fine-tuning the original object image and action instruction text into the action editing model, the model's recognition of the object image is realized, and then the representation of the fine-tuned action instruction text is merged with the representation of the original action instruction text to generate a new representation, and the new representation is used to drive the action editing model to generate the target image.

[0193] The implementation principle of the diffusion model provided in the embodiment of the present application is to encode the original image after adding noise. Variational auto-encoding can be used to map it to the latent feature space. After the diffusion process, the latent space representation at time T is obtained. The original image features without noise are restored through T noise reduction processes. The restored encoded features are then decoded through variational auto-decoding to obtain the target image. For action instruction text, the text embedding code obtained through the CLIP text branch is controlled by the attention mechanism of the U-shaped network. Diffusion sampling is used to obtain the latent space representation of the noise image encoding vector. Subsequently, in the denoising process of the noise image encoding vector, a fitting of the noise representation is learned to generate. Thus, the original image is subtracted from the noise representation to obtain the actual image representation, and the actual image is obtained through the decoder.

[0194] Since the image-to-image mechanism cannot perform object action editing, the embodiments of the present application use the following methods to ensure object action editing: 1) fine-tune the image information and action instruction text as training samples into the text representation so that the text representation can contain image information; 2) use the image information and control text as training samples to fine-tune the bypass generation control structure of the model - the image information network, and ensure that the image generated by the model is more similar to the original image through this bypass structure; 3) splice the unfine-tuned text representation with the fine-tuned text representation to obtain a text representation with strong control, and then use the fine-tuned model to generate the image.

[0195] As shown in Figure 9, during the fine-tuning process of object action editing, the text representation and the image information network are fine-tuned in sequence. For example, if the edit text entered by the user is "with a smile" and the base image description is "a man", then the action instruction text is "a man with a smile". Here, a random seed i is taken to generate a noise image, and the noise image is superimposed with the base image to generate an overlay image C. The overlay image C is encoded and represented in the latent space (diffusion process) to obtain the noise image encoding vector Z T First, the noise image encoding vector and the original text representation of the action instruction text are input into the T cascaded denoising U-type networks shown in Figure 9 (each sampling layer in the denoising U-type network is an attention layer, and the attention layer is represented by QKV in Figure 9). The intermediate result Z can be obtained through the first denoising U-type network. T-1 ', then pass through the second denoising U-shaped network X T-1 Processing continues, and the final output result Z' is decoded to obtain the output image Y. Based on the difference between the output image Y and the base image, the original textual representation of the action instruction text is fine-tuned. At this point, the original diffusion model is used to fine-tune the original textual representation of the action instruction text, without going through the image information network. This fine-tunes the image information network. Instead of directly fine-tuning the diffusion model structure, the bypass image information network is fine-tuned. By fine-tuning the supervised image information into the bypass image information network, the image information can be embedded into the action editing model.

[0196] As shown in Figure 10, in the inference process of object action editing, a random seed i is taken to generate a noise image, and the noise image is superimposed on the base image to generate an overlay image C. The overlay image C is encoded and represented in the latent space (diffusion process) to obtain the noise image encoding vector Z T. First, fine-tune the original text representation in the above manner, and generate the final text representation (fusion coding) by splicing the fine-tuned text representation and the unfine-tuned text representation. The final text representation and the noise image encoding vector are input into the action editing model that fine-tunes the image information network to generate the output image. Specifically, the action editing model shown in Figure 10 includes T cascaded denoising U-type networks (each sampling layer in the denoising U-type network is an attention layer) and an image information network. The image information network is also obtained by cascading multiple attention layers. The attention layer is represented by QKV in Figure 10. The intermediate result Z can be obtained through the first denoising U-type network. T-1 ', then pass through the second denoising U-shaped network X T-1 Continue processing and decode the final output result Z' to obtain the output image Y.

[0197] Here, the untuned text representation must be placed first. This is because it retains more editing capabilities, while the fine-tuned text representation represents the base image. The base image information cannot provide editing control information, so the untuned text representation must be placed first to ensure higher editing capabilities. It is also important to emphasize that since the action instruction text is short, the first 38 vectors are sufficient to cover all meaningful text representations. Therefore, the final text representation adopts a concatenation of the first 38 and the last 39 vectors to meet the requirements of action editing and base image representation. The merging method can use text representation (embedding) concatenation or a weighted summation method to obtain the final embedding.

[0198] The following describes the fine-tuning process. The fine-tuning image-text pair is (base image, action instruction text), i.e., (base image, a man with a big smile). N (e.g., 20) iterations are performed on a single input image. Each iteration, in which the model is fully trained on the input image, is called an iteration. Training uses sample image-text pairs. For each image-text pair, the original image is added with noise and then fed into the variational autoencoder as a noise map. The text is used to generate constraints, and the original image is also used for loss calculation. The following describes the training scheme.

[0199] First, we initialized the parameters. We used the open-source, pre-trained diffusion model parameters for the variational autoencoder, text encoder, and U-network. In this training, only the U-network parameters were updated; the other parameters remained unchanged. We initialized the learning rate with a learning rate of 0.0004. After every 10 epochs, the learning rate was increased by 0.1, for a total of 20 epochs.

[0200] Next, a random seed i is taken to generate a noise image, and the noise image is superimposed on the original image to generate image x. Image x is represented by the latent space to generate Z TThen, the text information is passed through the text-image comparison model to obtain the text representation, which is input into the action editing model (the text representation is used as KV information), and the Z T Perform T forward calculations of the U-shaped network under the KV constraint, and get Z after the first forward calculation. T-1 , and finally after T times, the U-shaped network output is predicted Z0, and the predicted image is obtained through the decoding network.

[0201] Next, we calculate the batch loss, which is the total loss of the batch of samples. Specifically, we use the MSE loss to calculate the mean square error of the output prediction image and the image-text pair. As an example, see formula (5)

[0202] Among them, y i is the second pixel value at position i in the original image, is the first pixel value at position i in the generated image, and MSE is the difference between the original image and the generated image.

[0203] Then, using the random gradient descent method, the loss is propagated back to the model to obtain the gradient of the model parameters (U-Net) and update the parameters. Finally, all multiple batches of training are completed and the iteration ends.

[0204] In some embodiments, the selected editing mode is determined based on the information in the unified input. Different branches of processing are used depending on the selected editing mode (style rendering, atmosphere rendering, or object motion editing), and the generated image is returned to the user. For style renderings that include both an edit text description and a reference image, two branches of generated effects are provided for the user to choose from.

[0205] The embodiment of the present application introduces object action editing capabilities to achieve semantic-level character image editing, as well as an effective object action editing method. The embodiment of the present application introduces various forms of editing, such as semantic-level image temporal atmosphere rendering, style conversion editing with or without guide images, to achieve rich overall editing functions. The embodiment of the present application uses a unified semantic image editing input interface. After the user specifies the input, the system generates an image based on the unified input information service and returns it for display, reducing the time users spend exploring the application and improving editing efficiency.

[0206] It is understandable that in the embodiments of the present application, when user information and other related data are involved, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0207] The following continues to describe an exemplary structure of the image processing device 255-1 provided in an embodiment of the present application implemented as a software module. In some embodiments, as shown in Figure 2, the software module stored in the image processing device 255-1 in the memory 250 may include an encoding module 2551, which is used to perform noise processing on the object image to obtain a noise image encoding vector, and perform text encoding processing on the action instruction text to obtain a first action text encoding vector; a fine-tuning module 2552, which is used to perform denoising processing on the noise image encoding vector based on the first action text encoding vector to obtain a first action image, and update the first action text encoding vector based on the difference between the first action image and the object image to obtain a second action text encoding vector; a fusion module 2553, which is used to perform fusion processing on the first action text encoding vector and the second action text encoding vector to obtain a fused action text encoding vector; and a generation module 2554, which is used to perform denoising processing on the noise image encoding vector based on the fused action text encoding vector to obtain a second action image, wherein the second action image is the result of applying the action corresponding to the action instruction text to the object included in the object image.

[0208] In some embodiments, text encoding processing is implemented by calling the text model in the text image contrast model. The encoding module 2551 is also used to obtain multiple first text samples and first image samples matching each first text sample; perform image encoding processing on each first image sample through the visual model of the text image contrast model to obtain the image encoding vector of each first image sample; perform text encoding processing on each first text sample through the text model of the text image contrast model to obtain the text encoding vector of each first text sample; determine the text image contrast loss based on the text encoding vector of each first text sample, the image encoding vector of each first image sample, and the matching relationship between each first text sample and each first image sample; and update the parameters of the text image contrast model based on the text image contrast loss.

[0209] In some embodiments, the encoding module 2551 is further configured to perform superposition processing on the object image and the noise image to obtain a superimposed image; and perform image latent space encoding processing on the superimposed image to obtain a noise image coding vector.

[0210] In some embodiments, the denoising process based on the first action text encoding vector is implemented by an image generation model. The image generation model includes N cascaded denoising networks and a decoding network, where the value range of N is 2 ≤ N. The fine-tuning module 2552 is further configured to: denoise the input of the n-th denoising network through the n-th denoising network in the N cascaded denoising networks, and transmit the n-th denoising result output by the n-th denoising network to the (n + 1)-th denoising network for continuous denoising to obtain the (n + 1)-th denoising result corresponding to the (n + 1)-th denoising network; decode the denoising result output by the N-th denoising network through the decoding network to obtain the first action image; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th denoising network is the noise image encoding vector and the first action text encoding vector. When the value range of n is 2 ≤ n < N, the input of the n-th denoising network is the (n - 1)-th denoising result output by the (n - 1)-th denoising network and the first action text encoding vector.

[0211] In some embodiments, the n-th denoising network includes M cascaded attention layers. The fine-tuning module 2552 is further configured to perform first attention processing on the input of the m-th attention layer in the n-th denoising network and the first action text encoding vector to obtain the first attention feature as the m-th attention result of the m-th attention layer in the n-th denoising network; transmit the m-th attention result of the m-th attention layer in the n-th denoising network to the (m + 1)-th attention layer for continuous attention processing to obtain the (m + 1)-th attention result of the (m + 1)-th attention layer in the n-th denoising network; use the M-th attention result output by the M-th attention layer in the n-th denoising network as the n-th denoising result; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the m-th attention layer is the (n - 1)-th fusion denoising result. When the value range of m is 2 ≤ m < M, the input of the m-th attention layer is the (m - 1)-th attention result output by the (m - 1)-th attention layer.

[0212] In some embodiments, the fine-tuning module 2552 is further configured to perform mapping processing on the input of the m-th attention layer based on a query matrix to obtain an attention query matrix; perform mapping processing on the first action text encoding vector based on a key matrix to obtain an attention key matrix; perform mapping processing on the first action text encoding vector based on a value matrix to obtain an attention value matrix; multiply the attention query matrix by the transposed matrix of the attention key matrix to obtain a multiplication result, and obtain the ratio between the multiplication result and the dimension of the attention key matrix; perform maximum likelihood processing on the comparison value, and multiply the maximum likelihood result by the attention value matrix to obtain the first attention feature.

[0213] In some embodiments, the fine-tuning module 2552 is also used to obtain a first pixel value for each position in the first action image and a second pixel value for each position in the object image; for each position, obtain the difference between the first pixel value and the second pixel value; fuse the differences between multiple positions to obtain the difference between the first action image and the object image, and update the first action text encoding vector based on the first loss to obtain the second action text encoding vector.

[0214] In some embodiments, the denoising processing based on the first action text encoding vector is implemented through an image generation model. The fine-tuning module 2552 is also used to update the image generation model based on the difference between the first action image and the object image when the first action text encoding vector is updated based on the difference between the first action image and the object image, so as to obtain an updated image generation model.

[0215] In some embodiments, the fusion module 2553 is further used to perform a first-number-based truncation process on the first action text coding vector to obtain a first truncation coding vector, wherein the first truncation coding vector includes the first-numbered vectors at the front of the first action text coding vector; perform a second-number-based truncation process on the second action text coding vector to obtain a second truncation coding vector, wherein the second truncation coding vector includes the second-numbered vectors at the front of the second action text coding vector; and splice the second truncation coding vector to the end of the first truncation coding vector to obtain a fused action text coding vector.

[0216] In some embodiments, denoising processing based on the first action text coding vector is implemented through an image generation model, and denoising processing based on the fused action text coding vector is implemented through an image editing model, which includes an image generation model and multiple image information networks; before performing denoising processing on the noise image coding vector based on the fused action text coding vector to obtain the second action image, the generation module 2554 is also used to forward propagate the noise image coding vector and the first action text coding vector in the image editing model to obtain a third action image; based on the difference between the third action image and the object image, the multiple image information networks in the image editing model are updated to obtain an updated image editing model.

[0217] In some embodiments, the image generation model includes N cascaded denoising networks and a decoding network, where the value range of N is 2 ≤ N. The image editing model is obtained by configuring an image information network for each denoising network based on the image generation model. Each denoising network and the corresponding image information network constitute a fusion denoising network, and the cascading relationship of the multiple fusion denoising networks is the same as that of the multiple denoising networks; the generation module 2554 is further configured to perform fusion denoising processing on the input of the nth fusion denoising network through the nth fusion denoising network in the N cascaded fusion denoising networks, and transmit the nth fusion denoising result output by the nth fusion denoising network to the (n + 1)th fusion denoising network to continue the fusion denoising processing, so as to obtain the (n + 1)th fusion denoising result corresponding to the (n + 1)th fusion denoising network; perform decoding processing on the fusion denoising result output by the Nth fusion denoising network to obtain the third action image; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the nth fusion denoising network is the noise image coding vector and the first action text coding vector. When the value range of n is 2 ≤ n < N, the input of the nth fusion denoising network is the (n - 1)th fusion denoising result output by the (n - 1)th fusion denoising network and the first action text coding vector.

[0218] In some embodiments, the nth fusion denoising network includes multiple downsampling networks, multiple upsampling networks, and the nth image information network corresponding to the nth denoising network; the generation module 2554 is further configured to perform bypass control processing on the noise image coding vector and the first action text coding vector through the nth image information network to obtain a bypass control result; perform downsampling processing on the first action text coding vector and the bypass control result through the downsampling network to obtain a downsampling result; perform upsampling processing on the downsampling result through the upsampling network to obtain the nth fusion denoising result.

[0219] In some embodiments, the nth image information network includes P cascaded attention layers; the generation module 2554 is further configured to transmit the pth attention result of the pth attention layer in the nth image information network to the (p + 1)th attention layer to continue the second attention processing, so as to obtain the (p + 1)th attention result of the (p + 1)th attention layer in the nth image information network; use the second attention result output by each attention layer as the bypass control result; where p is an integer variable starting from 1 and increasing, and the value range of p is 1 ≤ p ≤ P - 1. When p takes the value of 1, the input of the pth attention layer is the (n - 1)th fusion denoising result. When the value range of p is 2 ≤ p < P, the input of the pth attention layer is the (p - 1)th attention result output by the (p - 1)th attention layer in the nth image information network.

[0220] In some embodiments, the downsampling network includes P cascaded attention layers; the generation module 2554 is further configured to perform first attention processing on the input of the p-th attention layer in the downsampling network and the first action text encoding vector through the p-th attention layer in the downsampling network to obtain a first attention feature; fuse the first attention feature with the p-th attention result output by the p-th attention layer in the n-th image information network to obtain the p-th attention result of the p-th attention layer in the downsampling network; transmit the p-th attention result of the p-th attention layer in the downsampling network to the (p + 1)-th attention layer in the downsampling network to obtain the (p + 1)-th attention result of the (p + 1)-th attention layer in the downsampling network; use the P-th attention result output by the P-th attention layer in the downsampling network as the downsampling result; where p is an integer variable starting from 1 and increasing, and the value range of p is 1 ≤ p ≤ P - 1. When p takes the value of 1, the input of the p-th attention layer is the (n - 1)-th fusion denoising result. When the value range of p is 2 ≤ p < P, the input of the p-th attention layer is the (p - 1)-th attention result output by the (p - 1)-th attention layer.

[0221] Next, the exemplary structure of the implementation of the image processing device 255-2 provided in the embodiments of the present application as a software module will be further described. In some embodiments, as shown in FIG. 2, the software module stored in the memory 250 of the image processing device 255-2 may include an acquisition module 2556, configured to acquire an image editing request, where the image editing request includes any one of the following: an image rendering request, an action editing request; a rendering module 2557, configured to, when the image editing request is an image rendering request, call an image rendering model to perform image rendering processing on the object image carried in the image editing request to obtain a rendered image; an action module 2558, configured to, when the image editing request is an action editing request, call an image editing model to execute the image processing method of the embodiments of the present application on the object image carried in the image editing request to obtain a second action image.

[0222] Next, the exemplary structure of the implementation of the image processing device provided in the embodiments of the present application as a software module will be further described. In some embodiments, the software module stored in the memory of the image processing device may include a display module, configured to display an image editing entry; an input module, configured to display the input image editing information in response to an information input operation at the image editing entry, where the image editing information includes a base image and editing information, and the editing information includes at least one of the following: editing text and a guiding image. The editing text is an action instruction text or a rendering text, and both the guiding image and the rendering text represent a rendering direction; an editing module, configured to display a target image obtained by editing the base image based on the editing information in response to an image processing operation based on the image editing information.

[0223] The present invention provides a computer program product including computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the method described in the present invention.

[0224] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the method provided by the embodiment of the present application.

[0225] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0226] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, or script, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0227] As an example, computer-executable instructions may, but do not necessarily, correspond to a file in a file system, may be stored as part of a file storing other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (for example, files storing one or more modules or subroutines).

[0228] As an example, executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0229] In summary, through the embodiment of the present application, the object image is denoised to obtain a noise image coding vector, and the action instruction text is text-encoded to obtain a first action text coding vector; the noise image coding vector is denoised based on the first action text coding vector to obtain a first action image, and the first action text coding vector is updated based on the difference between the first action image and the object image to obtain a second action text coding vector. This is equivalent to fine-tuning the representation of the action instruction text, thereby ensuring the cognition of the original object image during the image processing process and controlling the consistency during the image editing process. The first action text coding vector and the second action text coding vector are fused to obtain a fused action text coding vector, and the noise image coding vector is denoised based on the fused action text coding vector to obtain the result of applying the action corresponding to the action instruction text to the object included in the object image. Since the result is generated based on the control of the fused action text coding vector, action editing can be achieved while ensuring image consistency.

[0230] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. An image processing method based on artificial intelligence, the method being performed by an electronic device, the method comprising: Performing noise processing on the object image to obtain a noise image coding vector; Performing text encoding processing on the action instruction text to obtain a first action text encoding vector; Performing denoising processing based on the first action text coding vector on the noise image coding vector to obtain a first action image, wherein the denoising processing based on the first action text coding vector is performed iteratively, and the denoising result of each denoising processing is the input of the next denoising processing, and the first action image is obtained by decoding the denoising result of the last denoising processing; Based on the difference between the first action image and the object image, the first action text encoding vector is updated to obtain a second action text encoding vector; The first action text encoding vector and the second action text encoding vector are fused to obtain a fused action text encoding vector; The noise image encoding vector is denoised based on the fused action text encoding vector to obtain a second action image, wherein the second action image is a result of applying the action corresponding to the action instruction text to the object included in the object image.

2. The method according to claim 1, wherein: The text encoding process is implemented by calling a text model in a text image comparison model. Before performing text encoding process on the action instruction text to obtain a first action text encoding vector, the method further includes: Acquire a plurality of first text samples and a first image sample matching each of the first text samples; Performing image coding processing on each of the first image samples by using the visual model of the text image contrast model to obtain an image coding vector for each of the first image samples; Performing text encoding processing on each of the first text samples by using the text model of the text image comparison model to obtain a text encoding vector for each of the first text samples; Determine a text image contrast loss based on a text encoding vector of each of the first text samples, an image encoding vector of each of the first image samples, and a matching relationship between each of the first text samples and each of the first image samples; Parameters of the text image contrast model are updated based on the text image contrast loss.

3. The method according to claim 1, wherein: The step of performing noise processing on the object image to obtain a noise image coding vector includes: Superimposing the object image and the noise image to obtain a superimposed image; Performing image latent space coding processing on the superimposed image to obtain the noise image coding vector.

4. The method according to any one of claims 1 to 3, wherein: Denoising the noise image coding vector based on the first action text coding vector is implemented by an image generation model, wherein the image generation model includes N cascaded denoising networks and decoding networks, and the value of N is greater than or equal to 2; The step of performing denoising processing on the noise image encoding vector based on the first action text encoding vector to obtain a first action image includes: The n-th denoising network among the N cascaded denoising networks performs denoising processing on the input of the n-th denoising network, and transmits the n-th denoising result output by the n-th denoising network to the (n + 1)-th denoising network to continue denoising processing, obtaining the (n + 1)-th denoising result corresponding to the (n + 1)-th denoising network; The decoding network performs decoding processing on the denoising result output by the N-th denoising network to obtain the first action image; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the n-th denoising network is the noise image coding vector and the first action text coding vector, when the value range of n is 2 ≤ n < N, the input of the n-th denoising network is the (n - 1)-th denoising result output by the (n - 1)-th denoising network and the first action text coding vector.

5. The method according to claim 4, wherein: The n-th denoising network includes M cascaded attention layers, and the value of M is greater than or equal to 2; The step of performing denoising processing on the input of the n-th denoising network by the n-th denoising network among the N cascaded denoising networks includes: performing first attention processing on the input of the m-th attention layer in the n-th denoising network and the first action text coding vector through the m-th attention layer in the n-th denoising network to obtain first attention features as the m-th attention result of the m-th attention layer in the n-th denoising network; transmitting the m-th attention result of the m-th attention layer in the n-th denoising network to the (m + 1)-th attention layer to continue attention processing, obtaining the (m + 1)-th attention result of the (m + 1)-th attention layer in the n-th denoising network; taking the M-th attention result output by the M-th attention layer in the n-th denoising network as the n-th denoising result; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m takes the value of 1, the input of the m-th attention layer is the (n - 1)-th denoising result. When the value range of m is 2 ≤ m < M, the input of the m-th attention layer is the (m - 1)-th attention result output by the (m - 1)-th attention layer.

6. The method according to claim 5, wherein: The step of performing first attention processing on the input of the m-th attention layer in the n-th denoising network and the first action text coding vector through the m-th attention layer in the n-th denoising network includes: performing mapping processing on the input of the m-th attention layer based on a query matrix to obtain an attention query matrix; performing mapping processing on the first action text coding vector based on a key matrix to obtain an attention key matrix; performing mapping processing on the first action text coding vector based on a value matrix to obtain an attention value matrix; multiplying the attention query matrix by the transposed matrix of the attention key matrix to obtain a multiplication result, and obtaining the ratio between the multiplication result and the dimension of the attention key matrix; performing maximum likelihood processing on the ratio, and multiplying the maximum likelihood result by the attention value matrix to obtain the first attention features.

7. The method according to any one of claims 1 to 6, wherein: Before updating the first action text encoding vector based on the difference between the first action image and the object image to obtain the second action text encoding vector, the method further includes: Acquire a first pixel value at each position in the first action image and a second pixel value at each position in the object image; For each of the positions, obtaining a difference between the first pixel value and the second pixel value; The differences at the plurality of positions are fused to obtain the difference between the first action image and the object image.

8. The method according to any one of claims 1 to 7, wherein: The denoising process based on the first action text encoding vector is implemented by an image generation model, and the method further includes: When the first action text encoding vector is updated based on the difference between the first action image and the object image, the image generation model is updated based on the difference between the first action image and the object image to obtain an updated image generation model.

9. The method according to any one of claims 1 to 8, wherein: The fusing the first action text encoding vector and the second action text encoding vector to obtain a fused action text encoding vector includes: Performing a first number-based interception process on the first action text encoding vector to obtain a first intercepted encoding vector, wherein the first intercepted encoding vector includes a first number of vectors located at the front of the first action text encoding vector; Performing a second number-based interception process on the second action text encoding vector to obtain a second intercepted encoding vector, wherein the second intercepted encoding vector includes a second number of vectors located at the front of the second action text encoding vector; The second truncated encoding vector is spliced ​​to the end of the first truncated encoding vector to obtain the fused action text encoding vector.

10. The method according to any one of claims 1 to 9, wherein: The denoising process based on the first action text encoding vector is implemented by an image generation model, and the denoising process based on the fused action text encoding vector is implemented by an action editing model, and the action editing model is composed of the image generation model and a plurality of image information networks; Before performing denoising processing on the noise image coding vector based on the fused action text coding vector to obtain the second action image, the method further includes: Forward propagating the noise image encoding vector and the action text encoding vector in the action editing model to obtain a third action image, wherein the action text encoding vector is the first action text encoding vector or the fused action text encoding vector; Based on the difference between the third action image and the object image, multiple image information networks in the action editing model are updated to obtain an updated action editing model.

11. The method according to claim 10, wherein: The image generation model includes N cascaded denoising networks and a decoding network, where N is greater than or equal to 2. The action editing model is obtained by configuring an image information network for each of the denoising networks based on the image generation model. Each of the denoising networks and the corresponding image information network form a fused denoising network, and the cascading relationship of the multiple fused denoising networks is the same as that of the multiple denoising networks; The forward propagation of the noise image encoding vector and the action text encoding vector in the action editing model to obtain a third action image includes: Through the nth fused denoising network in the N cascaded fused denoising networks, perform fused denoising processing on the input of the nth fused denoising network, and transmit the nth fused denoising result output by the nth fused denoising network to the (n + 1)th fused denoising network to continue the fused denoising processing, obtaining the (n + 1)th fused denoising result corresponding to the (n + 1)th fused denoising network; Perform decoding processing on the fused denoising result output by the Nth fused denoising network to obtain the third action image; Where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n takes the value of 1, the input of the nth fused denoising network is the noise image encoding vector and the action text encoding vector. When the value range of n is 2 ≤ n < N, the input of the nth fused denoising network is the (n - 1)th fused denoising result output by the (n - 1)th fused denoising network and the action text encoding vector.

12. The method according to claim 11, wherein: The nth fused denoising network includes multiple downsampling networks, multiple upsampling networks, and the nth image information network corresponding to the nth denoising network; The performing of fused denoising processing on the input of the nth fused denoising network through the nth fused denoising network in the N cascaded fused denoising networks includes: Performing bypass control processing on the noise image encoding vector and the action text encoding vector through the nth image information network to obtain a bypass control result; Performing downsampling processing on the action text encoding vector and the bypass control result through the downsampling network to obtain a downsampling result; Performing upsampling processing on the downsampling result through the upsampling network to obtain the nth fused denoising result.

13. The method according to claim 12, wherein: The nth image information network includes P cascaded attention layers, where P is greater than or equal to 2; The performing of bypass control processing on the noise image encoding vector and the action text encoding vector through the nth image information network to obtain a bypass control result includes: Transmitting the pth attention result of the pth attention layer in the nth image information network to the (p + 1)th attention layer to continue the second attention processing, obtaining the (p + 1)th attention result of the (p + 1)th attention layer in the nth image information network; Taking the second attention result output by each attention layer as the bypass control result; Wherein, p is an integer variable starting from 1 and increasing incrementally, and the value range of p is 1 ≤ p ≤ P - 1. When p takes the value of 1, the input of the p-th attention layer is the (n - 1)-th fusion denoising result. When the value range of p is 2 ≤ p < P, the input of the p-th attention layer is the (p - 1)-th attention result output by the (p - 1)-th attention layer in the n-th image information network.

14. The method according to claim 13, wherein: The downsampling network includes P cascaded attention layers; The downsampling the action text encoding vector and the bypass control result through the downsampling network to obtain a downsampling result includes: Performing first attention processing on the input of the p-th attention layer and the action text encoding vector through the p-th attention layer in the downsampling network to obtain a first attention feature; Fusing the first attention feature with the (p - 1)-th attention result output by the p-th attention layer in the n-th image information network to obtain the p-th attention result of the p-th attention layer in the downsampling network; Transmitting the p-th attention result of the p-th attention layer in the downsampling network to the (p + 1)-th attention layer in the downsampling network to obtain the (p + 1)-th attention result of the (p + 1)-th attention layer in the downsampling network; Taking the P-th attention result output by the P-th attention layer in the downsampling network as the downsampling result; Wherein, p is an integer variable starting from 1 and increasing incrementally, and the value range of p is 1 ≤ p ≤ P - 1. When p takes the value of 1, the input of the p-th attention layer is the (n - 1)-th fusion denoising result. When the value range of p is 2 ≤ p < P, the input of the p-th attention layer is the (p - 1)-th attention result output by the (p - 1)-th attention layer.

15. An artificial intelligence-based image processing method, which is executed by an electronic device, and the method includes: Obtaining an image editing request, wherein the image editing request includes any one of the following: an image rendering request, an action editing request; When the image editing request is an image rendering request, calling an image rendering model to perform image rendering processing on the object image carried by the image editing request to obtain a rendered image; When the image editing request is an action editing request, calling an action editing model to execute the method according to any one of claims 1 to 14 on the object image carried by the image editing request to obtain a second action image.

16. An artificial intelligence-based image processing method, which is executed by an electronic device, and the method includes: Displaying an image editing entry; In response to an information input operation at the image editing entry, displaying the input image editing information, wherein the image editing information includes a base image and editing information, and the editing information includes at least one of the following: editing text and a guiding image, the editing text is action instruction text or rendering text, and both the guiding image and the rendering text represent a rendering direction; When the editing text is an action instruction text, in response to an image processing operation based on the image editing information, a target image obtained by editing the basic image based on the editing information is displayed, wherein the target image is obtained by executing the artificial intelligence-based image processing method described in any one of claims 1 to 14 on the basic image.

17. An image processing device based on artificial intelligence, the device comprising: The encoding module is used to perform noise processing on the object image to obtain a noise image encoding vector, and perform text encoding processing on the action instruction text to obtain a first action text encoding vector; A fine-tuning module, configured to perform denoising processing on the noise image encoding vector based on the first action text encoding vector to obtain a first action image, and to update the first action text encoding vector based on a difference between the first action image and the object image to obtain a second action text encoding vector, wherein the image generation model includes N cascaded denoising networks and decoding networks, and the value of N is greater than or equal to 2; A fusion module, used for fusing the first action text encoding vector and the second action text encoding vector to obtain a fused action text encoding vector; A generation module is used to perform denoising processing on the noise image coding vector based on the fused action text coding vector to obtain a second action image, wherein the second action image is the result of applying the action corresponding to the action instruction text to the object included in the object image.

18. An electronic device, comprising: A memory for storing computer executable instructions; A processor, for implementing the artificial intelligence-based image processing method according to any one of claims 1 to 14, claim 15 or claim 16 when executing computer executable instructions stored in the memory.

19. A computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions, when executed by a processor, implement the artificial intelligence-based image processing method described in any one of claims 1 to 14, claim 15, or claim 16.

20. A computer program product comprising computer executable instructions or a computer program, wherein: When the computer executable instructions or computer program are executed by a processor, the artificial intelligence-based image processing method according to any one of claims 1 to 14, claim 15 or claim 16 is implemented.

Images