An image editing method and related apparatus
By acquiring and transforming visual feature information through a multimodal large language model into conditional embeddings, the problem of image editing models being unable to accurately edit images is solved, resulting in better image editing effects.
Patent Information
- Application Number
- CN202511464322.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-14
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2045-10-14
AI Technical Summary
In existing image editing solutions, image editing models struggle to accurately infer the visual information of the target image based on concise and abstract editing instructions, resulting in poor editing performance.
By utilizing a multimodal large language model to obtain visual feature information of the original image and transforming it into conditional embeddings, an image editing model is guided to perform editing. Through joint training of the multimodal large language model and the image editing model, the accuracy and richness of editing are improved.
By introducing visual feature information obtained from a multimodal large language model as conditional embedding, the original image can be edited under the guidance of conditional embedding to obtain a better edited image.
Smart Images
Figure CN120931769B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and in particular to an image editing method and related apparatus. Background Technology
[0002] In some applications, it is necessary to edit images, such as adding or deleting content from an image.
[0003] Current image editing solutions utilize image editing models to edit the original image based on editing instructions (usually natural language instructions input by the user).
[0004] Current image editing solutions can edit original images according to editing instructions, but the editing results are not good. Summary of the Invention
[0005] In view of this, this application provides an image editing method and related apparatus to solve the problem of poor editing effect in current image editing solutions, and the technical solution is as follows:
[0006] The first aspect of this application provides an image editing method, comprising:
[0007] Acquire the original image and the editing instructions for the original image;
[0008] Using a multimodal large language model, visual feature information of the target image corresponding to the original image is obtained based on the original image and the editing instructions;
[0009] The visual feature information of the target image corresponding to the original image is processed into conditional embeddings to guide image editing;
[0010] Using an image editing model, guided by the conditional embedding, the original image is edited to obtain an edited image.
[0011] In one possible implementation, the step of using a multimodal large language model to obtain the visual feature information of the target image corresponding to the original image based on the original image and the editing instructions includes:
[0012] The target task description text is obtained by concatenating the editing instructions with preset text, wherein the target task description text describes the task of generating description text for the target image corresponding to the original image;
[0013] Using a multimodal large language model, visual feature information of the target image corresponding to the original image is obtained based on the original image and the target task description text. The multimodal large language model is obtained by fine-tuning the training on the description text generation task of the target image corresponding to the first training image using first training data including a first training image and a first training task description text. The first training task description text contains editing instructions for the first training image.
[0014] In one possible implementation, the training process of the multimodal large language model includes:
[0015] Using a multimodal large language model, a description text for the target image corresponding to the first training image is generated based on the first training image and the description text of the first training task.
[0016] Using a summarization model, the descriptive text of the target image corresponding to the first training image is summarized to obtain the image description summary text;
[0017] By adding several predefined image tokens after the image description summary text, the target description text of the target image corresponding to the first training image is obtained.
[0018] With the goal of enabling the multimodal large language model to generate the target description text, the parameters of the multimodal large language model are updated.
[0019] In one possible implementation, the multimodal large language model includes: an image embedding mapping module, a text embedding layer, a decoding layer, and a classification module;
[0020] The step of generating a description text for the target image corresponding to the first training image using a multimodal large language model, based on the first training image and the description text of the first training task, includes:
[0021] The first training task description text is segmented into words to obtain a word sequence, and the image embedding mapping module is used to map the feature information of the first training image to the text feature space to obtain the mapped features.
[0022] Using the text embedding layer, the mapped features and the word sequence are encoded respectively to obtain a first embedding vector and a second embedding vector;
[0023] Using the decoding layer, the first embedding vector and the second embedding vector are decoded to obtain the visual feature information of the target image corresponding to the first training image;
[0024] Using the classification module, a descriptive text for the target image corresponding to the first training image is generated based on the visual feature information of the target image corresponding to the first training image.
[0025] In one possible implementation, processing the visual feature information of the target image corresponding to the original image into a conditional embedding for guiding image editing includes:
[0026] Using a conditional embedding transformation model, the visual feature information of the target image corresponding to the original image is processed into a conditional embedding to guide image editing;
[0027] The conditional embedding transformation model and the image editing model are obtained through joint training using the second training data and with the assistance of a fine-tuned multimodal large language model. The second training data includes a second training image, a second training task description text, and a target image corresponding to the second training image. The second training task description text contains editing instructions for the second training image.
[0028] In one possible implementation, the joint training process of the conditional embedding transformation model and the image editing model includes:
[0029] Using a fine-tuned multimodal large language model, visual feature information of the target image corresponding to the second training image is obtained based on the second training image and the description text of the second training task.
[0030] Using a conditional embedding transformation model, the visual feature information of the target image corresponding to the second training image is converted into a conditional embedding.
[0031] Using an image editing model, guided by the obtained conditional embedding, the target image corresponding to the second training image is reconstructed;
[0032] With the goal of enabling the image editing model to reconstruct the target image corresponding to the second training image, the parameters of the conditional embedding transformation model and the image editing model are updated.
[0033] In one possible implementation, the step of reconstructing the target image corresponding to the second training image using an image editing model, guided by the obtained conditional embedding, includes:
[0034] Noise is added to the target image corresponding to the second training image to obtain a noisy image;
[0035] The second training image is encoded to obtain the latent space vector of the second training image;
[0036] The latent space vector of the second training image is concatenated with the noisy image to obtain the concatenation result;
[0037] Using an image editing model, noise is predicted based on the obtained conditional embedding and stitching results;
[0038] Using an image editing model, the target image corresponding to the second training image is reconstructed based on the stitching result and the predicted noise.
[0039] In one possible implementation, the conditional embedding transformation model includes a decoding layer and a linear layer, wherein the decoding layer is a stack of several decoders;
[0040] Each decoder consists of, in sequence: a self-attention layer, a first normalization layer, a feedforward layer, and a second normalization layer;
[0041] The visual feature information input to the conditional embedding transformation model is processed sequentially by the several decoders and the linear layer, and finally outputs a conditional embedding used to guide image editing.
[0042] A second aspect of this application provides an image editing apparatus, comprising: a data acquisition module, a target image visual feature acquisition module, a target image visual feature processing module, and an image editing module;
[0043] The data acquisition module is used to acquire the original image and editing instructions for the original image;
[0044] The target image visual feature acquisition module is used to acquire the visual feature information of the target image corresponding to the original image based on the original image and the editing instructions using a multimodal large language model.
[0045] The target image visual feature processing module is used to process the visual feature information of the target image corresponding to the original image into a conditional embedding for guiding image editing;
[0046] The image editing module is used to edit the original image using an image editing model, guided by the conditional embedding, to obtain an edited image.
[0047] A third aspect of this application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0048] The memory is used to store computer programs;
[0049] The processor is used to execute the computer program so that the electronic device can implement the steps of any of the above-described image editing methods.
[0050] A fourth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, can implement the steps of any of the above-described image editing methods.
[0051] The fifth aspect of this application discloses a computer program product comprising computer-readable instructions that, when executed on an electronic device, cause the electronic device to perform the steps of any of the above-described image editing methods.
[0052] Considering that editing instructions are often concise and abstract, and image editing models struggle to infer the visual information of the target image based on these instructions, the image editing method provided in this application, after obtaining the original image and editing instructions, first utilizes a multimodal large language model to acquire the visual feature information of the target image corresponding to the original image, based on the original image and editing instructions. Then, the visual feature information obtained using the multimodal large language model is transformed into a conditional embedding for the image editing model to guide the image editing process. Finally, the image editing model, guided by the conditional embedding, edits the original image. The image editing method provided in this application introduces a conditional embedding derived from the visual feature information obtained by the multimodal large language model into the image editing model. Since the conditional embedding can provide accurate and rich guidance information for the editing process, editing the original image under the guidance of the conditional embedding can yield a better-quality edited image. Attached Figure Description
[0053] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0054] Figure 1 This is a schematic diagram of a system architecture involved in this application;
[0055] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of this application;
[0056] Figure 3 A schematic diagram of a server hardware structure provided in an embodiment of this application;
[0057] Figure 4 A flowchart illustrating the image editing method provided in this application embodiment;
[0058] Figure 5A schematic diagram illustrating image editing based on a multimodal large language model, a conditional embedding transformation model, and an image editing model, provided for embodiments of this application;
[0059] Figure 6 A schematic diagram illustrating the process of fine-tuning and training a multimodal large language model, provided for embodiments of this application;
[0060] Figure 7 A schematic diagram illustrating the fine-tuning process of a multimodal large language model provided in an embodiment of this application;
[0061] Figure 8 This is a schematic diagram of the image embedding mapping module in a multimodal large language model provided in an embodiment of this application;
[0062] Figure 9 A schematic diagram illustrating the process of jointly training a conditional embedding transformation model and an image editing model, provided in an embodiment of this application;
[0063] Figure 10 A schematic diagram illustrating the joint training process of the conditional embedding transformation model and the image editing model provided in the embodiments of this application;
[0064] Figure 11 This is a schematic diagram of the structure of the conditional embedding transformation model provided in the embodiments of this application;
[0065] Figure 12 This is a schematic diagram of the structure of the image editing device provided in the embodiments of this application. Detailed Implementation
[0066] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.
[0067] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.
[0068] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.
[0069] In one possible implementation, such as Figure 1 As shown, the system architecture involved in this application may include a terminal 101 and a server 102. The terminal 101 can interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers (…). Figure 1 (The example includes a server). Terminal 101 can obtain the original image and editing instructions for the original image, send the original image and editing instructions to server 102, server 102 edits the original image according to the editing instructions, and sends the edited image to terminal 101.
[0070] In another possible implementation, the system architecture involved in this application may include a terminal. The terminal has strong data processing capabilities, enabling it to acquire the original image and editing instructions for the original image, and to edit the original image according to the editing instructions.
[0071] The product form of the aforementioned terminal will be described next.
[0072] The aforementioned terminals can be mobile phones, tablets, wearable devices, in-vehicle devices, robots, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., and this application embodiment does not impose any limitations on them.
[0073] Figure 2 A schematic diagram of one possible hardware architecture for the terminal is shown.
[0074] refer to Figure 2As shown, the terminal may include components such as a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), a headphone jack 263 (optional), a processor 270, an external interface 280, and a power supply 290. Those skilled in the art will understand that... Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. It may include more or fewer components than shown in the illustration, or combine certain components, or use different components.
[0075] The input unit 230 can be used to receive input numerical or character information, and to generate key signal inputs related to user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect touch operations performed by the user on or near it (such as operations performed by the user using fingers, knuckles, styluses, or any suitable object on or near the touch screen), and drive the corresponding connection devices according to a pre-set program. The touch screen can detect the user's touch actions, convert the touch actions into touch signals and send them to the processor 270, and can receive and execute commands sent by the processor 270; the touch signal includes at least touch point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, various types of touch screens, such as resistive, capacitive, infrared, and surface acoustic wave, can be used to implement the touch screen. Besides the touch screen 231, the input unit 230 may also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.
[0076] The display unit 240 can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display and / or playback of any multimedia file.
[0077] The memory 220 can be used to store instructions and data. The memory 220 may primarily include an instruction storage area and a data storage area. The data storage area can store various types of data, such as multimedia files and text. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or subsets or extended sets thereof. It may also include non-volatile random access memory. It provides the processor 270 with hardware, software, and data resources for managing the computing device, supporting control software and applications. It is also used for storing multimedia files, as well as storing running programs and applications.
[0078] The processor 270 is the control center of the terminal, connecting various parts of the terminal through various interfaces and lines. It executes instructions stored in the memory 220 and calls data stored in the memory 220 to perform various functions and process data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 270. In some embodiments, the processor and memory can be implemented on a single chip; in some embodiments, they can also be implemented on separate chips. The processor 270 can also be used to generate corresponding operation control signals, send them to the corresponding components of the computing device, read and process data in the software, especially read and process data and programs in the memory 220, so that the various functional modules therein perform corresponding functions, thereby controlling the corresponding components to act according to the instructions.
[0079] The memory 220 can be used to store software code related to image editing methods. The processor 270 can execute the software code in the memory 220, and can also schedule other units (such as the input unit 230 and the display unit 240 mentioned above) to achieve the corresponding functions.
[0080] The radio frequency (RF) unit 210 (optional) can be used for receiving and transmitting signals during information transmission or calls. For example, it can receive downlink information from the base station and process it for the processor 270; additionally, it can transmit uplink data to the base station. Typically, the RF unit 210 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc. Furthermore, the RF unit 210 can also communicate wirelessly with network devices and other devices. This wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.
[0081] In this embodiment, the radio frequency unit 210 can transmit data to other devices and receive data from other devices. It should be understood that the radio frequency unit 210 is optional and can be replaced with other communication interfaces, such as a network port.
[0082] The terminal also includes a power supply 290 (such as a battery) that supplies power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.
[0083] The terminal also includes an external interface 280, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the terminal to other devices for communication or to connect a charger to charge the terminal.
[0084] Although not shown, the terminal may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.
[0085] The product form of the above-mentioned server will be described next.
[0086] Figure 3 A schematic diagram of the above-mentioned server structure is provided, such as... Figure 3 As shown, the server may include a bus 301, a processing device 302, a communication interface 303, and a storage device 304. The processing device 302, the storage device 304, and the communication interface 303 communicate with each other via the bus 301.
[0087] Bus 301 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0088] The processing device 302 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).
[0089] Storage device 304 may include volatile memory, such as random access memory (RAM). Storage device 304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0090] Storage device 304 can be used to store software code related to image editing methods. Processing device 302 can call the software code stored in storage device 304, and can also schedule other units to achieve the corresponding functions.
[0091] The processor 270 in the aforementioned terminal and the processing device 302 in the server can be hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors, or microcontrollers, etc.) or combinations of these hardware circuits. For example, they can be hardware systems with instruction execution capabilities, such as CPUs or DSPs, or hardware systems without instruction execution capabilities, such as ASICs or FPGAs, or combinations of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.
[0092] In current image editing solutions, image editing models directly edit the original image based on editing instructions. Since the editing instructions are natural language commands input by the user, such as "remove the building" or "make the skycloudy," and these natural language commands are often concise and abstract, the image editing model has difficulty accurately inferring information such as the range and degree of the target visual transformation from such expressions. This leads to the final edited image deviating from expectations.
[0093] To address the problems existing in current image editing solutions, this application proposes a more effective image editing method. The following embodiments will describe the image editing method provided by this application.
[0094] Please see Figure 4 The diagram illustrates a flowchart of an image editing method provided in an embodiment of this application. This image editing method may include:
[0095] Step S401: Obtain the original image and editing instructions for the original image.
[0096] The original image can be represented as I src It is the image to be edited, targeting the original image I. src The editing command can be represented as T ins It is a natural language instruction, such as "Add a flower in front of the snowy mountain".
[0097] It should be noted that this embodiment uses the original image I. src The editing can be appearance adjustment editing (only changing the visual presentation of the image without substantial modification of pixel content, such as brightness adjustment, contrast adjustment, etc.) or content transformation editing (changing pixel content, such as adding or deleting objects, etc.). Accordingly, the editing instruction T in this embodiment... ins It can be an editing command for adjusting the appearance or an editing command for changing the content.
[0098] Step S402: Using a multimodal large language model, obtain the visual feature information of the target image corresponding to the original image based on the original image and the editing instructions for the original image.
[0099] It should be noted that the target image corresponding to the original image is the edited image that the user expects to obtain.
[0100] In one possible implementation, the process of obtaining visual feature information of the target image corresponding to the original image based on the original image and editing instructions for the original image, using a multimodal large language model, may include:
[0101] Step S4021: Obtain the target task description text by concatenating the preset text with the editing instructions.
[0102] The target task description text describes the original image I. src The task of generating descriptive text for the corresponding target image.
[0103] For example, the editing instruction is "Add a flower in front of the snowy mountain", and the preset text is "What will this image be like if". Concatenating the preset text with the editing instruction yields the target task description text "What will this image be like if Add a flower in front of the snowy mountain". This target task description text contains the editing instruction T. ins It describes the task of generating descriptive text for the target image corresponding to the original image.
[0104] Step S4022: Using a multimodal large language model, obtain the visual feature information of the target image corresponding to the original image based on the original image and the target task description text.
[0105] like Figure 5 As shown, the original image I src The image is used as the input to the multimodal large language model, and the target task description text is used as the text input to the multimodal large language model. The multimodal large language model is then used to analyze the original image I. src And the target task description text, to obtain the original image I src The visual feature information of the corresponding target image.
[0106] In this embodiment, the multimodal large language model is obtained by fine-tuning training on the task of generating descriptive text for the target image corresponding to the first training image, using first training image and first training task description text. The first training task description text contains editing instructions for the first training image and describes the task of generating descriptive text for the target image corresponding to the first training image.
[0107] Step S403: Process the visual feature information of the target image corresponding to the original image into a conditional embedding to guide image editing.
[0108] In obtaining the original image I src After obtaining the visual feature information of the corresponding target image, the original image I is... src The visual feature information of the corresponding target image is transformed into a conditional embedding that can guide the image editing model and is used to guide the editing process of the image editing model.
[0109] In this embodiment, the visual feature information of the target image corresponding to the original image is used as conditional information and encoded into an embedding vector, which is used as the conditional input of the image editing model to guide the direction of image generation or editing.
[0110] In one possible implementation, such as Figure 5 As shown, the conditional embedding transformation model can be used to transform the visual feature information of the target image corresponding to the original image into a conditional embedding used to guide image editing.
[0111] Step S404: Using an image editing model, under the guidance of conditional embedding, the original image is edited to obtain the edited image.
[0112] like Figure 5 As shown, after obtaining the conditional embedding used to guide image editing, the original image I... src Conditional embeddings are used as input to the image editing model. Guided by the conditional embeddings, the image editing model processes the original image I... src Edit the image and output the edited image.
[0113] In one possible implementation, the conditional embedding transformation model and the image editing model are obtained through joint training using second training data and with the assistance of a fine-tuned multimodal large language model. The second training data includes a second training image, a second training task description text, and a target image corresponding to the second training image. The second training task description text contains editing instructions for the second training image and describes the task of generating description text for the target image corresponding to the second training image.
[0114] Considering that editing instructions are often concise and abstract, and image editing models struggle to infer the visual information of the target image based on these instructions, the image editing method provided in this application, after obtaining the original image and the editing instructions for it, first utilizes a multimodal large language model to acquire the visual feature information of the target image corresponding to the original image, based on the original image and the editing instructions. Then, the visual feature information obtained using the multimodal large language model is transformed into a conditional embedding for the image editing model to guide the image editing process. Finally, the image editing model, guided by the conditional embedding, edits the original image. The image editing method provided in this application introduces a conditional embedding derived from the visual feature information obtained by the multimodal large language model into the image editing model. Because this conditional embedding provides accurate and rich guidance information for the editing process, editing the original image under the guidance of the conditional embedding can yield a better-quality edited image.
[0115] The above embodiments mention that the multimodal large language model is obtained by fine-tuning training on the task of generating descriptive text for the target image corresponding to the first training image using first training data (including first training images and first training task description text). In another embodiment of this application, the fine-tuning training process of the multimodal large language model is described.
[0116] Please see Figure 6 This illustrates the fine-tuning training process of a multimodal large language model, which may include:
[0117] Step S601: Using a multimodal large language model, generate a description text for the target image corresponding to the first training image based on the first training image and the description text of the first training task.
[0118] The descriptive text of the target image corresponding to the first training image is the text describing the target image corresponding to the first training image.
[0119] In one possible implementation, such as Figure 7 As shown, the multimodal large language model may include an image embedding mapping module 701, a text embedding layer 702, a decoding layer 703, and a classification module 704.
[0120] The process of generating descriptive text for the target image corresponding to the first training image using a multimodal large language model, based on the first training image and the descriptive text for the first training task, may include:
[0121] Step S6011a: Using the image embedding mapping module 701 of the multimodal large language model, the feature information of the first training image is mapped to the text feature space to obtain the mapped features.
[0122] like Figure 7As shown, the first training image is input into the image embedding mapping module 701 of the multimodal large language model. Through the image embedding mapping module 701, the feature information of the first training image is mapped to the text feature space of the multimodal large language model, providing the multimodal large language model with real-world image information perception, so that it can become a cross-modal large language model with image-text pairs as input.
[0123] Please see Figure 8 This diagram illustrates a network structure for an image embedding mapping module 701, which may include a first linear layer, an image encoder (e.g., a ViT image encoder), a second linear layer, and a cross-attention layer (e.g., a single-layer cross-attention layer). First, a set of learnable parameter vectors (learnable query vectors) is randomly initialized. This vector passes through the first linear layer to obtain a query matrix Q. The input image (the first training image) is encoded by the image encoder and then passes through the second linear layer to obtain a key matrix K and a value matrix V. The query matrix Q, key matrix K, and value matrix V pass through the cross-attention layer to obtain image features mapped onto the text feature space. The cross-attention calculation formula is shown below:
[0124] (1).
[0125] Where, d k To query the dimensions of matrix Q and key matrix K, the size of the attention is used to scale the attention size.
[0126] Step S6011b: Segment the first training task description text into words to obtain a word sequence.
[0127] like Figure 7 As shown, the text describing the first training task can be input into the word segmenter for word segmentation to obtain a word sequence.
[0128] Step S6012a: Encode the mapped features using the text embedding layer 702 of the multimodal large language model to obtain the first embedding vector.
[0129] The mapped features output by the image embedding mapping module 701 are input into the text embedding layer 702 for encoding, and the text embedding layer 702 outputs the first embedding vector.
[0130] Step S6012b: Encode the word sequence using the text embedding layer 702 of the multimodal large language model to obtain the second embedding vector.
[0131] The word sequence output by the word segmenter is input into the text embedding layer 702 for encoding, and the text embedding layer 702 outputs the second embedding vector.
[0132] Step S6013: Using the decoding layer 703 of the multimodal large language model, decode the first embedding vector and the second embedding vector to obtain the visual feature information of the target image corresponding to the first training image.
[0133] The first embedding vector and the second embedding vector are input into the decoding layer 703, and the decoding layer 703 outputs the visual feature information of the target image corresponding to the first training image.
[0134] Step S6014: Using the classification module 704 of the multimodal large language model, generate descriptive text for the target image corresponding to the first training image based on the visual feature information of the target image corresponding to the first training image.
[0135] The visual feature information of the target image corresponding to the first training image is input into the classification module 704, and the classification module 704 outputs the descriptive text of the target image corresponding to the first training image.
[0136] Step S602: Using the summarization model, summarize the descriptive text of the target image corresponding to the first training image to obtain the image description summary text.
[0137] Considering that the image description text generated by multimodal large language models is usually quite long and involves some abstract descriptions, in order to avoid misleading subsequent image editing, this embodiment uses a summarization model to summarize the description text of the target image corresponding to the first training image after obtaining the description text of the target image corresponding to the first training image, so as to obtain a concise and clear image description summary text.
[0138] Optionally, the summary model can be Spark-X1, but this embodiment is not limited to it. Other models with text summary functions are also applicable to this application.
[0139] Step S603: Add several set image tokens after the image description summary text to obtain the target description text of the target image corresponding to the first training image.
[0140] Considering that the image features output by the decoding layer of the multimodal large language model are limited to the text modality, this embodiment adds several special image tokens, such as "[IMG1], [IMG2], ..., [IMGn]", after the image description summary text to overcome this limitation. The multimodal large model is trained to learn how to generate these tokens. These special image tokens can be seen as the understanding of image-related editing instructions in the image description summary text, serving as a bridge between the image modality and the text modality.
[0141] Step S604: Update the parameters of the multimodal large language model with the target description text of the target image corresponding to the first training image generated by the multimodal large language model as the objective.
[0142] Specifically, the cross-entropy loss can be determined based on the description text obtained in step S601 and the target description text obtained in step S603, and then the parameters of the multimodal large language model can be updated based on the cross-entropy loss.
[0143] In one possible implementation, parameters can be updated for all parts of the multimodal large language model. In order to avoid catastrophic forgetting of the capabilities of the multimodal large language model, in another possible implementation, the parameters of the decoding layer 703 can be locked, and only the parameters of the image embedding mapping module 701, the text embedding layer 702, and the classification module 704 can be updated.
[0144] Using multiple different first training data, the multimodal large language model is trained multiple times according to the above steps S601 to S604 until the training termination condition is met.
[0145] In this embodiment, the purpose of fine-tuning the multimodal large model is to enable it to learn and output a clear and concise description of the target image.
[0146] After obtaining the fine-tuned multimodal large model, a second training data (including a second training image, a second training task description text, and a target image corresponding to the second training image) can be used to jointly train the conditional embedding transformation model and the image editing model with the assistance of the fine-tuned multimodal large language model. In another embodiment of this application, the joint training process of the conditional embedding transformation model and the image editing model is described.
[0147] like Figure 9 As shown, the joint training process of the conditional embedding transformation model and the image editing model may include:
[0148] Step S901: Using the fine-tuned multimodal large language model, obtain the visual feature information of the target image corresponding to the second training image based on the second training image and the description text of the second training task.
[0149] It should be noted that when jointly training the conditional embedding transformation model and the image editing model, the parameters of the fine-tuned multimodal large language model are locked.
[0150] like Figure 10As shown, the process of obtaining the visual feature information of the target image corresponding to the second training image based on the second training image and the description text of the second training task using a fine-tuned multimodal large language model may include: segmenting the description text of the second training task using a word segmenter to obtain a word sequence, and using the image embedding mapping module 701 of the fine-tuned multimodal large language model to map the feature information of the second training image to the text feature space to obtain the mapped features; encoding the obtained word sequence and the mapped features using the text embedding layer 702 of the fine-tuned multimodal large language model respectively; and decoding the embedding vector output by the text embedding layer 702 using the decoding layer 703 of the fine-tuned multimodal large language model to obtain the visual feature information of the target image corresponding to the second training image.
[0151] It should be noted that the visual feature information of the target image corresponding to the second training image is the feature output of the last hidden layer of the decoding layer when the fine-tuned multimodal large language model generates each image token, while the visual feature information of the target image corresponding to the second training image is a feature sequence. :
[0152] (2).
[0153] Among them, h i This is the feature output of the last hidden layer of the decoding layer when generating the i-th image token in the fine-tuned multimodal large language model. n is the number of image tokens.
[0154] Step S902: Using the conditional embedding transformation model, the visual feature information of the target image corresponding to the second training image is transformed into a conditional embedding.
[0155] The visual feature information of the target image corresponding to the second training image is input into the conditional embedding transformation model, which then transforms the input visual feature information into a conditional embedding used to guide image editing.
[0156] In one possible implementation, such as Figure 11 As shown, the conditional embedding transformation model can include a decoding layer and a linear layer. The decoding layer consists of a stack of M (e.g., 8) decoders, which are the visual feature information input to the conditional embedding transformation model. After being processed sequentially through M decoders and linear layers, the final output is a conditional embedding e used to guide image editing. f .
[0157] Each decoder includes a self-attention layer and a feedforward layer, and a normalization layer (Layer Norm) is set after the self-attention layer and the feedforward layer respectively. That is, each decoder includes a self-attention layer, a first normalization layer, a feedforward layer and a second normalization layer in sequence.
[0158] The visual feature information input to the conditional embedding transformation model is a feature sequence. The self-attention layer enables the model to consider all elements in the feature sequence when generating conditional embeddings. The normalization layer stabilizes the training process, and the feedforward layer further processes the feature information. This layer stacking allows the conditional embedding transformation model to effectively process sequence information. The layers are connected by residuals, which helps the gradient propagation. Finally, a linear layer transforms the output of the decoding layer into the final conditional embedding. In this way, the conditional embedding transformation model transforms the visual feature information provided by the multimodal large language model into conditional embeddings that can guide the image editing model.
[0159] Step S903: Using the image editing model, under the guidance of the obtained conditional embedding, reconstruct the target image corresponding to the second training image.
[0160] The image editing model in this embodiment can be a diffusion model (such as Stable Diffusion 1.5).
[0161] Specifically, the process of reconstructing the target image corresponding to the second training image using an image editing model, guided by the obtained conditional embedding, may include: adding noise (Gaussian noise) to the target image corresponding to the second training image to obtain a noisy image; encoding the second training image (e.g., VAE encoding) to obtain the latent space vector of the second training image; concatenating the latent space vector of the second training image with the noisy image to obtain a concatenated result; using the image editing model, predicting noise based on the obtained conditional embedding and the concatenated result; and using the image editing model, reconstructing the target image corresponding to the second training image based on the concatenated result and the predicted noise.
[0162] In this embodiment, the noisy image of the second training image and the target image corresponding to the second training image are concatenated and then input into the image editing model. Conditional embedding is injected into the image editing model through a cross-attention mechanism. The cross-attention mechanism can be expressed as the following equations (3) and (4):
[0163] (3);
[0164] (4).
[0165] Where Q represents the query matrix, K represents the key matrix, V represents the value matrix, and d k This represents the dimensionality of the query matrix Q and the key matrix K, and o represents the latent space vector obtained after encoding the second training image (e.g., VAE encoding). Representing a time series, z t Represents the latent space characteristics at time t. ef This indicates conditional embedding. and This represents a learnable attention matrix. This represents the flattening operation, Concat(z) t ,o) means to combine o and z t Then, the parts are assembled.
[0166] Step S904: With the goal of enabling the image editing model to reconstruct the target image corresponding to the second training image, update the parameters of the conditional embedding transformation model and the image editing model.
[0167] Specifically, with the goal of enabling the image editing model to reconstruct the target image corresponding to the second training image, the process of updating the parameters of the conditional embedding transformation model and the image editing model may include: determining the noise prediction loss of the image editing model based on the added noise and the predicted noise; and updating the parameters of the conditional embedding transformation model and the image editing model based on the noise prediction loss.
[0168] In this embodiment, the noise prediction loss can be the mean squared error loss, which is shown below:
[0169] (5).
[0170] in, This represents the noise added during the forward diffusion process. This represents the noise in the prediction.
[0171] Using multiple different second training data, the conditional embedding transformation model and the image editing model are trained multiple times according to the above steps S901 to S904 until the training termination condition is met.
[0172] After obtaining the trained multimodal large language model, as well as the trained conditional embedding transformation model and image editing model, these models can be used to process the original image to be edited. src The editor.
[0173] Specifically, after obtaining the original image I to be edited src and targeting I src Editing command T ins Then, first, by combining the preset text with the editing command T... ins The target task description text is obtained by concatenation, and then the trained multimodal large language model is used to analyze the original image I. src And the target task description text, obtain the original image I src The visual feature information of the corresponding target image is then used to transform the original image I using the trained conditional embedding transformation model. srcThe visual feature information of the corresponding target image is converted into conditional embeddings. Finally, the trained editing model, guided by the obtained conditional embeddings, is used to process the original image I. src Edit.
[0174] Among them, the trained multimodal large language model is used to analyze the original image I. src And the target task description text, obtain the original image I src The process of obtaining the visual feature information of the target image is similar to the process described above, which uses a trained multimodal large language model to obtain the visual feature information of the target image corresponding to the second training image based on the second training image and the second task description text. The trained conditional embedding transformation model is then used to transform the original image I... src The process of converting the visual feature information of the target image into conditional embedding is similar to the process of converting the visual feature information of the target image corresponding to the second training image into conditional embedding using the conditional embedding transformation model described above. This embodiment will not elaborate on this process.
[0175] This application also provides an apparatus for performing the image editing method provided in the above embodiments. Please refer to... Figure 12 , Figure 12 This is a schematic diagram of the structure of an image editing device provided in an embodiment of this application. The image editing device may include: a data acquisition module 1201, a target image visual feature acquisition module 1202, a target image visual feature processing module 1203, and an image editing module 1204.
[0176] The data acquisition module 1201 is used to acquire the original image and the editing instructions for the original image.
[0177] The target image visual feature acquisition module 1202 is used to acquire the visual feature information of the target image corresponding to the original image by using a multimodal large language model, based on the original image and the editing instructions for the original image.
[0178] The target image visual feature processing module 1203 is used to process the visual feature information of the target image corresponding to the original image into a conditional embedding for guiding image editing.
[0179] Image editing module 1204 is used to edit the original image using an image editing model under the guidance of conditional embedding, to obtain the edited image.
[0180] In one possible implementation, when the target image visual feature acquisition module 1202 acquires the visual feature information of the target image corresponding to the original image based on the original image and editing instructions using a multimodal large language model, it is specifically used for:
[0181] The target task description text is obtained by concatenating editing instructions with preset text. The target task description text describes the task of generating description text for the target image corresponding to the original image.
[0182] Using a multimodal large language model, visual feature information of the target image corresponding to the original image is obtained based on the original image and the target task description text. The multimodal large language model is obtained by fine-tuning the training on the description text generation task of the target image corresponding to the first training image using first training data including a first training image and a first training task description text. The first training task description text contains editing instructions for the first training image.
[0183] In one possible implementation, the image editing apparatus provided in this application embodiment further includes a multimodal large language model training module for fine-tuning training of the multimodal large language model.
[0184] The multimodal large language model training module is specifically used for fine-tuning the training of the multimodal large language model:
[0185] Using a multimodal large language model, a description text for the target image corresponding to the first training image is generated based on the first training image and the description text for the first training task.
[0186] Using a summarization model, the descriptive text of the target image corresponding to the first training image is summarized to obtain the image description summary text;
[0187] Add several set image tokens after the image description summary text to obtain the target description text of the target image corresponding to the first training image;
[0188] The goal is to enable the multimodal large language model to generate target descriptive text, and the parameters of the multimodal large language model are updated accordingly.
[0189] In one possible implementation, a multimodal large language model may include: an image embedding mapping module, a text embedding layer, a decoding layer, and a classification module;
[0190] Using a multimodal large language model, based on the first training image and the description text of the first training task, a description text for the target image corresponding to the first training image is generated, including:
[0191] The first training task description text is segmented into words to obtain word sequences. Then, the image embedding mapping module is used to map the feature information of the first training image to the text feature space to obtain the mapped features.
[0192] By using a text embedding layer, the mapped features and word sequences are encoded respectively to obtain the first embedding vector and the second embedding vector;
[0193] Using the decoding layer, the first embedding vector and the second embedding vector are decoded to obtain the visual feature information of the target image corresponding to the first training image;
[0194] Using the classification module, a descriptive text for the target image corresponding to the first training image is generated based on the visual feature information of the target image corresponding to the first training image.
[0195] In one possible implementation, when the target image visual feature processing module 1203 processes the visual feature information of the target image corresponding to the original image into a conditional embedding for guiding image editing, it specifically performs the following:
[0196] By using a conditional embedding transformation model, the visual feature information of the target image corresponding to the original image is processed into a conditional embedding to guide image editing;
[0197] Among them, the conditional embedding transformation model and the image editing model use the second training data, which are obtained through joint training with the help of the fine-tuned multimodal large language model. The second training data includes the second training image, the second training task description text, and the target image corresponding to the second training image. The second training task description text contains editing instructions for the second training image.
[0198] In one possible implementation, the image editing apparatus provided in this application embodiment further includes a joint training module for jointly training the conditional embedding transformation model and the image editing model.
[0199] The joint training module is specifically used for: jointly training the conditional embedding transformation model and the image editing model.
[0200] Using a fine-tuned multimodal large language model, visual feature information of the target image corresponding to the second training image is obtained based on the second training image and the description text of the second training task.
[0201] The visual feature information of the target image corresponding to the second training image is converted into a conditional embedding using a conditional embedding model.
[0202] Using an image editing model, guided by the obtained conditional embedding, the target image corresponding to the second training image is reconstructed;
[0203] With the goal of enabling the image editing model to reconstruct the target image corresponding to the second training image, the parameters of the conditional embedding transformation model and the image editing model are updated.
[0204] In one possible implementation, the joint training module utilizes an image editing model, guided by the obtained conditional embeddings, to reconstruct the target image corresponding to the second training image, specifically for:
[0205] Noise is added to the target image corresponding to the second training image to obtain a noisy image;
[0206] The second training image is encoded to obtain the latent space vector of the second training image;
[0207] The latent space vector of the second training image is concatenated with the noisy image to obtain the concatenated result;
[0208] Using an image editing model, noise is predicted based on the obtained conditional embedding and stitching results;
[0209] Using an image editing model, the target image corresponding to the second training image is reconstructed based on the splicing result and the predicted noise.
[0210] In one possible implementation, the joint training module aims to enable the image editing model to reconstruct the target image corresponding to the second training image. When updating the parameters of the conditional embedding transformation model and the image editing model, it is specifically used for:
[0211] The noise prediction loss of the image editing model is determined based on the added noise and the predicted noise.
[0212] Based on the noise prediction loss, the parameters of the conditional embedding transformation model and the image editing model are updated.
[0213] In one possible implementation, the conditional embedding transformation model may include a decoding layer and a linear layer, wherein the decoding layer is a stack of several decoders; each decoder includes, in sequence, a self-attention layer, a first normalization layer, a feedforward layer, and a second normalization layer; the visual feature information input to the conditional embedding transformation model is processed sequentially by several decoders and linear layers, and finally outputs a conditional embedding used to guide image editing.
[0214] Each module in the aforementioned image editing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware within or independently of the processing unit of the computer device, or stored in software within the computer device's memory, so that the processor can invoke and execute the operations corresponding to each module.
[0215] This application also provides an electronic device, which may include at least one processor and a memory connected to the processor.
[0216] The processor may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application; the memory may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage.
[0217] The memory is used to store computer programs, and the processor is used to execute the computer programs so that the electronic device can implement the steps of the image editing method provided in the above embodiments.
[0218] This application also provides a computer storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the image editing method provided in the above embodiments.
[0219] This application also provides a computer program product, including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to perform the steps of the image editing method provided in the above embodiments.
[0220] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0221] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0222] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0223] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
Claims
1. An image editing method, characterized in that, include: Acquire the original image and the editing instructions for the original image; Using a multimodal large language model, visual feature information of the target image corresponding to the original image is obtained based on the original image and the editing instructions; The visual feature information of the target image corresponding to the original image is processed into conditional embeddings to guide image editing; Using an image editing model, guided by the conditional embedding, the original image is edited to obtain an edited image; The step of processing the visual feature information of the target image corresponding to the original image into a conditional embedding for guiding image editing includes: Using a conditional embedding transformation model, the visual feature information of the target image corresponding to the original image is processed into a conditional embedding to guide image editing; The conditional embedding transformation model includes a decoding layer and a linear layer. The decoding layer is a stack of several decoders. Each decoder includes a self-attention layer, a first normalization layer, a feedforward layer, and a second normalization layer. The visual feature information input to the conditional embedding transformation model is processed by the several decoders and the linear layer in sequence, and finally outputs a conditional embedding for guiding image editing.
2. The image editing method according to claim 1, characterized in that, The step of using a multimodal large language model to obtain visual feature information of the target image corresponding to the original image based on the original image and the editing instructions includes: The target task description text is obtained by concatenating the editing instructions with preset text, wherein the target task description text describes the task of generating description text for the target image corresponding to the original image; Using a multimodal large language model, visual feature information of the target image corresponding to the original image is obtained based on the original image and the target task description text. The multimodal large language model is obtained by fine-tuning the training on the description text generation task of the target image corresponding to the first training image using first training data including a first training image and a first training task description text. The first training task description text contains editing instructions for the first training image.
3. The image editing method according to claim 2, characterized in that, The training process of the multimodal large language model includes: Using a multimodal large language model, a description text for the target image corresponding to the first training image is generated based on the first training image and the description text of the first training task. Using a summarization model, the descriptive text of the target image corresponding to the first training image is summarized to obtain the image description summary text; By adding several predefined image tokens after the image description summary text, the target description text of the target image corresponding to the first training image is obtained. With the goal of enabling the multimodal large language model to generate the target description text, the parameters of the multimodal large language model are updated.
4. The image editing method according to claim 3, characterized in that, The multimodal large language model includes: an image embedding mapping module, a text embedding layer, a decoding layer, and a classification module; The step of generating a description text for the target image corresponding to the first training image using a multimodal large language model, based on the first training image and the description text of the first training task, includes: The first training task description text is segmented into words to obtain a word sequence, and the image embedding mapping module is used to map the feature information of the first training image to the text feature space to obtain the mapped features. Using the text embedding layer, the mapped features and the word sequence are encoded respectively to obtain a first embedding vector and a second embedding vector; Using the decoding layer, the first embedding vector and the second embedding vector are decoded to obtain the visual feature information of the target image corresponding to the first training image; Using the classification module, a descriptive text for the target image corresponding to the first training image is generated based on the visual feature information of the target image corresponding to the first training image.
5. The image editing method according to claim 2, characterized in that, The conditional embedding transformation model and the image editing model are obtained through joint training using the second training data and with the assistance of a fine-tuned multimodal large language model. The second training data includes a second training image, a second training task description text, and a target image corresponding to the second training image. The second training task description text contains editing instructions for the second training image.
6. The image editing method according to claim 5, characterized in that, The joint training process of the conditional embedding transformation model and the image editing model includes: Using a fine-tuned multimodal large language model, visual feature information of the target image corresponding to the second training image is obtained based on the second training image and the description text of the second training task. Using a conditional embedding transformation model, the visual feature information of the target image corresponding to the second training image is converted into a conditional embedding. Using an image editing model, guided by the obtained conditional embedding, the target image corresponding to the second training image is reconstructed; With the goal of enabling the image editing model to reconstruct the target image corresponding to the second training image, the parameters of the conditional embedding transformation model and the image editing model are updated.
7. The image editing method according to claim 6, characterized in that, The step of reconstructing the target image corresponding to the second training image using an image editing model, guided by the obtained conditional embedding, includes: Noise is added to the target image corresponding to the second training image to obtain a noisy image; The second training image is encoded to obtain the latent space vector of the second training image; The latent space vector of the second training image is concatenated with the noisy image to obtain the concatenation result; Using an image editing model, noise is predicted based on the obtained conditional embedding and stitching results; Using an image editing model, the target image corresponding to the second training image is reconstructed based on the stitching result and the predicted noise.
8. An image editing device, characterized in that, include: The module includes a data acquisition module, a target image visual feature acquisition module, a target image visual feature processing module, and an image editing module. The data acquisition module is used to acquire the original image and editing instructions for the original image; The target image visual feature acquisition module is used to acquire the visual feature information of the target image corresponding to the original image based on the original image and the editing instructions using a multimodal large language model. The target image visual feature processing module is used to process the visual feature information of the target image corresponding to the original image into a conditional embedding for guiding image editing; The image editing module is used to edit the original image using an image editing model, guided by the conditional embedding, to obtain an edited image; When the target image visual feature processing module processes the visual feature information of the target image corresponding to the original image into a conditional embedding for guiding image editing, it specifically performs the following functions: Using a conditional embedding transformation model, the visual feature information of the target image corresponding to the original image is processed into a conditional embedding to guide image editing; The conditional embedding transformation model includes a decoding layer and a linear layer. The decoding layer is a stack of several decoders. Each decoder includes a self-attention layer, a first normalization layer, a feedforward layer, and a second normalization layer. The visual feature information input to the conditional embedding transformation model is processed by the several decoders and the linear layer in sequence, and finally outputs a conditional embedding for guiding image editing.
9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the steps of the image editing method as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to perform the steps of the image editing method as described in any one of claims 1 to 7.
11. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to perform the steps of the image editing method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Image conversion method and system, computer equipment and storage medium
CN119128200A
Image editing method and device, storage medium and electronic equipment
CN119399327A