Image editing method, training method, device, electronic equipment and storage medium

Abstract

The application provides an image editing method and device, a training method and device, an electronic device and a storage medium. The method comprises: obtaining a source image to be processed, source text and target text; encoding the source text and the target text by using a pre-trained image editing model to obtain a source text vector corresponding to the source text and a target text vector corresponding to the target text; calculating an editing direction vector based on the target text vector and the source text vector; inputting the source image into the image editing model to obtain a source image vector corresponding to the source image; obtaining a target image vector based on the editing direction vector and the source image vector; and processing the target image vector by using the image editing model to obtain a target image corresponding to the target text. The application can improve the image editing efficiency and reduce the image editing time.

Description

Technical Field

[0001] This application relates to the field of image processing, specifically to an image editing method, training method, apparatus, electronic device, and storage medium. Background Technology

[0002] When using an image editing model, text and an image describing the editing direction are input. The model is then iteratively optimized based on the generated images until the actual image described in the text is obtained. This method results in low image editing efficiency and a long editing time. Summary of the Invention

[0003] In view of this, this application provides an image editing method, training method, apparatus, electronic device, and storage medium, which can effectively improve image editing efficiency and reduce image editing time.

[0004] One embodiment of this application provides an image editing method, comprising: acquiring a source image to be processed, source text, and target text, wherein the source text is descriptive text corresponding to the source image, the target text is descriptive text corresponding to the target image, and the target image is an image obtained by editing the source image; encoding the source text and target text using a pre-trained image editing model to obtain a source text vector corresponding to the source text and a target text vector corresponding to the target text; calculating an editing direction vector based on the target text vector and the source text vector; inputting the source image into the image editing model to obtain a source image vector corresponding to the source image; obtaining the target image vector based on the editing direction vector and the source image vector; and processing the target image vector using the image editing model to obtain the target image corresponding to the target text.

[0005] This application obtains the image editing direction based on the source text vector and the target text vector, and performs image editing based on the source image vector to obtain the target image vector. This makes image editing no longer limited to a single editing scenario, enabling image editing for multiple editing scenarios without the need for iterative optimization during image editing, thus improving image editing efficiency and reducing image editing time.

[0006] In some embodiments, the image editing model includes a text encoder, which encodes the source text and target text using a pre-trained image editing model to obtain a source text vector corresponding to the source text and a target text vector corresponding to the target text, including:

[0007] The source text and target text are encoded by the text encoder to obtain the source text vector and the target text vector; the step of calculating the editing direction vector based on the target text vector and the source text vector includes: calculating the vector difference between the target text vector and the source text vector to obtain the editing direction vector.

[0008] In some embodiments, the image editing model includes: a feature space transformation network, wherein the source image vector is a vector in a target feature space; the step of obtaining the target image vector based on the editing direction vector and the source image vector includes: inputting the editing direction vector into the feature space transformation network to obtain a target editing direction vector located in the target feature space; and adding the target editing direction vector to the source image vector to obtain the target image vector.

[0009] In some embodiments, the image editing model further includes an inversion network and an image generator. The inversion network is used to generate a vector located in the target feature space based on the image, and the image generator is used to generate an image based on the vector in the target feature space. The step of inputting the source image into the image editing model to obtain the source image vector corresponding to the source image includes: inputting the source image into the inversion network in the image editing model to obtain the source image vector, wherein the source image vector is a vector in the target feature space; the step of processing the target image vector through the image editing model to obtain the target image corresponding to the target text includes: processing the target image vector through the image generator to obtain the target image.

[0010] In some embodiments, obtaining the source image, source text, and target text to be processed includes: obtaining the user's image editing instructions through a human-computer interaction interface; determining the source image, source text, and target text to be processed based on the image editing instructions; and after processing the target image vector through the image editing model to obtain the target image corresponding to the target text, further including: displaying the target image through a human-computer interaction interface.

[0011] An embodiment of this application also provides a method for training an image editing model, comprising:

[0012] Obtain the first and second initial vectors;

[0013] The first initial vector and the second initial vector are processed by the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector.

[0014] The source image sample and the target image sample are encoded to obtain the source image vector corresponding to the source image sample and the target image vector corresponding to the target image sample;

[0015] Based on the target image vector and the source image vector, the editing direction vector is calculated;

[0016] Based on the edit direction vector and the first initial vector, determine the prediction vector of the second initial vector;

[0017] Based on the predicted vector of the second initial vector and the second initial vector, the total loss value is calculated; if the total loss value meets the preset threshold condition, the image editing model is determined to be trained successfully.

[0018] During training, the image editing model of this application does not require text describing the image editing direction. Instead, it obtains the image editing direction vector based on the target image vector and the source image vector. Training based on this vector enables the trained image editing model to be no longer limited to a single editing scenario, but to achieve image editing for multiple editing scenarios. Furthermore, it eliminates the need for iterative optimization during image editing, thereby improving image editing efficiency and reducing image editing time.

[0019] In some embodiments, the image editing model includes a text encoder; encoding the source image sample and the target image sample to obtain the source image vector corresponding to the source image sample and the target image vector corresponding to the target image sample includes: encoding the source image sample and the target image sample using a preset image encoder to obtain the source image vector and the target image vector; the image encoder and the text encoder belong to the same contrastive image-text pre-trained model; calculating the editing direction vector based on the target image vector and the source image vector includes: calculating the vector difference between the target image vector and the source image vector to obtain the editing direction vector.

[0020] In some embodiments, the first initial vector and the second initial vector are both vectors in the target feature space, and the image editing model further includes a feature space transformation network;

[0021] The step of determining the prediction vector of the second initial vector based on the edit direction vector and the first initial vector includes: inputting the edit direction vector into the feature space transformation network to obtain the target edit direction vector located in the target feature space; and adding the target edit direction vector to the first initial vector to obtain the prediction vector of the second initial vector.

[0022] In some embodiments, the image editing model further includes an inversion network and an image generator, wherein the inversion network is used to generate vectors located in a target feature space based on the image, and the image generator is used to generate an image based on the vectors in the target feature space;

[0023] The step of processing the first initial vector and the second initial vector through the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector includes: processing the first initial vector and the second initial vector through the image generator to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector.

[0024] After processing the first initial vector and the second initial vector using the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector, the method further includes: inputting the target image sample into the inversion network to obtain a target inversion vector, and / or inputting the source image sample into the inversion network to obtain a source inversion vector; determining a first loss value based on a first sub-loss value and / or a second sub-loss value, wherein the first sub-loss value is a loss value determined according to the source inversion vector and the first initial vector, and the second sub-loss value is a loss value determined according to the target inversion vector and the second initial vector;

[0025] The step of calculating the total loss value based on the predicted vector and the second initial vector includes: calculating a second loss value based on the difference between the predicted vector and the second initial vector; and obtaining the total loss value based on the sum of the first loss value and the second loss value.

[0026] In some embodiments, the image generator is an image generator in an adversarial training network, which further includes an image discriminator;

[0027] After processing the first initial vector and the second initial vector using the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector, the method further includes: inputting the source image sample and / or the target image sample into the image discriminator to obtain a first discriminant output value corresponding to the source image sample and / or a second discriminant output value corresponding to the target image sample; and obtaining a third loss value based on the first discriminant output value and / or the second discriminant output value.

[0028] The step of obtaining the total loss value based on the sum of the first loss value and the second loss value includes obtaining the total loss value based on the sum of the first loss value, the second loss value, and the third loss value.

[0029] An embodiment of this application also provides an image editing apparatus, including:

[0030] The system includes: an acquisition module for acquiring a source image, source text, and target text to be processed; a source text being a descriptive text corresponding to the source image, a target text being a descriptive text corresponding to the target image, and a target image being an image obtained by editing the source image; an encoding module for encoding the source text and target text using a pre-trained image editing model to obtain a source text vector corresponding to the source text and a target text vector corresponding to the target text; a calculation module for calculating an editing direction vector based on the target text vector and the source text vector; an inversion module for inputting the source image into the image editing model to obtain a source image vector corresponding to the source image; and a generation module for obtaining the target image vector based on the editing direction vector and the source image vector, and for processing the target image vector through the image editing model to obtain the target image corresponding to the target text.

[0031] An embodiment of this application also provides a training apparatus for an image editing model, comprising:

[0032] The system includes: an acquisition module for acquiring a first initial vector and a second initial vector; a generation module for processing the first initial vector and the second initial vector using the current image editing model to obtain a source image sample corresponding to the first initial vector and a target image sample corresponding to the second initial vector; an encoding module for encoding the source image sample and the target image sample to obtain a source image vector corresponding to the source image sample and a target image vector corresponding to the target image sample; a calculation module for calculating an editing direction vector based on the target image vector and the source image vector; a determination module for determining a predicted vector of the second initial vector based on the editing direction vector and the first initial vector; a judgment module for calculating a total loss value based on the predicted vector of the second initial vector and the second initial vector; and a determination module for determining that the image editing model training is complete if the total loss value meets a preset threshold condition.

[0033] This application also provides an electronic device, which includes a processor and a memory. The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the above-described image editing method or the above-described image editing model training method.

[0034] This application also provides a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the image editing method described above, or the image editing model training method described above. Attached Figure Description

[0035] Figure 1A schematic diagram of the hardware structure of a robot provided in an embodiment of this application;

[0036] Figure 2 This is a schematic diagram of the mechanical structure of a robot provided in one embodiment of this application;

[0037] Figure 3 This is a flowchart of the steps of an image editing method provided in an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of an image editing model provided in an embodiment of this application;

[0039] Figure 5 This is a flowchart illustrating the steps of a training method for an image editing model provided in an embodiment of this application;

[0040] Figure 6 This is an overall architecture diagram for training an image editing model provided in one embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the structure of an image editing device provided in an embodiment of this application;

[0042] Figure 8 This is a schematic diagram of the structure of a training device for an image editing model provided in one embodiment of this application;

[0043] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0044] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0045] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.

[0046] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.

[0047] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0048] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.

[0049] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0050] This application provides an image editing method, an image editing model training method, an image editing device, an image editing model training device, an electronic device, and a storage medium.

[0051] The image editing method and image editing model training method of this application can be applied to one or more electronic devices. This electronic device is capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, processors, microprogrammed controllers (MCUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0052] First, this application provides an image editing method. The execution subject of the image editing method is an image editing device, which can be applied to an electronic device, such as a robot or other types of electronic devices, such as smart terminals.

[0053] Please see Figure 1 , Figure 1 This is a schematic diagram of the hardware structure of a robot 100 provided in one embodiment of this application. The robot 100 can be any of various robot types, specifically including but not limited to at least one of wheeled robots, legged robots, tracked robots, crawling robots, worm-like robots, or swimming robots. For example, the robot 100 can specifically be a legged robot, or a robot combining legged and wheeled configurations. Legged robots include monopodial robots, bipodial robots, or multipodial robots. Multipodial robots refer to robots with three or more legs; for example, a multipodial robot can specifically be a quadrupedal robot. A robot is a machine capable of semi-autonomous or fully autonomous operation. Robots are not limited to humanoid machine devices and can also include robots with configurations such as dog-shaped, horse-shaped, snake-shaped, or fish-shaped. For example, a robot can specifically be a quadrupedal robotic horse. Figure 1 In the illustrated embodiment, robot 100 includes a mechanical unit 101, a communication unit 102, a sensing unit 103, an interface unit 104, a storage unit 105, a display unit 106, an input unit 107, a control module 110, and a power supply 111. The various components of robot 100 can be connected in any way, including wired or wireless connections. Those skilled in the art will understand that... Figure 1 The specific structure of the robot 100 shown does not constitute a limitation on the robot 100. The robot 100 may include more or fewer parts than shown. Some parts are not essential components of the robot 100 and may be omitted or combined as needed without changing the nature of the invention.

[0054] Figure 2 This is a schematic diagram of the mechanical structure of a robot according to an embodiment of this application. The following is in conjunction with... Figure 1 and Figure 2 A detailed introduction to each component of Robot 100:

[0055] Mechanical unit 101 is the hardware of robot 100. For example... Figure 1 As shown, the mechanical unit 101 may include a drive board 1011, a motor 1012, and a mechanical structure 1013, such as... Figure 2As shown, the mechanical structure 1013 may include a main body 1014, extendable legs 1015, and feet 1016. In other embodiments, the mechanical structure 1013 may also include an extendable robotic arm (not shown), a rotatable head structure 1017, a rocking tail structure 1018, a cargo-carrying structure 1019, a saddle structure 1020, a camera structure 1021, etc. It should be noted that the various component modules of the mechanical unit 101 can be one or multiple, depending on the specific situation. For example, there may be four legs 1015, and each leg 1015 may be equipped with three motors 1012, resulting in a total of twelve motors 1012.

[0056] The communication unit 102 can be used for receiving and sending signals, and can also communicate with networks and other devices. For example, it can receive instructions from a remote control or other robot 100 to move in a specific direction at a specific speed according to a specific gait, and then transmit these instructions to the control module 110 for processing. The communication unit 102 includes modules such as WiFi, 4G, 5G, Bluetooth, and infrared modules.

[0057] The sensing unit 103 is used to acquire information data about the environment surrounding the robot 100 and to monitor parameter data of various components inside the robot 100, and then sends this data to the control module 110. The sensing unit 103 includes various sensors, such as sensors for acquiring information about the surrounding environment: lidar (for remote object detection, distance determination, and / or velocity determination), millimeter-wave radar (for short-range object detection, distance determination, and / or velocity determination), cameras, infrared cameras, and Global Navigation Satellite System (GNSS). Sensors for monitoring various components inside the robot 100 include: an inertial measurement unit (IMU) (for measuring velocity, acceleration, and angular velocity values), foot sensors (for monitoring the position of the foot's contact point, foot posture, magnitude and direction of the contact force), and temperature sensors (for detecting component temperature). Other sensors that can be configured on the robot 100, such as load sensors, touch sensors, motor angle sensors, and torque sensors, are not detailed here.

[0058] The interface unit 104 can be used to receive input from external devices (e.g., data, power, etc.) and transmit the received input to one or more components within the robot 100, or it can be used to output to external devices (e.g., data, power, etc.). The interface unit 104 may include a power port, a data port (such as a USB port), a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, etc.

[0059] Storage unit 105 is used to store software programs and various data. Storage unit 105 may mainly include a program storage area and a data storage area. The program storage area may store operating system programs, motion control programs, application programs (such as text editors), etc.; the data storage area may store data generated by the robot 100 during use (such as various sensor data acquired by the sensing unit 103, log file data, etc.). Furthermore, storage unit 105 may include high-speed random access memory, and may also include non-volatile memory, such as disk storage, flash memory, or other volatile solid-state memory.

[0060] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0061] Input unit 107 can be used to receive input numerical or character information. Specifically, input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as touch screen, can collect user touch operations (such as operations performed by the user using their palm, fingers, or suitable accessories on or near touch panel 1071) and drive corresponding connection devices according to a pre-set program. Touch panel 1071 may include two parts: touch detection device 1073 and touch controller 1074. Touch detection device 1073 detects the user's touch position and the signal generated by the touch operation, and transmits the signal to touch controller 1074; touch controller 1074 receives touch information from touch detection device 1073, converts it into touch point coordinates, and sends it to control module 110, and can also receive and execute commands from control module 110. In addition to touch panel 1071, input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include, but are not limited to, one or more of the following: remote control handles, etc., without any specific limitation here.

[0062] Furthermore, the touch panel 1071 can cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the control module 110 to determine the type of touch event. Subsequently, the control module 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Although in Figure 1In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components that implement input and output functions respectively. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to implement input and output functions. The specific implementation is not limited here.

[0063] The control module 110 is the control center of the robot 100. It connects all the components of the robot 100 through various interfaces and lines. It controls the robot 100 as a whole by running or executing the software program stored in the storage unit 105 and calling the data stored in the storage unit 105.

[0064] Power supply 111 supplies power to various components. Power supply 111 may include a battery and a power control board. The power control board controls battery charging, discharging, and power consumption management. Figure 1 In the illustrated embodiment, power supply 111 is electrically connected to control module 110. In other embodiments, power supply 111 may also be electrically connected to sensing unit 103 (such as camera, radar, speaker, etc.) and motor 1012. It should be noted that each component may be connected to a different power supply 111, or may be powered by the same power supply 111.

[0065] Based on the above embodiments, specifically, in some embodiments, a terminal device can be used to communicate with the robot 100. When the terminal device communicates with the robot 100, it can send instruction information to the robot 100. The robot 100 can receive the instruction information through the communication unit 102 and, upon receiving the instruction information, can transmit it to the control module 110, so that the control module 110 can process the instruction information to obtain the target speed value. The terminal device includes, but is not limited to, mobile phones, tablets, servers, personal computers, wearable smart devices, and other electrical appliances with image capture capabilities.

[0066] The instruction information can be determined based on preset conditions. In one embodiment, the robot 100 may include a sensing unit 103, which can generate instruction information based on the current environment of the robot 100. The control module 110 can determine whether the current speed value of the robot 100 meets the corresponding preset conditions based on the instruction information. If it does, the robot 100 will maintain its current speed value and current gait; if it does not, the control module 110 will determine a target speed value and a corresponding target gait based on the corresponding preset conditions, thereby controlling the robot 100 to move at the target speed value and the corresponding target gait. Environmental sensors may include temperature sensors, air pressure sensors, vision sensors, and sound sensors. Instruction information may include temperature information, air pressure information, image information, and sound information. The communication method between the environmental sensors and the control module 110 can be wired or wireless. Wireless communication methods include, but are not limited to: wireless networks, mobile communication networks (3G, 4G, 5G, etc.), Bluetooth, and infrared.

[0067] It is understood that the use of suffixes such as "module," "component," or "unit" to denote parts is merely for the convenience of describing the invention and has no specific meaning in itself. Therefore, "module," "component," or "unit" can be used interchangeably.

[0068] Figure 3 This is a flowchart illustrating the steps of an image editing method according to an embodiment of this application. The order of the steps in the flowchart can be changed, and some steps can be omitted, depending on different requirements.

[0069] See Figure 3 As shown, the image editing method may include the following steps.

[0070] Step 301: Obtain the source image, source text, and target text to be processed. The source text is the descriptive text corresponding to the source image, the target text is the descriptive text corresponding to the target image, and the target image is the image obtained by editing the source image.

[0071] For example, if the source image displays "cat" and the corresponding descriptive text is "cat", and the user wants to edit the "cat" displayed in the source image to "dog", the target text can be "dog".

[0072] In some embodiments, the robot can obtain image editing instructions from the user through a human-computer interaction interface; based on the image editing instructions, it determines the source image, source text, and target text to be processed; after the electronic device edits the source image based on the source text and target text to obtain the target image, the electronic device can display the target image through the human-computer interaction interface.

[0073] The human-computer interaction interface can be a voice interaction interface, such as a user inputting image editing commands through a voice interaction interface, or a human-computer interaction interface displayed on a monitor, where the user inputs image editing commands, but is not limited to these.

[0074] In this embodiment, users can interact with electronic devices to obtain target images, thereby improving the user experience.

[0075] The following briefly describes the process of electronic devices interacting with users to obtain a target image in the application scenario of a robot greeting friends. If the user does not have a photo of their friend, the robot can obtain an average face and its corresponding descriptive text. The average face is used as the source image to be processed, and the corresponding descriptive text is used as the source text. The user can then interact with the robot to edit the image. The user can transmit the target text to the robot through the human-computer interaction interface, such as saying "eyes bigger," "orange shirt," "jeans," "long hair," etc. The robot edits the source image based on the user's input target text and displays the edited target image on the human-computer interaction interface until the face in the target image resembles the friend to be greeted. At this point, the friend can be successfully greeted even without a photo.

[0076] Step 302: Encode the source text and target text using a pre-trained image editing model to obtain the source text vector corresponding to the source text and the target text vector corresponding to the target text.

[0077] The source text vector is the text feature extracted from the source text, and the target text vector is the text feature extracted from the target text.

[0078] In some embodiments, reference may be made to Figure 4 As shown, Figure 4 This is a schematic diagram of an image editing model. The image editing model may include: a text encoder 401, which can be used to extract text features to obtain text vectors. Step 302 may include: encoding the source text and the target text through the text encoder to obtain the source text vector and the target text vector.

[0079] The text encoder 401 can be the text encoder (CLIPText Encoder) in Contrastive Language-Image Pretraining (CLIP). This text encoder can adopt the text transformer model commonly used in Natural Language Processing (NLP), but is not limited to it.

[0080] In this embodiment, the source text is input into a text encoder to obtain a source text vector; the target text is input into a text encoder to obtain a target text vector. For example, the source text and the target text can be encoded by CLIP Text Encoder to obtain two 1*512 dimension vectors, namely the source text vector and the target text vector.

[0081] In this embodiment, the source text and target text are encoded separately by the text encoder 401, so that the source text vector and the target text vector are located in the same feature space, which facilitates the subsequent processing of the source text vector and the target text vector.

[0082] Step 303: Calculate the editing direction vector based on the target text vector and the source text vector.

[0083] Edit direction vectors can be used to characterize the differences between target text and source text.

[0084] In some embodiments, step 303 may include: calculating the vector difference between the target text vector and the source text vector to obtain the editing direction vector. That is, the vector difference between the target text vector and the source text vector can be used as the editing direction vector.

[0085] This embodiment accurately reflects the editing direction of the source image by using the vector difference between the target text vector and the source text vector, thus facilitating the subsequent acquisition of an accurate target image based on the editing direction vector and the source image. In other embodiments, step 303 may also include: performing layer normalization on both the target text vector and the source text vector, and then calculating the vector difference between the layer-normalized target text vector and the layer-normalized source text vector to obtain the editing direction vector. That is, using the vector difference between the layer-normalized target text vector and the layer-normalized source text vector as the editing direction vector, layer normalization can effectively stabilize the hidden state of the dynamic neural network.

[0086] Step 304: Input the source image into the image editing model to obtain the source image vector corresponding to the source image.

[0087] The source image vector is the image feature extracted from the source image.

[0088] In some embodiments, reference may be made to Figure 4As shown, the image editing model may further include: an inversion network 402, which can be denoted as an E-network. The inversion network 402 is used to extract image features to obtain an image vector. Step 304 may include: inputting the source image into the inversion network 402 to obtain the source image vector corresponding to the source image. For example, inputting the image tensor of a 3*512*512 source image into the inversion network can yield a 1*512 source image vector.

[0089] In some embodiments, to facilitate subsequent processing of the source image vectors and to extract more reasonable image vectors, the inversion network 402 can be used to generate vectors located in the target feature space based on the image. The target feature space can be a w-space, which is a subset of the image feature space, and the vectors in the w-space have a more linear relationship.

[0090] This embodiment processes the source image through an inversion network to obtain the source image vector located in the target feature space, which facilitates subsequent vector operations in the target feature space. In addition, the interrelationships between vectors in the target feature space are more linear, enabling the source image vector to accurately reflect the image features.

[0091] The structure of the inversion network 402 can be: multi-layer convolution + batch normalization (BN) + rectified linear unit (ReLU) + residual, but is not limited to this.

[0092] Step 305: Based on the editing direction vector and the source image vector, obtain the target image vector.

[0093] The target image vector can be used to characterize the features of the target image.

[0094] In some embodiments, the editing direction vector and the source image vector can be first converted into vectors in the same feature space, and then the editing direction vector and the source image vector can be summed to obtain the target image vector.

[0095] For example, if the source image vector is a vector in the target feature space, the editing direction vector can be transformed into a vector in the target feature space, so that the source image vector and the target image vector are located in the same feature space.

[0096] Specifically, in some embodiments, the editing direction vector can first be input into the feature space transformation network 403 to obtain the target editing direction vector located in the target feature space. Then, the target editing direction vector and the source vector located in the target feature space are added together to obtain the target image vector.

[0097] This embodiment uses a feature space transformation network to convert the editing direction vector into a vector located in the target feature space, so that the editing direction vector is converted into a vector located in the same target feature space as the source image vector, thereby facilitating the operation between the target editing direction vector and the source image vector.

[0098] When the target feature space is W space, the feature space transformation network 403 can be a W network. A W network can transform the feature space of a vector to W space.

[0099] The structure of a W network can include Layer normalization and Multilayer Perceptron (MLP). A Multilayer Perceptron can have three layers stacked, for example, a structure of a Multilayer Perceptron can be three stacks of fully connected + Batch Normalization + ReLU, but it is not limited to this.

[0100] When the edit direction vector is the vector difference between the target text vector and the source text vector, the W network can first process the input edit direction vector through layer normalization, and then process it through a 3-layer multilayer perceptron to obtain the target edit direction vector located in the W space.

[0101] When the editing direction vector is the vector difference between the target text vector and the source text vector after layer normalization, the vector difference between the target text vector and the source text vector after layer normalization can be input into the 3-layer MLP in the W network for processing.

[0102] After obtaining the target image vector, step 306 can be executed.

[0103] Step 306: Process the target image vector using an image editing model to obtain the target image corresponding to the target text.

[0104] In some embodiments, the image editing model may further include an image generator (Generative Model) 403, which may be referred to as a G network. The image generator is used to generate a target image based on the target image vector. Step 306 may further include: inputting the target image vector into the image generator 403 to obtain the target vector corresponding to the target text.

[0105] For example, inputting the target image vector into the image generator 403 will yield a vector of 3*512*512 dimensions.

[0106] In this application embodiment, a generative adversarial network (GAN), such as the generative model in StyleGAN, can be used as the image generator 403.

[0107] The image generator can have a total of 18 layers. Each layer obtains a tensor with a higher resolution through convolution, modulation of the mapped vector, and modulation of noise. For example, a 4*4 tensor as a learnable fixed input can be used to obtain a 512*512 resolution image after 18 layers.

[0108] This application embodiment is based on the vector difference between the source text vector and the target text vector, which can accurately obtain the editing direction of the image, and perform image editing based on the source image vector to obtain the target image vector. It is no longer limited to a single editing scenario, but can realize image editing for multiple editing scenarios without iterative optimization during image editing, thereby improving image editing efficiency and reducing image editing time.

[0109] This application also provides a training method for an image editing model. This training method is used to train the image editing model described in the above embodiments. The training method can be applied to an image editing model training device, which can be used in electronic devices such as servers. This application does not limit this application.

[0110] The following combination Figure 5 and Figure 6 The training method for the image editing model in the above embodiments of the specification. Wherein, Figure 5 This is a flowchart illustrating the steps of a training method for an image editing model provided in one embodiment of this application. Figure 6 This is an overall architecture diagram for training an image editing model according to an embodiment of this application. It is understood that the order of steps in the flowchart can be changed, some steps can be omitted, and the overall architecture can be modified accordingly, depending on different requirements.

[0111] Training methods for image editing models include:

[0112] Step 501: Obtain the first initial vector and the second initial vector.

[0113] In some embodiments, step 501 may include: first, obtaining noise vector Z1 and noise vector Z2, for example, by randomly sampling a Gaussian distribution to obtain noise vector Z1 and noise vector Z2; then, inputting noise vector Z1 and noise vector Z2 into a mapping network 601 to obtain a first initial vector W1 corresponding to noise vector Z1 and a second initial vector W2 corresponding to noise vector Z2. The mapping network may include 8 fully connected layers, and is essentially an MLP. The mapping network can process a 1*512 vector to obtain a new 1*512 vector.

[0114] For example, two 1*512 dimensional noise vectors are obtained by random sampling through a Gaussian distribution, one as noise vector Z1 and the other as noise vector Z2. Then, Z1 and Z2 are respectively input into the mapping network to obtain 1*512 dimensional W1 and W2.

[0115] The main function of the mapping network 601 is to project randomly sampled noise vectors onto the target feature space, where the first initial vector W1 and the second initial vector W2 are vectors located in the target feature space. The target feature space can be the w-space, which is a subset of the image feature space where the vectors have a more linear relationship.

[0116] Mapping networks can decouple the latent space. In this embodiment, the mapping network can map Gaussian noise to another space (w space). The mapped vector is more reasonable and closer to the real image distribution.

[0117] Step 502: Process the first initial vector and the second initial vector using the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector.

[0118] In some embodiments, the image editing model may include an image generator 602, which is used to generate an image based on vectors in the target feature space. The electronic device may input a first initial vector W1 and a second initial vector W2 into the image generator 602 to obtain a source image sample img1 and a target image sample img2.

[0119] In this embodiment, a Generative Adversarial Network (GAN), such as the generative model in StyleGAN, can be used as the image generator 602. The GAN model also includes a discriminative model, which can be used as the image discriminator 603.

[0120] Image generator 602 and image discriminator 603 are two competing neural network models in a GAN. Image generator 602 takes noise as input and generates samples; image discriminator 603 receives generator data and real training data and is trained to obtain a classifier that can correctly distinguish data types. Image generator 602 and image discriminator 603 compete against each other. The goal of image generator 602 is to learn to generate new samples that more closely approximate real data, thus fooling image discriminator 603, while image discriminator 603 aims to better distinguish between generated and real data. After training these two networks jointly for a large number of rounds, image discriminator 603 will be unable to distinguish between generated samples and real data.

[0121] In some embodiments, after acquiring source image samples and target image samples, the source image samples and / or target image samples can be input into the image discriminator 603 to obtain a first discrimination output value corresponding to the source image sample and / or a second discrimination output value corresponding to the target image sample; based on the first discrimination output value and / or the second discrimination output value, a third loss value is obtained.

[0122] For example, the first discriminant output value is input into the GAN loss function to obtain the loss value corresponding to the source image sample, and the second discriminant output value is input into the GAN loss function to obtain the loss value corresponding to the target image sample. The third loss value can be chosen from either the loss value corresponding to the source image sample or the loss value corresponding to the target sample, or the two can be combined using a weighted average. The GAN loss function includes generator loss and discriminant loss.

[0123] The objective function of GAN can be shown below:

[0124] min G max D V(D,G)=E x～q(x) [logD(x)]+E z～p(z) [log(1-D(()))];

[0125] E represents the expectation, q(x) represents the distribution of the sampled real images x, D(x) represents the probability that x is a real image, p(z) represents the distribution of random noise z, G(z) represents the image generated by random noise z through image generator G, and D(G(z)) represents the probability that the generated image is a real image.

[0126] During training, the labels of real images can be set to 1, and the labels of generated images can be set to 0. When training the image discriminator D, in the above objective function, it is expected that the real images will get closer and closer to 1, that is, D(x) = 1, while the generated images will get closer and closer to 0, that is, D(G(z)) = 0. When training the image generator G, it is expected that D(G(z)) will tend to 1, thus obtaining the GAN loss.

[0127] In some embodiments, the image editing model may further include an inversion network 604, which may also be referred to as an E-network, for generating vectors based on the image in the target feature space.

[0128] The structure of the inversion network can be: multi-layer convolution + batch normalization (BN) + rectified linear unit (ReLU) + residual, but is not limited to this.

[0129] In some embodiments, after obtaining the source image sample img1 and the target image sample img2, firstly, the target image sample img2 can be input into the inversion network 604 to obtain the target inversion vector, and the source image sample img1 can be input into the inversion network 604 to obtain the source inversion vector; then, the first loss value, i.e., the inversion network loss (LossE), is calculated based on the source inversion vector, the first initial vector, the target inversion vector, and the second initial vector.

[0130] For example, a first sub-loss value is determined based on the source inversion vector and the first initial vector; a second sub-loss value is determined based on the target inversion vector and the second initial vector; and the inversion network loss is obtained based on the first sub-loss value and / or the second sub-loss value. For example, the first sub-loss value can be used as the first loss value, or the second sub-loss value can be used as the first loss value, or the first sub-loss value and the second sub-loss value can be combined by weighted averaging, etc., to obtain the first loss value. The embodiments of this application are not limited to this.

[0131] The loss function used in the above calculation of the inversion network loss can be the Mean Square Error Loss (MSE). The MSE represents the Euclidean distance between the predicted value and the true value; the closer the predicted value and the true value are, the smaller their MSE. In the calculation of the first loss value, the source inversion vector is the predicted value, the first initial vector is the true value corresponding to the source inversion vector, the target inversion vector is the predicted value, and the second initial vector is the true value corresponding to the target inversion vector.

[0132] In this embodiment, the inversion network is trained using source inversion vectors and target inversion vectors. The inversion network can generate vectors located in the target feature space based on the image, which facilitates subsequent vector operations in the target feature space. In addition, the interrelationships between vectors in the target feature space are more linear, so that the vectors obtained after processing by the inversion network can accurately reflect the image features.

[0133] After obtaining the source image sample img1 and the target image sample img2, step 503 can be executed.

[0134] Step 503: Encode the source image sample and the target image sample to obtain the source image vector corresponding to the source image sample and the target image vector corresponding to the target image sample.

[0135] In some embodiments, the source image sample img1 and the target image sample vector img2 can be input into the image encoder 605 respectively to obtain the source image vector (imgvector1) and the target image vector (imgvector2). For example, inputting a 3*512*512 image into the image encoder 605 can obtain a 1*512 vector.

[0136] The image editing model includes a text encoder 401 and an image encoder 605. The text encoder 401 in the image editing model comes from the same clip model. That is to say, the clip includes two encoders: a text encoder and an image encoder (CLIPImage Encoder). The image encoder is used to extract features from the image and can use common CNN models, vision transformer models, VGG models, etc.

[0137] During training, the image is encoded by the image encoder in the clip model, so that the source image vector and the target image vector are located in the same feature space, which facilitates the subsequent operation and processing of the source image vector and the target image vector. In addition, the text encoder in the image editing model of this embodiment is the text encoder in the clip model, which can map the image and text to the same feature space, making it easier to train the image editing model.

[0138] After obtaining the target image vector and the source image vector, step 504 can be executed.

[0139] Step 504: Calculate the editing direction vector based on the target image vector and the source image vector.

[0140] Editing the direction vector can characterize the differences between the target image and the source image.

[0141] In some embodiments, the vector difference between the target image vector and the source image vector is calculated to obtain the editing direction vector.

[0142] Subtracting img vector1 from img vector2 can represent the direction between the two image features, i.e., the editing direction vector, which is parallel to the direction between the corresponding text of the two images.

[0143] This embodiment accurately reflects the editing direction of the source image by using the vector difference between the target image vector and the source image vector, thus facilitating the subsequent acquisition of an accurate target image based on the editing direction vector and the source image.

[0144] In other embodiments, both the target image vector and the source image vector can be layer-normalized. Then, the vector difference between the layer-normalized target image vector and the layer-normalized source image vector is calculated to obtain the editing direction vector. That is, the vector difference between the layer-normalized target image vector and the layer-normalized source image vector is used as the editing direction vector.

[0145] Step 505: Based on the edit direction vector and the first initial vector, determine the prediction vector of the second initial vector.

[0146] In some embodiments, the edit direction vector and the first initial vector can be first converted into vectors in the same feature space, and then the edit direction vector and the first initial vector can be summed to obtain the prediction vector of the second initial vector.

[0147] For example, if the first initial vector is a vector in the target feature space, the editing direction vector can be transformed into a vector in the target feature space, so that the source image vector and the target image vector are located in the same feature space, which facilitates operations between the two vectors.

[0148] Specifically, the image editing model also includes a feature space transformation network 606, which can be denoted as the W network. The electronic device can input the editing direction vector into the feature space transformation network 606 to obtain the target editing direction vector W* located in the target feature space. Then, the target editing direction vector W* and the first initial vector W1 located in the target feature space are summed to obtain the prediction vector of the second initial vector. The prediction vector of the second initial vector = W1 + W*.

[0149] When the target feature space is W space, the feature space transformation network 606 can be a W network. A W network can transform the feature space of a vector to W space.

[0150] The structure of a W network can include layer normalization and a 3-layer multilayer perceptron (MLP). The structure of a 3-layer multilayer perceptron can be a fully connected + BN + ReLU stacked 3 times, but it is not limited to this.

[0151] When the edit direction vector is the vector difference between the target image vector and the source image vector, the W network can first process the input edit direction vector through layer normalization, and then process it through a 3-layer multilayer perceptron to obtain the target edit direction vector located in the W space.

[0152] When the editing direction vector is the vector difference between the target image vector and the source image vector after layer normalization, the vector difference between the target image vector and the source image vector after layer normalization can be input into the 3-layer MLP in the W network for processing.

[0153] This embodiment also trains a feature space transformation network, which can transform the editing direction vector into a vector located in the target feature space, so that the editing direction vector is transformed into a vector located in the same target feature space as the first initial vector, thereby facilitating the operation between the editing direction vector and the first initial vector.

[0154] Step 506: Calculate the total loss value based on the predicted vector and the second initial vector.

[0155] For example, the second initial vector and the predicted vector of the second initial vector can be input into a preset loss function, such as MSEloss, to calculate the second loss value. Then, the sum of the first loss value, the second loss value and the third loss value mentioned above can be calculated to obtain the total loss value.

[0156] Step 507: If the total loss value meets the preset threshold condition, the image editing model training is considered complete.

[0157] That is, the preset threshold condition can be that the total loss value converges. If the total loss value converges, it can be determined that the image editing model training is complete; if the total loss value does not converge, the next round of training will be carried out.

[0158] In this embodiment, the image editing model does not require text describing the image editing direction during training. Instead, it subtracts the target image vector from the source image vector to obtain the image editing direction vector. Furthermore, the image encoders of the two images used in the training process belong to the same feature space as the text encoder of the image editing model. This allows the difference between the text vectors obtained through the text encoder to also reflect the image editing direction. Thus, editing can be performed based on the editing direction of the image generator and the original image. This enables the trained image editing model to be no longer limited to a single editing scenario, achieving image editing for multiple editing scenarios, while eliminating the need for iterative optimization during image editing, thereby improving image editing efficiency and reducing image editing time.

[0159] Based on the same ideas as the image editing method and image editing model training method in the above embodiments, this application also provides an image editing device and an image editing model training device. The image editing device can be used to execute the above image editing method, and the image editing model training device can be used to execute the above image editing model training method.

[0160] For ease of explanation, the schematic diagrams of the two devices provided in this application only show the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structures do not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0161] like Figure 7 As shown, Figure 7 This is a schematic diagram of the structure of an image editing device provided in an embodiment of the present application. The image editing device includes an acquisition module 701, an encoding module 702, a calculation module 703, an inversion module 704, and a generation module 705.

[0162] The acquisition module 701 is used to acquire a source image, source text, and target text to be processed, wherein the source text is the descriptive text corresponding to the source image, the target text is the descriptive text corresponding to the target image, and the target image is an image obtained by editing the source image;

[0163] The encoding module 702 is used to encode the source text and the target text using a pre-trained image editing model to obtain the source text vector corresponding to the source text and the target text vector corresponding to the target text;

[0164] Calculation module 703 is used to calculate the editing direction vector based on the target text vector and the source text vector;

[0165] Inversion module 704 is used to input the source image into the image editing model to obtain the source image vector corresponding to the source image;

[0166] The generation module 705 is used to obtain the target image vector based on the editing direction vector and the source image vector, and to process the target image vector through the image editing model to obtain the target image corresponding to the target text.

[0167] In some embodiments, the image editing model includes: a text encoder, wherein the encoding module 702 encodes the source text and the target text using a pre-trained image editing model to obtain a source text vector corresponding to the source text and a target text vector corresponding to the target text, including:

[0168] The source text and target text are encoded by the text encoder to obtain the source text vector and the target text vector; the step of calculating the editing direction vector based on the target text vector and the source text vector includes: calculating the vector difference between the target text vector and the source text vector to obtain the editing direction vector.

[0169] In some embodiments, the image editing model includes: a feature space transformation network, wherein the source image vector is a vector in the target feature space; the calculation module 703 obtains the target image vector based on the editing direction vector and the source image vector, including: inputting the editing direction vector into the feature space transformation network to obtain a target editing direction vector located in the target feature space; and adding the target editing direction vector to the source image vector to obtain the target image vector.

[0170] In some embodiments, the image editing model further includes an inversion network and an image generator. The inversion network is used to generate a vector located in the target feature space based on the image, and the image generator is used to generate an image based on the vector in the target feature space. The inversion module 704 inputs the source image into the image editing model to obtain the source image vector corresponding to the source image, including: inputting the source image into the inversion network in the image editing model to obtain the source image vector, wherein the source image vector is a vector in the target feature space; and processing the target image vector through the image editing model to obtain the target image corresponding to the target text includes: processing the target image vector through the image generator to obtain the target image.

[0171] In some embodiments, the acquisition module 701 acquires the source image, source text, and target text to be processed, including: acquiring the user's image editing instructions through a human-computer interaction interface; determining the source image, source text, and target text to be processed based on the image editing instructions; and after the generation module 705 processes the target image vector through the image editing model to obtain the target image corresponding to the target text, it is further used to: display the target image through the human-computer interaction interface.

[0172] like Figure 8 As shown, Figure 8 This is a schematic diagram of the structure of a training device for an image editing model provided in an embodiment of this application. The training device for the image editing model includes: an acquisition module 801, a generation module 802, an encoding module 803, a calculation module 804, a determination module 805, and a judgment module 806.

[0173] The acquisition module 801 is used to acquire the first initial vector and the second initial vector;

[0174] The generation module 802 is used to process the first initial vector and the second initial vector through the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector.

[0175] The encoding module 803 is used to encode the source image sample and the target image sample to obtain the source image vector corresponding to the source image sample and the target image vector corresponding to the target image sample;

[0176] Calculation module 804 is used to calculate the editing direction vector based on the target image vector and the source image vector;

[0177] The determining module 805 is used to determine the prediction vector of the second initial vector based on the editing direction vector and the first initial vector;

[0178] The judgment module 806 is used to calculate the total loss value based on the prediction vector of the second initial vector and the second initial vector; and to determine that the image editing model training is complete if the total loss value meets a preset threshold condition.

[0179] In some embodiments, the image editing model includes a text encoder; the encoding module 803 encodes the source image sample and the target image sample to obtain the source image vector corresponding to the source image sample and the target image vector corresponding to the target image sample, including: encoding the source image sample and the target image sample using a preset image encoder to obtain the source image vector and the target image vector; the image encoder and the text encoder belong to the same contrastive image-text pre-training model; the step of calculating the editing direction vector based on the target image vector and the source image vector includes: calculating the vector difference between the target image vector and the source image vector to obtain the editing direction vector.

[0180] In some embodiments, the first initial vector and the second initial vector are both vectors in the target feature space, and the image editing model further includes a feature space transformation network;

[0181] The determination module 805 determines the prediction vector of the second initial vector based on the edit direction vector and the first initial vector, including: inputting the edit direction vector into the feature space transformation network to obtain the target edit direction vector located in the target feature space; and adding the target edit direction vector to the first initial vector to obtain the prediction vector of the second initial vector.

[0182] In some embodiments, the image editing model further includes an inversion network and an image generator, wherein the inversion network is used to generate vectors located in a target feature space based on the image, and the image generator is used to generate an image based on the vectors in the target feature space;

[0183] In the generation module 802, the first initial vector and the second initial vector are processed by the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector. This includes: processing the first initial vector and the second initial vector by the image generator to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector.

[0184] After the generation module 802 processes the first initial vector and the second initial vector through the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector, it is further configured to: input the target image sample into the inversion network to obtain the target inversion vector, and / or input the source image sample into the inversion network to obtain the source inversion vector; determine a first loss value based on a first sub-loss value and / or a second sub-loss value, wherein the first sub-loss value is a loss value determined according to the source inversion vector and the first initial vector, and the second sub-loss value is a loss value determined according to the target inversion vector and the second initial vector;

[0185] The judgment module 806 calculates the total loss value based on the predicted vector of the second initial vector and the second initial vector, including: calculating a second loss value based on the difference between the predicted vector of the second initial vector and the second initial vector; and obtaining the total loss value based on the sum of the first loss value and the second loss value.

[0186] In some embodiments, the image generator is an image generator in an adversarial training network, which further includes an image discriminator;

[0187] After the generation module 802 processes the first initial vector and the second initial vector through the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector, it is further configured to: input the source image sample and / or the target image sample into the image discriminator to obtain the first discriminant output value corresponding to the source image sample and / or the second discriminant output value corresponding to the target image sample;

[0188] Based on the first discrimination output value and / or the second discrimination output value, a third loss value is obtained;

[0189] The judgment module 806 is further configured to obtain a total loss value based on the sum of the first loss value and the second loss value, including: obtaining a total loss value based on the sum of the first loss value, the second loss value and the third loss value.

[0190] It is understood that, in some embodiments, the above-mentioned module may be programmable software instructions stored in memory and executable by the processor. In other embodiments, the above-mentioned module may also be program instructions or firmware embedded in the processor.

[0191] Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application.

[0192] The electronic device 200 includes a memory 20, a processor 30, and a computer program 40 stored in the memory 20 and executable on the processor 30. When the processor 30 executes the computer program 40, it implements the steps described in the above-described image editing method embodiments, for example... Figure 3 Steps 301 to 306 shown, or steps implemented by processor 30 when executing computer program 40 to train the image editing model as described above, for example... Figure 5 Steps 501 to 507 are shown.

[0193] For example, computer program 40 can also be divided into one or more modules / units, which are stored in memory 20 and executed by processor 30. The one or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 40 in electronic device 200. For example, it can be divided into... Figure 7 The acquisition module 701, encoding module 702, calculation module 703, inversion module 704, and generation module 705 shown can also be divided into, for example... Figure 8 The module shown includes an acquisition module 801, a generation module 802, an encoding module 803, a calculation module 804, a determination module 805, and a judgment module 806.

[0194] Those skilled in the art will understand that the schematic diagram is merely an example of the electronic device 200 and does not constitute a limitation on the electronic device 200. It may include more or fewer components than shown in the diagram, or combine certain components, or different components. For example, the electronic device 200 may also include input / output devices, network access devices, buses, etc.

[0195] Processor 30 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, a single-chip microcomputer, or any conventional processor.

[0196] The memory 20 can be used to store computer programs 40 and / or modules / units. The processor 30 implements various functions of the electronic device 200 by running or executing the computer programs and / or modules / units stored in the memory 20 and by calling data stored in the memory 20. The memory 20 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device 200 (such as audio data), etc. In addition, the memory 20 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0197] If the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0198] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the electronic device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and other division methods may be used in actual implementation.

[0199] Furthermore, the functional units in the various embodiments of this application can be integrated into the same processing unit, or each unit can exist physically separately, or two or more units can be integrated into the same unit. The integrated units described above can be implemented in hardware or in the form of hardware plus software functional modules.

[0200] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and not restrictive in all respects. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or electronic devices recited in the electronic device claims may also be implemented by the same unit or electronic device through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any particular order.

[0201] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application and are not intended to limit it. Although this application has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this application without departing from the spirit and scope of the technical solutions of this application.

Claims

1. An image editing method, characterized in that, The method includes: Obtain the source image, source text, and target text to be processed. The source text is the descriptive text corresponding to the source image, the target text is the descriptive text corresponding to the target image, and the target image is the image obtained by editing the source image. The source text and target text are encoded using a pre-trained image editing model to obtain the source text vector corresponding to the source text and the target text vector corresponding to the target text. Based on the target text vector and the source text vector, an editing direction vector is calculated, which is used to characterize the difference between the target text and the source text; The source image is input into the image editing model to obtain the source image vector corresponding to the source image; The target image vector is obtained based on the editing direction vector and the source image vector; The target image vector is processed by the image editing model to obtain the target image corresponding to the target text; The step of calculating the editing direction vector based on the target text vector and the source text vector includes: The editing direction vector is obtained by calculating the vector difference between the target text vector and the source text vector.

2. The image editing method as described in claim 1, characterized in that, The image editing model includes a text encoder, which encodes the source text and target text using a pre-trained image editing model to obtain a source text vector corresponding to the source text and a target text vector corresponding to the target text, including: The source text and target text are encoded by the text encoder to obtain the source text vector and the target text vector.

3. The image editing method as described in claim 1, characterized in that, The image editing model includes: a feature space transformation network, wherein the source image vector is a vector in the target feature space; The process of obtaining the target image vector based on the editing direction vector and the source image vector includes: The edit direction vector is input into the feature space transformation network to obtain the target edit direction vector located in the target feature space; The target editing direction vector is added to the source image vector to obtain the target image vector.

4. The image editing method as described in claim 3, characterized in that, The image editing model further includes an inversion network and an image generator. The inversion network is used to generate vectors located in the target feature space based on the image, and the image generator is used to generate an image based on the vectors in the target feature space. The step of inputting the source image into the image editing model to obtain the source image vector corresponding to the source image includes: The source image is input into the inversion network in the image editing model to obtain the source image vector, which is a vector in the target feature space; The step of processing the target image vector through the image editing model to obtain the target image corresponding to the target text includes: The target image is obtained by processing the target image vector through the image generator.

5. The image editing method according to any one of claims 1 to 4, characterized in that, The process of acquiring the source image, source text, and target text to be processed includes: Obtain the user's image editing instructions through the human-computer interaction interface; Based on the image editing instructions, the source image, source text, and target text to be processed are determined; After processing the target image vector through the image editing model to obtain the target image corresponding to the target text, the method further includes: The target image is displayed through a human-computer interaction interface.

6. A training method for an image editing model, characterized in that, The training method for the image editing model includes: Obtain the first and second initial vectors; The first initial vector and the second initial vector are processed by the current image editing model to obtain the source image sample corresponding to the first initial vector and the target image sample corresponding to the second initial vector. The source image sample and the target image sample are encoded to obtain the source image vector corresponding to the source image sample and the target image vector corresponding to the target image sample; Based on the target image vector and the source image vector, an editing direction vector is calculated, which is used to characterize the difference between the target image sample and the source image sample; Based on the edit direction vector and the first initial vector, determine the prediction vector of the second initial vector; The total loss value is calculated based on the predicted vector and the second initial vector; If the total loss value meets the preset threshold condition, it is determined that the image editing model training is complete, and the pre-trained image editing model as described in any one of claims 1 to 5 is obtained; The step of calculating the editing direction vector based on the target image vector and the source image vector includes: The editing direction vector is obtained by calculating the vector difference between the target image vector and the source image vector.

7. The training method for the image editing model as described in claim 6, characterized in that, When the image editing model includes a text encoder; encoding the source image sample and the target image sample to obtain the source image vector corresponding to the source image sample and the target image vector corresponding to the target image sample includes: The source image sample and the target image sample are encoded by a preset image encoder to obtain the source image vector and the target image vector; the image encoder and the text encoder belong to the same contrastive image-text pre-training model.

8. An image editing device, characterized in that, The device includes: The acquisition module is used to acquire the source image, source text, and target text to be processed. The source text is the descriptive text corresponding to the source image, the target text is the descriptive text corresponding to the target image, and the target image is an image obtained by editing the source image. The encoding module is used to encode the source text and the target text using a pre-trained image editing model to obtain the source text vector corresponding to the source text and the target text vector corresponding to the target text; The calculation module is used to calculate an editing direction vector based on the target text vector and the source text vector, wherein the editing direction vector is used to characterize the difference between the target text and the source text; The inversion module is used to input the source image into the image editing model to obtain the source image vector corresponding to the source image; The generation module is used to obtain the target image vector based on the editing direction vector and the source image vector, and to process the target image vector through the image editing model to obtain the target image corresponding to the target text; The step of calculating the editing direction vector based on the target text vector and the source text vector includes: The editing direction vector is obtained by calculating the vector difference between the target text vector and the source text vector.

9. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the image editing method as described in any one of claims 1 to 5, or the image editing model training method as described in any one of claims 6 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform an image editing method as described in any one of claims 1 to 5, or a training method for an image editing model as described in any one of claims 6 to 7.

Images