Image editing methods, image editing devices, computer equipment, and computer programs

Abstract

This application provides an image editing method, apparatus, device, and storage medium in the field of artificial intelligence. The method includes the steps of: obtaining an input image and an editing command for the input image; extracting a first image feature from the input image that includes a plurality of feature items; extracting an instruction text feature from the editing command that includes a plurality of feature items; obtaining a fused feature by fusing a target image feature and the instruction text feature, wherein the fused feature represents a description of executing the editing command on the input image, and the target image feature includes the first image feature; determining a first object and an editing operation on the first object from the input image based on the fused feature; generating an edited image by executing the editing operation on the first object; and obtaining a target image by merging the edited image and the input image.

Description

Technical Field

[0001] This application claims priority based on a Chinese patent application filed with the China National Intellectual Property Administration on September 15, 2023, with an application number of No. 2023112083362 and an invention title of "Image Editing Method, Apparatus, Device, and Storage Medium", and all of its content is incorporated herein by reference.

[0002] This application relates to the field of artificial intelligence and provides an image editing method, apparatus, device, and storage medium.

Background Art

[0003] In recent years, with the rapid development of computer network technology, artificial intelligence (AI) technology has been widely applied in the field of image processing, and in particular, the application of AI-based image editing models has become increasingly popular.

[0004] For example, an image editing model based on Stable Diffusion (SD) can provide image editing services. However, since the text encoder incorporated in SD has low text understanding and inference capabilities, the image editing model cannot understand complex editing instructions and, as a result, cannot generate an edited image that matches the content of the instructions.

Summary of the Invention

Problems to be Solved by the Invention

[0005] Embodiments of this application provide an image editing method, apparatus, device, and storage medium.

Means for Solving the Problems

[0006] According to a first aspect, in embodiments of this application, an image editing method is provided. This method includes: obtaining an input image and an editing instruction for the input image; The steps include: extracting a first image feature containing multiple feature items from the input image; The steps include: extracting command text features containing multiple feature items from the aforementioned editing command; A step of obtaining a fused feature by fusing a target image feature and the instruction text feature, wherein the fused feature represents a description of executing the editing instruction on the input image, and the target image feature includes the first image feature, The steps include determining a first object and an editing operation for the first object from the input image based on the aforementioned fusion features, The steps include generating an edited image by performing the editing operation on the first object, The method includes the step of obtaining a target image by merging the edited image and the input image.

[0007] According to a second embodiment of the present application, an image editing device is further provided. This device is A feature extraction unit that acquires an input image and an editing command for the input image, extracts a first image feature containing multiple feature items from the input image, and extracts an instruction text feature containing multiple feature items from the editing command, A feature mining unit that obtains a fused feature by fusing a target image feature and the instruction text feature, wherein the fused feature represents a description for executing the editing instruction on the input image, and the target image feature includes the aforementioned first image feature, The system includes an image editing unit that, based on the aforementioned fusion features, determines a first object and an editing operation for the first object from the input image, generates an edited image by performing the editing operation on the first object, and obtains a target image by merging the edited image with the input image.

[0008] According to a third aspect, an embodiment of the present invention further provides a computer device comprising a processor and memory. The memory stores program code, and when the program code is executed by the processor, the processor is caused to perform any one of the steps of the image editing method described above.

[0009] According to a fourth aspect, embodiments of the present application further provide a computer-readable storage medium containing program code. When the program product is executed on a computer device, the program code is used to cause the computer device to perform any one of the steps of the image editing method described above.

[0010] According to a fifth embodiment of the present invention, a computer program product including computer instructions is further provided, which, when executed by a processor, causes any one of the steps of the above-described image editing method to be performed.

[0011] The drawings described herein are provided to further understand the present application and constitute part of the present application. The schematic embodiments and descriptions thereof are for interpretive purposes and do not constitute an inappropriate limitation of the present application. [Brief explanation of the drawing]

[0012] [Figure 1A] This is a schematic diagram of the configuration of the image editing model provided in the embodiment of the present application. [Figure 1B] This is an optional schematic diagram of one application scenario in an embodiment of the present application. [Figure 1C] This is a schematic diagram showing a comparison of images before and after editing, as provided in the embodiment of the present application. [Figure 2A] This is a schematic diagram of the flow for training the image editing model provided in the embodiment of the present invention. [Figure 2B] This is a schematic diagram of the logic for training the image editing model provided in the embodiment of the present invention. [Figure 2C]It is a schematic diagram of a logic for generating a predetermined target image corresponding to a training sample image of an editing function provided in an embodiment of the present application. [Figure 2D] It is a schematic diagram of a logic for a predetermined target image corresponding to a training sample image provided in an embodiment of the present application. [Figure 3A] It is a schematic diagram of a flow for performing an image editing operation on an input image provided in an embodiment of the present application. [Figure 3B] It is a schematic diagram of a logic for performing an image editing operation on an input image provided in an embodiment of the present application. [Figure 3C] It is a schematic diagram of a flow for obtaining a fusion feature provided in an embodiment of the present application. [Figure 3D] It is a schematic diagram of a logic for obtaining a fusion feature provided in an embodiment of the present application. [Figure 3E] It is a schematic diagram of a flow for generating a drawing area provided in an embodiment of the present application. [Figure 3F] It is a schematic diagram of a logic for generating a drawing area provided in an embodiment of the present application. [Figure 3G] It is a schematic diagram of a logic for generating a drawing area provided in an embodiment of the present application. [Figure 3H] It is a schematic diagram of a logic for drawing a blue drink provided in an embodiment of the present application. [Figure 3I] It is a schematic diagram of a logic for drawing one cat provided in an embodiment of the present application. [Figure 3J] It is a schematic diagram of a logic for obtaining a target image by an image fusion method 1 provided in an embodiment of the present application. [Figure 3K] It is a schematic diagram of a logic for obtaining a target image by an image fusion method 2 provided in an embodiment of the present application. [Figure 3L] It is a schematic diagram of a logic for obtaining a target image by an image fusion method 3 provided in an embodiment of the present application. [Figure 4A] It is a schematic diagram of a flow for performing image editing of a first object according to a second object in a reference image provided in an embodiment of the present application. [Figure 4B]This is a schematic diagram of the logic for performing image editing on a first object in accordance with a second object in a reference image provided in an embodiment of the present application. [Figure 4C] This is a schematic diagram of the logic for drawing glasses in the reference image provided in the embodiment of the present application. [Figure 5] This is a comparative diagram of the related technologies provided in the embodiments of the present application and the editing effects of the present application. [Figure 6A] This is an image editing model for the landing scenario provided in the embodiment of the present invention. [Figure 6B] This figure shows how to generate a target image of a girl wearing glasses by editing the image provided in the embodiment of the present invention. [Figure 6C] This is a schematic diagram of the logic for obtaining a target image by editing the input image provided in the embodiment of the present invention. [Figure 6D] This is a schematic diagram of the interface for batch uploading attached files provided in the embodiment of the present invention. [Figure 6E] This is a schematic diagram of the interface for batch uploading the editable packages provided in the embodiment of the present invention. [Figure 7A] This is an image editing model for the landing scenario provided in the embodiment of the present invention. [Figure 7B] This is a target image in which the cat in the grass provided in the embodiment of the present invention has been replaced with a dog in the reference image. [Figure 7C] This is a schematic diagram of the logic for obtaining a target image by editing the input image provided in the embodiment of the present invention. [Figure 8] This is a schematic diagram of the configuration of the image editing device provided in the embodiment of the present application. [Figure 9] This is a schematic diagram of the hardware configuration of a computer device to which the embodiment of the present invention is applied. [Figure 10] This is a schematic diagram of the hardware configuration of another computer device to which the embodiments of the present invention apply. [Modes for carrying out the invention]

[0013] To further clarify the purpose, structure, and advantages of the embodiments of this application, the present invention will be described clearly and completely below with reference to the drawings of the embodiments. As will be obvious, the embodiments described are some, but not all, embodiments of the present invention. All other embodiments that a person skilled in the art can derive from the embodiments described herein without creative work are within the scope of the present invention.

[0014] To facilitate understanding for those skilled in the art, some of the terms used in the embodiments of this application are interpreted and explained below.

[0015] Large-scale language models (LLMs) are deep learning models trained using unsupervised or semi-supervised learning on large amounts of text data. LLMs can automatically learn language patterns in the field of natural language processing, generating natural language text and understanding the meaning of language text. LLMs can handle various types of natural language tasks, such as text classification, question-and-answer, and conversation, and represent an important approach to artificial intelligence.

[0016] Stable Diffusion (SD) is a text-to-image diffusion model capable of generating high-quality images. These models gradually reconstruct a target image from a noisy image under the given condition of a text description. Stable Diffusion is an open-source text-to-image model.

[0017] Q-Former (officially known as Querying Transformer) is a lightweight Transformer structure specifically designed for visual and linguistic integration. It achieves efficient visual feature extraction and linguistic representation learning by introducing a learnable query vector set between a frozen visual model and a large-scale linguistic model. Q-Former consists of two Transformer submodules: an image transformer and a text transformer. These two submodules share the same self-attention layer to enable efficient computation and information sharing.

[0018] Image Transmitter: Interacts with a frozen image encoder and extracts visual features via a learnable set of query vectors. These query vectors interact not only with each other but also with the output of the image encoder via a cross-attention layer to extract the visual representations most relevant to the text.

[0019] Text Converter: Can be used as both a text encoder and a text decoder. In the representation learning phase, it primarily functions as a text encoder, sharing an image converter and a self-attention layer. In the generation learning phase, it functions as a text decoder, responsible for generating text that matches the visual representation.

[0020] The following briefly introduces the design concept of the embodiment of this application.

[0021] In some embodiments, an image editing model based on SD can provide image editing services. However, the text encoder incorporated into SD has poor text comprehension and reasoning capabilities, preventing the image editing model from understanding complex editing instructions and consequently failing to generate a target image that matches the content of the instructions.

[0022] To improve the model's text comprehension and reasoning capabilities, LLM (Limited Language Modeling) is introduced to understand the content of editing instructions. The image editing model processes the input image and editing instructions using LLM to obtain one-dimensional text features that indicate the editing operation to be performed on the image. The model also calls an image editing plugin to perform the corresponding operation. The image editing plugin then draws the target image based on the one-dimensional text features.

[0023] While image editing models in related technologies can offer multiple image editing functions, each function is implemented by a different image editing plugin. Because the underlying frameworks of each image editing plugin are not common to one another, the model can only call one image editing plugin at a time to handle a task.

[0024] If an input image contains two or more objects of the same type, and the editing command performs image editing operations on only some of those objects, the image editing model may mistakenly recognize all objects belonging to that type in the image as the first object and perform local editing operations on those objects as well. As a result, the generated target image and the editing command may not match.

[0025] Therefore, in order to solve the above problem, the present invention proposes a new image editing method. This method includes the steps of: obtaining an input image and an editing command for the input image; extracting a first image feature containing a plurality of feature items from the input image; extracting an instruction text feature containing a plurality of feature items from the editing command; obtaining a fused feature by fusing a target image feature and the instruction text feature, wherein the fused feature represents a description for executing the editing command on the input image, and the target image feature includes the first image feature; determining a first object and an editing operation on the first object from the input image based on the fused feature; generating an edited image by executing the editing operation on the first object; and obtaining a target image by merging the edited image and the input image.

[0026] In summary, the embodiment of the present invention can improve the ability to understand instructions by taking the steps of extracting instruction text features containing multiple feature items from the editing instruction, and obtaining fused features by fusing target image features with the instruction text features. In this way, even if the input image contains multiple objects of the same type, it is possible to accurately determine which object is the first object and which other objects do not require image editing operations, thereby avoiding the recognition of all objects belonging to that type in the figure as the first object, reducing the false detection rate of objects, and further improving the accuracy of image editing.

[0027] Preferred embodiments of the present application will be described below with reference to the drawings of the specification. It should be understood that the preferred embodiments described herein are for the purpose of describing and interpreting the present application only and are not intended to limit the present application. Furthermore, where there is no contradiction, the embodiments and features described herein can be combined with each other.

[0028] The image editing method provided in the embodiment of the present invention is implemented by an image editing model, which is designed as an end-to-end integrated structure. As shown in Figure 1A, the image editing model includes an instruction understanding module and an image editing module.

[0029] Here, the instruction understanding module understands the content of the editing instructions and obtains fused features by incorporating the understood instruction text features into the image features of the input image. The fused features can help the image editing module determine a first object and the editing operations on the first object from the input image, and obtain the drawing area of ​​the first object from the image to be drawn. In this application, the instruction understanding module may be an LLM or other large-scale language model, but is not limited thereto.

[0030] The image editing module obtains an edited image by drawing a second object acquired based on an editing command within the drawing area of ​​the image to be drawn, and then obtains a target image by fusing the edited image with the input image.

[0031] If the image size is too large or the number of elements in the image is too large, additional image encoding and image selection modules are added to reduce the computational load on the image editing model. The former converts the pixel-based image into data that the model can recognize, while the latter selects the necessary image in response to an image selection command.

[0032] Furthermore, to further enhance image editing effects, a dimension transformation module is added to the model that converts the spatial dimensions of fused features to spatial dimensions compatible with the image editing module.

[0033] The image editing model can be applied to various application scenarios where image editing is needed, such as in the image processing field, browsers, and social platforms. When using related products based on the image editing model, users input editing commands in natural language and an input image, and the image editing model is triggered to perform image editing operations on the input image, combined with icon preservation functionality, and output a corresponding target image. Because editing commands based on natural language are more in line with users' habits than obscure commands, the learning barrier and difficulty of using the product are reduced, providing users with a convenient and fast method of image editing.

[0034] Figure 1B shows one of the application scenarios. The application scenario includes two terminal devices 110 and one server 130, and establishes a communication connection between each terminal device 110 and the server 130 using either a wired or wireless network.

[0035] The image editing model may be deployed on the terminal device 110 as a machine learning model and provide image editing functionality to the user via a local call method, or it may be deployed within the server 130, within another independent server in the network, within a server cluster in the network, or within a distributed system in the network and provide image editing functionality to the user via a network call method.

[0036] Here, terminal device 110 includes, but is not limited to, mobile phones, computers (e.g., tablet computers, laptop computers, desktop computers, etc.), smart home appliances, smart voice interaction devices (e.g., smartwatches, smart speakers, etc.), in-vehicle terminals, aircraft, etc.

[0037] Server 130 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and base cloud computing services such as big data and artificial intelligence platforms. The present invention is not limited thereto.

[0038] Terminal device 110 receives the editing command "Change the cat in the image to a dog" entered by the user via the image editing interface 120, along with the input image, and transmits the above data to server 130 via a pre-established communication link. Server 130 calls an image editing model and, in response to the editing command, performs an image editing operation on the input image to obtain a target image as shown in Figure 1C.

[0039] During the model training phase, the image editing model is trained based on each training sample image. Here, editing commands include those that perform global editing operations on the image and those that perform local editing operations on the image. Thanks to LLM's strong understanding of commands and input images, its reasoning ability, knowledge learned from large amounts of data, and its ability to fuse multiple types of information, it can generalize different functions and accurately edit specified objects, saving the training cost of the model.

[0040] Referring to the schematic diagrams shown in Figures 2A and 2B, the image editing model is trained by performing the following operations.

[0041] In step S201, an initial trained image editing model is obtained by repeatedly training an untrained image editing model based on each training sample image for training the editing function, until the iterative training stops.

[0042] Before formal training, it is necessary to pre-generate a predetermined target image for each training sample. Assuming the editing command is "add one cat to the chair," the predetermined target image shown in Figure 2C is obtained by drawing one cat on the training sample image.

[0043] Here, each round iteration is: The steps include obtaining each target image by performing an editing operation on the corresponding training sample image based on at least one sample editing instruction, The process includes the steps of obtaining a predetermined target image for each of the corresponding training sample images, obtaining an image editing loss value generated in this round of iterations based on the target image and the corresponding predetermined target image, and sequentially optimizing the parameters of each module in the model based on the image editing loss value.

[0044] Here, if the parameters of the instruction understanding module and the image editing module in the image editing model are optimized separately, the instruction understanding module cannot optimize parameter adjustments based on the abundant image monitoring signals of the image editing module during the parameter adjustment process, and the image editing module cannot optimize the parameter adjustment effect based on the text monitoring signals of the instruction understanding module during the parameter adjustment process.

[0045] To solve this problem, the present invention employs an integrated optimization method. First, based on the image editing loss value generated in the current iteration, the parameters of the image editing module are optimized. The image monitoring signal generated by the image editing module and the image editing loss value are then sent back to the dimension transformation module in a single batch, where the parameters of the dimension transformation module are optimized. The image monitoring signal, the image editing loss value, and the corresponding monitoring signal generated by the dimension transformation module are then sent back to the instruction understanding module in a single batch, where the parameters of the instruction understanding module are optimized. By analogy, the parameter optimization of the image selection module and the image encoding module is completed.

[0046] The iteration termination conditions include: (1) the difference between the image editing loss value generated in the current iteration and the image editing loss value generated in the previous iteration does not exceed a predetermined threshold; (2) the image editing loss value generated in the current iteration does not exceed a predetermined threshold; and (3) the number of rounds in the current iteration reaches a predetermined number of rounds threshold.

[0047] If any one of the above iteration stopping conditions is met, it indicates that the internal parameters of the model are stabilizing after multiple rounds of iterations. In this case, the model after parameter adjustments in the round that satisfies the iteration stopping condition is considered the first-trained image editing model. If none of the iteration stopping conditions are met, the next batch of training sample images is read, and model training continues.

[0048] In step S202, the image editing model, which has been trained twice, is obtained and output by repeatedly training the image editing model, which has been trained once, based on each training sample image for training the object detection function, until the iterative training stops.

[0049] Before formal training, it is necessary to pre-generate a predetermined target image for each training sample. Assuming the editing command is "detect a sparrow located on the left side of the image," the predetermined target image, as shown in Figure 2D, is obtained by drawing a detection box on the training sample image.

[0050] Here, each round iteration is: The steps include obtaining each target image by performing a detection operation on the corresponding training sample image based on at least one editing instruction, The process includes the steps of obtaining a predetermined target image for each of the corresponding training sample images, obtaining the loss value generated in this round of iterations based on the target image and the corresponding predetermined target image, and sequentially performing parameter tuning optimization for each functional module in the model based on the loss value.

[0051] Referring to the schematic diagrams shown in Figures 3A and 3B, the target image is obtained by performing the following operations.

[0052] In step S301, an input image and an editing command for the input image are obtained.

[0053] In step S302, a first image feature containing multiple feature items is extracted from the input image. For example, an image coding model may extract a first image feature representing a visual feature from the input image. Here, each feature item corresponds to at least one pixel point of the input image. The first image feature is, for example, a feature vector containing multiple feature items.

[0054] In step S303, an instruction text feature containing multiple feature items is extracted from the editing instruction. The instruction text feature is, for example, a one-dimensional text vector.

[0055] This invention supports the following two input image acquisition methods. Here, Method 1: Obtain the original image entered by the user, and use the original image as the input image.

[0056] Method 2: In response to an image selection command, the input image is acquired by selecting an original image entered by the user.

[0057] In the second input image acquisition method, an image coding model is used to characterize the original images. The characterization of each original image and the image selection command are input together into the image selection module. The original image characterization is matched with the command text characterization of the image selection command. The image content corresponding to the original image characterization that has a high correlation with the command text characterization is then acquired to obtain the input image.

[0058] The model's instruction understanding module extracts features from the input image to obtain first-order image features, converting the pixel-based image into data that the instruction understanding module can recognize. Furthermore, the instruction understanding module extracts features from editing instructions to obtain instruction text features, converting the editing instructions, which exist in text or string format, into data that the model can recognize.

[0059] In step S304, a fused feature is obtained by fusing the target image feature and the instruction text feature, the fused feature represents a description of executing the editing instruction on the input image, and the target image feature includes the first image feature.

[0060] In some embodiments, a first degree of relevance is obtained between each feature item in the target image features and each feature item in the instruction text features, and then the fused features are obtained by fusing the target image features and the instruction text features using the first degree of relevance.

[0061] In some embodiments, the fused features may be obtained using an attention mechanism.

[0062] For example, referring to the schematic diagrams shown in Figures 3C-3D, the process of fusing the first image feature and the instruction text feature includes the following steps.

[0063] In step S3041, the first image feature and the linear transformation matrix are used.

number

number

number

number

number

number

[0064] In step S3042, the attention weights of the instruction text features for each target image feature are obtained by multiplying the query vector matrix Q by the key vector matrix K. Here, the attention weights can represent the degree of association between the feature items in the first image feature and the feature items in the instruction text feature.

[0065] In step S3043, the fused features are obtained by multiplying the value vector matrix and the attention weight matrix.

[0066] In step S305, based on the fusion features, a first object and an editing operation for the first object are determined from the input image.

[0067] In step S306, the edited image is generated by performing the editing operation on the first object.

[0068] In step S307, the target image is obtained by merging the edited image and the input image.

[0069] In summary, the embodiment of the present invention can improve the ability to understand instructions by taking the steps of extracting instruction text features containing multiple feature items from the editing instruction, and obtaining fused features by fusing target image features with the instruction text features. In this way, even if the input image contains multiple objects of the same type, it is possible to accurately determine which object is the first object and which other objects do not require image editing operations, thereby avoiding the recognition of all objects belonging to that type in the figure as the first object, reducing the false detection rate of objects, and further improving the accuracy of image editing.

[0070] In some embodiments, a dimensional transformation module is added to the model to further improve image editing effects. This module obtains fused features located in a second feature space by dimensionally transforming the fused features located in a first feature space. The spatial dimensions of the transformed fused features are more compatible with the image editing module. Here, the spatial dimension of the first feature space is lower than that of the second feature space.

[0071] In some embodiments, the image editing module uses an attention mechanism to obtain a second degree of relevance between each feature item in the first image features of the input image and the fused feature, and determines the first object and the corresponding editing operation from the input image based on the feature items with higher relevance.

[0072] In some embodiments, step S305 includes the steps of: determining a second degree of relevance between each feature item in the first image feature and the fused feature; obtaining a first feature group by determining, based on the second degree of relevance, feature items in the first image feature whose degree of relevance with the fused feature reaches a first threshold; and determining from the input image a region composed of pixel points corresponding to the first feature group, and determining the first object corresponding to the region. The first threshold is a predetermined value, for example, 0.8.

[0073] In some embodiments, step S306 includes: determining a drawing area corresponding to the first object from a drawing target image having the same size as the input image; obtaining the edited image by modifying the attributes of the first object within the drawing area based on the editing operation, if the editing operation is an operation to modify the attributes of the first object; and obtaining the edited image by selecting a second object from a predetermined object library and drawing the second object within the drawing area, if the editing operation is an operation to replace the first object with a second object and no reference image is specified for generating the second object.

[0074] In some embodiments, step S306 includes, if the editing operation is an operation to replace a first object with a second object and a reference image for generating the second object is specified, obtaining an edited image by determining the image features of the second object based on fusion features and generating an image of the second object within a drawing area corresponding to the first object based on the image features of the second object.

[0075] In some embodiments, the degree of association between each feature item in the drawing image features of the image to be drawn and the fused feature is obtained, and based on the feature item with the higher degree of association, a drawing region corresponding to the first object is obtained from the image to be drawn, which has the same size as the input image. The implementation of obtaining the degree of association between features is similar to the step shown in S3042 in Figure 3C.

[0076] In some embodiments, the step of obtaining a drawing region corresponding to the first object from a drawing target image having the same size as the input image includes the steps of obtaining drawing image features that include multiple feature items by performing feature extraction on the drawing target image having the same size as the input image; obtaining a third degree of relevance between each feature item in each drawing image feature and the fused feature; and obtaining the drawing region by determining a region composed of pixel points corresponding to feature items whose third degree of relevance is equal to or greater than a first threshold.

[0077] Referring to the schematic diagrams shown in Figures 3E to 3F, the specific process for obtaining the drawing area corresponding to the first object includes the following steps.

[0078] In step S3061, the corresponding drawing image features are obtained by extracting features from the drawing target image, which has the same size as the input image.

[0079] In step S3062, the third degree of relevance between each feature item in the drawn image features and the fused feature is obtained, and the drawing region is obtained by determining the region composed of pixel points corresponding to feature items whose third degree of relevance is equal to or greater than the first threshold.

[0080] Here, the specific implementation process of step 3062 is as shown in Figure 3G.

[0081] First, at least one group of drawn image features is obtained by clustering feature items whose relevance is equal to or greater than the first threshold.

[0082] Next, for each of the at least one group of drawing image features, Obtain a pixel region that contains at least one pixel point corresponding to each feature item in a single drawn image feature group, Based on the acquired pixel regions, the boundary points of the drawing regions are determined, and the drawing regions are acquired from the image to be drawn by connecting these boundary points.

[0083] In the embodiments of this application, the image to be drawn is a noisy image. However, due to the constraints of the drafting requirements and in order to clearly show the pixel points and drawing area, the image to be drawn in Figures 3B, 3H, and 3I is a white background image. These drawings are merely simple schematic diagrams to illustrate the corresponding steps.

[0084] If no input image is provided to the image editing module, the module will not edit based on the input image, but will instead generate a completely new target image based entirely on the one-dimensional text features output from the LLM. As a result, it becomes difficult to maintain local structural and local texture consistency between objects other than the first object in the target image and objects other than the first object in the input image.

[0085] To solve this problem, in this application, the input image, the noisy target image, and the fused features are input to the image editing module all at once. An edited image is obtained by drawing a second object acquired based on the editing operation of the first object within the target image, and the target image is obtained by merging the edited image with the input image. As a result, other unedited objects in the target image are from the input image, and the local structure and local texture of the images before and after editing match.

[0086] In this application, image modification tasks can be further classified into two categories. One is to directly edit the first object itself, such as attribute editing operations like adjusting the color temperature of an image, adjusting the contrast of an image, or adjusting the color of an object. The other is to edit the first object based on a pre-built object library, such as adding a kitten to an image, putting sunglasses on a person in an image, or modifying the drawing style of an input image.

[0087] An edited image is generated by performing the editing operation on the first object.

[0088] If the editing operation is an operation to modify the attributes of the first object, the target image is obtained by modifying the attributes of the first object within the drawing area based on the editing operation.

[0089] If the editing operation is an operation to replace the first object with the second object, and no reference image is specified for generating the second object, the target image is obtained by selecting the second object from a predetermined object library and drawing the second object within the drawing area.

[0090] In some embodiments, if the editing operation is an operation to replace a first object with a second object, and a reference image for generating the second object is specified, the edited image is obtained by determining the image features of the second object based on fusion features, and generating an image of the second object within the drawing area corresponding to the first object based on the image features of the second object.

[0091] In some embodiments, if the editing operation is an operation to modify the attributes of the first object, to indicate that the editing operation edits the first object itself, the object obtained by performing the editing operation on the first object is designated as the second object, and the second object is drawn within the drawing area of ​​the first object.

[0092] In some embodiments, the editing operation is an operation that replaces a first object with a second object. For example, the editing operation "change the cat in the figure to a dog" is an editing operation on the first object "cat". In the input image before editing, the object at the position of the first object is "cat", and in the image after editing, the object at the position of the original first object is "dog". Another example is the editing operation "change the coffee-colored drink to a blue drink". Since this is a modification operation on the attributes of the first object "drink", the object at the position of the object is similarly "drink" in the images before and after editing.

[0093] For example, the editing operation is "change the coffee-colored drink to a blue drink." The corresponding editing operation modifies the color of the first object, "a cup of coffee-colored drink." Since the modified object is "a cup of blue drink," the edited image shown in Figure 3H is obtained by drawing "a cup of blue drink" as the second object within the corresponding drawing area.

[0094] Note that in Figure 3H, the blue drink is indicated by diagonal lines. Figure 3H is merely a simplified schematic diagram to illustrate this example.

[0095] In some embodiments, if the editing operation is an operation to replace a first object with a second object, and no reference image is specified for generating the second object, the target image is obtained by selecting a second object from a predetermined object library and drawing the second object in the drawing area. For example, the second object is obtained based on each similar object in the predetermined object library related to the type of the second object, and the second object is drawn in the drawing area of ​​the first object.

[0096] Here, the process of obtaining a second object based on each similar object related to the type of the second object in a given object library includes obtaining similar objects related to the type of the second object from the given object library based on the type of the second object, and obtaining the second object by making one of the similar objects the second object, or by fusing multiple similar objects.

[0097] For example, the editing operation is "add one cat to the chair." The corresponding editing operation is to add a new second object to the input image. The instruction information included in this operation is "cat." A set of similar "cat" objects is obtained from a given image library, and one cat object from this set is selected as the second object, or a single second object is obtained by fusing multiple cat objects. Next, the second object "cat" is drawn within the drawing area of ​​the image to be drawn, thereby obtaining the edited image shown in Figure 3I.

[0098] In the embodiments of this application, the following three image merging methods are provided.

[0099] Method 1: The target image is obtained by overlaying the edited image onto the input image.

[0100] For example, in response to an editing operation such as "change the cat in the image to a dog," a single edited image is drawn. The drawing area in the edited image is an opaque layer, while other areas of the image are transparent layers. When the edited image is directly overlaid on the input image, the second object obscures the first object below, but other areas of the edited image do not obscure the input image. This ensures that other unedited objects in the target image shown in Figure 3J match those in the input image, avoiding abnormal situations such as image distortion.

[0101] Method 2: Incorporate other unedited objects from the input image into the edited image.

[0102] The target image is obtained by extracting objects other than at least one first object from the input image and merging at least one other object into the edited image.

[0103] For example, in response to an editing operation such as "change a coffee-colored drink to a blue drink," the system draws an edited image, decomposes the input image into multiple images of other objects, and then merges the multiple images of other objects with the edited image to obtain the target image shown in Figure 3K.

[0104] The input image displays a cup of coffee-colored beverage, while the target image displays a cup of blue beverage. However, due to the constraints of the drawing requirements, in Figure 3K, the coffee-colored beverage is shown in white, and the blue beverage is shown in diagonal lines. Figure 3K is merely a simplified schematic diagram to illustrate this example.

[0105] Method 3: Merge the second object in the edited image into the input image.

[0106] The target image is obtained by merging the target image with the input image from which the first object has been cut out.

[0107] For example, in response to an editing operation such as "change the cat in the image to a dog," an edited image is drawn, the second object "dog" in the image is extracted, and it is merged with the input image from which the first object "cat" has been cut out to obtain a target image as shown in Figure 3L.

[0108] Furthermore, in this application, the edited second object may be obtained by image editing the corresponding first object based on the reference image. Referring to the schematic diagrams shown in Figures 4A to 4B, the process of modifying the input image to the second object in the reference image includes the following steps.

[0109] In step S401, an input image, an editing command for the input image, and a reference image are acquired.

[0110] In step S402, a first image feature containing multiple feature items is extracted from the input image, and a second image feature containing multiple feature items is extracted from the reference image.

[0111] Furthermore, this application supports the following two reference image acquisition methods. Here, Method 1: Obtain the original reference image entered by the user, and use the original reference image as the reference image.

[0112] Method 2: In response to an image selection command, the reference image is acquired by selecting an image from the input original reference image.

[0113] To implement the image selection function, two image encoding modules and two image selection modules are added to the model. Here, Image encoding module 1 and Image selection module 1 acquire the input image by selecting the input image entered by the user. Image encoding module 2 and Image selection module 2 acquire the reference image by selecting another original reference image entered by the user.

[0114] Subsequently, the model's instruction understanding module extracts features from the input image to obtain features for each edited image, and extracts features from the reference image to obtain second image features, converting the pixel-based image into data that the model can recognize. Furthermore, the instruction understanding module extracts features from the edited instructions to obtain instruction text features, converting the edited instructions, which exist in text or string format, into data that the model can recognize.

[0115] In step S403, a fused feature is obtained by fusing the target image feature and the instruction text feature, the fused feature represents a description of executing the editing instruction on the input image based on the reference image, and the target image feature includes the first image feature and the second image feature.

[0116] In some embodiments, a command understanding module of the attention mechanism concept is used to obtain fused features by incorporating command text features into the image features of the input image and reference image based on the degree of relevance between the target image features and command text features.

[0117] The fusion feature can help the image editing module determine a first object and its corresponding editing operation from the input image, obtain a drawing area corresponding to the first object from the image to be drawn, and generate a second object based on the features of a specified reference image by referring to the fusion feature. In this way, the embodiment of the present invention can generate a second object that references a reference image from an input image, thereby improving the accuracy of image editing.

[0118] In step S404, based on the fusion features, a first object and an editing operation for the first object are determined from the input image. Here, since the fusion features can specify features of a reference image, the embodiment of the present invention can accurately execute editing instructions that specify a reference image, and further improves the execution accuracy of complex editing instructions (for example, editing instructions that specify a reference image).

[0119] In step S405, an edited image is generated by performing the editing operation on the first object. In some embodiments, step S405 includes, if the editing operation is an operation to replace the first object with a second object and a reference image for generating the second object is specified, the step of obtaining the edited image by determining the image features of the second object based on fusion features and generating an image of the second object in a drawing area corresponding to the first object based on the image features of the second object. In some embodiments, step S405 includes, determining a drawing area corresponding to the first object from a drawing target image having the same size as the input image, obtaining the edited image by modifying the attributes of the first object in the drawing area based on the editing operation if the editing operation is an operation to modify the attributes of the first object, and obtaining the edited image by selecting a second object from a predetermined object library and drawing the second object in the drawing area if the editing operation is an operation to replace the first object with a second object and a reference image for generating the second object is not specified.

[0120] In step S406, the target image is obtained by merging the edited image and the input image.

[0121] Compared to related technologies, the image editing method provided in this application can achieve local editing that requires higher recognition accuracy, and in particular, when performing local editing operations on an input image containing multiple similar objects, the model's performance is superior to that of image editing models disclosed in related technologies.

[0122] Assuming the editing command is "change the cat in the mirror in the figure to a tiger," the two target images shown in Figure 5 are obtained by editing and drawing the input image in Figure 5 using the related technology and the image editing method provided in this application, respectively. As is clear from the figure, in the target image output by the related technology, both cats in the figure have been changed to tigers, but in the target image output by this application, the background image and the image of the kitten in the real world from the input image remain, and only the cat in the mirror has been changed to a tiger.

[0123] In the image editing model shown in Figure 6A, the image encoder is the image encoding module, the IN-QFormer is the image selection module, the LLM is the instruction comprehension module, the OUT-QFormer is the dimension transformation module, and the SD model is the image editing module.

[0124] In the image editing interface shown in Figure 6B, you input editing commands such as "Put glasses on the girl in the image" and image selection commands such as "Extract the entire image," upload the attached image file, and click the "Next" button. The above data is then input into the image editing model shown in Figure 6A. As shown in Figure 6C, the target image is obtained after going through several steps such as feature extraction and feature fusion.

[0125] The image editing model also supports batch editing. As shown in Figure 6D, users can upload additional input image attachments by clicking the "Add More" button within the image editing interface. The image editing module performs the same image editing operation on multiple images.

[0126] Furthermore, by clicking the batch edit button, the user can enter the batch edit interface shown in Figure 6E, package documents containing editing commands and image selection commands together with at least one input image to obtain an editable package, and simultaneously upload multiple editable packages to the image editing model shown in Figure 6A. If the editing commands written in each editing package are different, the image editing module will perform different image editing operations on the images in each editable package, but will perform the same image editing operation on images in the same editable package.

[0127] In the image editing model shown in Figure 7A, Image Encoder 1 and IN-QFormer 1 are used to process the input image, Image Encoder 2 and IN-QFormer 2 are used to process the reference image, LLM is used to incorporate the characterized instruction information into the image features of the input image and the image features of the reference image, OUT-QFormer is used to transform the spatial dimensions of the fused features, and SD model is used to perform image editing operations on the input image.

[0128] In the image editing interface shown in Figure 7B, you input editing commands such as "change the cat in the image to the dog in the image below" and an image selection command such as "extract the entire image," upload the attached file of the input image, input the image selection command "extract the dog in the image," upload the attached file of the original reference image, and click the "Next" button. The above data is then input into the image editing model shown in Figure 7A. As shown in Figure 7C, the target image is obtained after going through several steps such as feature extraction and feature fusion.

[0129] Similarly, the image editing model shown in Figure 7A also supports batch editing. For specific implementation details, please refer to the related content in Figures 6C-6D, but we will omit the explanation here.

[0130] Furthermore, it should be noted that, in the specific embodiments of this application, with respect to object data related to the acquisition of input images, acquisition of original reference images, etc., permission or consent of the objects is required to apply the above embodiments of this application to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0131] Based on the same inventive concept as the embodiments of the method described above, the embodiments of the present application further provide an image editing device. As shown in Figure 8, the image editing device 800 is A feature extraction unit 801 acquires an input image and an editing command for the input image, extracts a first image feature containing multiple feature items from the input image, and extracts an instruction text feature containing multiple feature items from the editing command. A feature mining unit 802 that obtains a fused feature by fusing a target image feature and the instruction text feature, wherein the fused feature represents a description for executing the editing instruction on the input image, and the target image feature includes the first image feature, The system may also include an image editing unit 803 that, based on the aforementioned fusion features, determines a first object and an editing operation for the first object from the input image, generates an edited image by performing the editing operation on the first object, and obtains a target image by merging the edited image with the input image.

[0132] For the sake of explanation, the above parts are described separately by function into modules (or units). Of course, when implementing this invention, the functions of each module (or unit) may be realized with the same or multiple software or hardware.

[0133] After introducing the image editing method and apparatus according to exemplary embodiments of the present application, we will now introduce computer equipment according to other exemplary embodiments of the present application.

[0134] Those skilled in the art will understand that various embodiments of the present application can be implemented as systems, methods, or program products. Therefore, these various embodiments may specifically be implemented as complete hardware embodiments, complete software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software. These embodiments may collectively be referred to here as “circuits,” “modules,” or “systems.”

[0135] Based on the same inventive concept as the embodiments of the method described above, embodiments of the present application further provide computer equipment. In one embodiment, the computer equipment may be a server (for example, server 130 shown in Figure 1B). In this embodiment, the configuration of the computer equipment 900 may include at least a memory 901, a communication module 903, and at least one processor 902, as shown in Figure 9.

[0136] Memory 901 stores computer programs executed by processor 902. Memory 901 may mainly include a program storage area and a data storage area. Here, the program storage area can store the operating system and programs necessary for executing instant messaging functions. The data storage area can store various instant messaging information and sets of operation instructions.

[0137] Memory 901 may be volatile memory, such as random-access memory (RAM). Memory 901 may be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD). Alternatively, memory 901 may be any other medium that can be used to carry or store a desired computer program having the form of instructions or data structures, and that is accessible from the computer, but is not limited to these. Memory 901 may be a combination of the above-mentioned memories.

[0138] The processor 902 may include one or more central processing units (CPUs), or it may be a digital processing unit or the like. The processor 902 implements the image editing method described above by calling a computer program stored in memory 901.

[0139] The communication module 903 communicates with terminal devices and other servers.

[0140] The embodiments of this application do not limit the specific connection medium between the memory 901, the communication module 903, and the processor 902. In the embodiments of this application, the memory 901 and the processor 902 are connected via a bus 904 in Figure 9. The bus 904 is shown by a thick line in Figure 9. The connection methods between other components are described schematically only and are not limited. The bus 904 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, it is shown by a single thick line in Figure 9, but this does not represent that there is only one bus or only one type of bus.

[0141] Memory 901 stores a computer storage medium, which stores computer executable instructions, and these computer executable instructions implement the image editing method of the embodiment of the present invention. Processor 902 executes the above image editing method as shown in Figure 3A.

[0142] In other embodiments, the computer equipment may be other computer equipment, for example, the terminal equipment 110 shown in Figure 1B. In this embodiment, the configuration of the computer equipment may include components such as a communication component 1010, a memory 1020, a display unit 1030, a camera 1040, a sensor 1050, an audio circuit 1060, a Bluetooth® module 1070, and a processor 1080, as shown in Figure 10.

[0143] The communication component 1010 is used to communicate with the server. In some embodiments, it may include a Wireless Fidelity (WiFi) module, which belongs to short-range wireless transmission technology. Electronic devices can use the WiFi module to help send and receive information about objects.

[0144] Memory 1020 can be used to store software programs and data. Processor 1080 performs various functions and data processing of terminal device 110 by executing software programs or data stored in memory 1020. Memory 1020 may include high-speed random-access memory, or it may include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Memory 1020 stores an operating system on which terminal device 110 can operate. Memory 1020 in this application can store an operating system and various application programs, and can also store a computer program that performs the image editing method of the embodiment of this application.

[0145] The display unit 1030 can also be used to display information input by an object, information provided to an object, and the graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1030 may include a display screen 1032 provided on the front of the terminal device 110. Here, the display screen 1032 may be in the form of a liquid crystal display or a light-emitting diode. The display unit 1030 can be used to display an image editing interface or the like in the embodiment of the present application.

[0146] The display unit 1030 can also be used to receive input numbers and characters and generate signal inputs related to object settings and function control of the terminal device 110. Specifically, the display unit 1030 may include a touchscreen 1031 provided on the front of the terminal device 110. The touchscreen 1031 can collect touch operations on or near objects (for example, operations such as clicking a button or dragging a scroll box).

[0147] Here, the touchscreen 1031 may be superimposed on the display screen 1032, or the touchscreen 1031 and the display screen 1032 may be integrated to realize the input and output functions of the terminal device 110, and the integrated unit can be called a touch display screen. The display unit 1030 in this application can display an application program and its corresponding operating procedures.

[0148] Camera 1040 can be used to capture still images, and the object can display the images captured by camera 1040 in the application. There may be one or more cameras 1040. An optical image is generated by the lens of the object and projected onto a photosensitive element. The photosensitive element may be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to processor 1080 to convert it into a digital image signal.

[0149] The terminal device may include at least one type of sensor 1050, for example, an accelerometer 1051, a distance sensor 1052, a fingerprint sensor 1053, and a temperature sensor 1054. The terminal device may also be equipped with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0150] The audio circuit 1060, speaker 1061, and microphone 1062 can provide an audio interface between the object and the terminal device 110. The audio circuit 1060 can transmit an electrical signal converted from the received audio data to the speaker 1061. The speaker 1061 converts the electrical signal into an acoustic signal and outputs it. The terminal device 110 may have volume buttons for adjusting the volume of the acoustic signal. Meanwhile, the microphone 1062 converts the collected acoustic signal into an electrical signal. The audio circuit 1060 receives the electrical signal, converts it into audio data, and outputs the audio data to the communication component 1010 for transmission to, for example, another terminal device 110, or outputs the audio data to the memory 1020 for further processing.

[0151] The Bluetooth module 1070 is used to exchange information with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a terminal device can establish a Bluetooth connection with a wearable electronic device (e.g., a smartwatch) that also has a Bluetooth module via the Bluetooth module 1070 for data exchange.

[0152] The processor 1080 is the control center of the terminal device, connecting all parts of the entire terminal using various interfaces and lines, executing or performing software programs stored in memory 1020, and retrieving data stored in memory 1020 to perform various functions of the terminal device and process data. In some embodiments, the processor 1080 may include one or more processing units. The processor 1080 may incorporate an application processor that mainly processes the operating system, user interface, and application programs, and a baseband processor that mainly processes wireless communication. To make it clear, the baseband processor does not have to be incorporated into the processor 1080. The processor 1080 in this application can execute the operating system, application programs, user interface display and touch response, and the image editing method of the embodiment of this application. The processor 1080 is also coupled to the display unit 1030.

[0153] In some possible embodiments, each aspect of the model inconsistency phenomenon recognition method provided herein may be implemented in the form of a program product containing a computer program. When the program product is executed on a computer device, the computer program is used to cause the computer device to perform steps of the image editing method according to various exemplary embodiments of the Application described herein. For example, the computer device may perform steps as shown in Figure 3A.

[0154] A program product may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) may include one or more wired electrical connections, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0155] The program product of the embodiments of the present application may use a portable compact disc read-only memory (CD-ROM), and may contain a computer program that can be executed on an electronic device. However, the program product of the present application is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device.

[0156] The readable signaling medium may include data signals propagated in the baseband or as part of a carrier, the data signals containing readable computer programs. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable signaling medium may be any readable medium other than a readable storage medium, the readable medium can transmit, propagate, or transmit programs for use by or in combination with instruction execution systems, apparatus, or devices.

[0157] Computer programs contained in a readable medium may be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, or any suitable combination thereof.

[0158] A computer program for performing the operations of this invention can be created in any combination of one or more programming languages. These programming languages ​​include object-oriented programming languages ​​(e.g., Java, C++, etc.) and conventional procedural programming languages ​​(e.g., the "C" language or similar programming languages). The computer program may run entirely on the user's computer equipment, partially on the user's computer equipment, as a standalone software package, partially on the user's computer equipment and partially on remote computer equipment, or entirely on remote computer equipment. In the case of remote computer equipment, the remote computer equipment may be connected to the user's computer equipment via any type of network, including a local area network (LAN) or wide area network (WAN), or to external computer equipment (e.g., via the Internet using an Internet service provider).

[0159] It should be noted that while the detailed description above refers to some units or subunits of the apparatus, such divisions are illustrative and not mandatory. In practice, according to embodiments of the present application, the features and functions of two or more units described above may be embodied in a single unit. Conversely, the features and functions of a single unit described above may be further divided so as to be embodied by multiple units.

[0160] Furthermore, although the drawings describe the operation of the method of this application in a specific order, this does not require or suggest that these operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, some steps may be omitted, several steps may be combined into one, and / or one step may be broken down into several steps.

[0161] As those skilled in the art should understand, embodiments of this application may be provided as methods, systems, or computer program products. Therefore, this application may take the form of complete hardware embodiments, complete software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application may take the form of a computer program product implemented on one or more computer-compatible storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) containing a computer-compatible computer program.

[0162] This application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that computer program instructions may implement each flow and / or block in the flowchart and / or block diagram, as well as combinations of flows and / or blocks in the flowchart and / or block diagram. These computer program instructions may be provided to a processor of a general-purpose computer, a dedicated computer, an embedded processor, or other programmable data processing device to generate a machine. The instructions executed by the processor of the computer or other programmable data processing device then generate a device for implementing one or more flows in the flowchart and / or one or more blocks in the block diagram.

Claims

1. A method of image editing performed on computer equipment, The steps include obtaining an input image and an editing command for the input image, The steps include: extracting a first image feature containing multiple feature items from the input image; The steps include: extracting command text features containing multiple feature items from the aforementioned editing command; A step of obtaining a fused feature by fusing a target image feature and the instruction text feature, wherein the fused feature represents a description for executing the editing instruction on the input image, and the target image feature includes the first image feature. The steps include determining a first object and an editing operation for the first object from the input image based on the aforementioned fusion features, The steps include generating an edited image by performing the editing operation on the first object, The process includes the step of obtaining a target image by merging the edited image and the input image. Image editing methods.

2. The step of obtaining a fused feature by fusing the target image feature and the instruction text feature is as follows: A step of obtaining a first degree of relevance between each feature item in the target image features and each feature item in the instruction text features, The process includes the step of obtaining the fused feature by fusing the target image feature and the instruction text feature using the aforementioned first relevance, The image editing method according to claim 1.

3. If a reference image is specified by the aforementioned editing command, Steps to obtain a reference image, The method further includes the step of extracting a second image feature containing multiple feature items from the aforementioned reference image, The target image feature further includes the second image feature, and the fused feature represents a description of executing the editing command on the input image based on the reference image. The image editing method according to claim 1 or 2.

4. The step of determining a first object and an editing operation on the first object from the input image based on the aforementioned fusion features is as follows: A step of determining a second degree of relevance between each feature item in the first image feature and the fused feature, The steps include obtaining a first feature group by determining feature items in the first image feature whose degree of relevance with the fused feature reaches a first threshold, based on the second degree of relevance; The process includes the steps of determining a region from the input image that consists of pixel points corresponding to the first feature group, and determining the first object corresponding to the region. The image editing method according to any one of claims 2.

5. The step of generating an edited image by performing the editing operation on the first object is: The steps include determining a drawing area corresponding to the first object from a drawing target image having the same size as the input image, If the editing operation is an operation to modify the attributes of the first object, the steps include obtaining the edited image by modifying the attributes of the first object within the drawing area based on the editing operation, The editing operation is an operation to replace a first object with a second object, and if no reference image is specified for generating the second object, the editing operation includes the step of selecting a second object from a predetermined object library and drawing the second object in a drawing area to obtain the editing image. The image editing method according to any one of claims 1 to 4.

6. The step of generating an edited image by performing the editing operation on the first object includes, if the editing operation is an operation to replace the first object with a second object and a reference image for generating the second object is specified, the step of obtaining the edited image by determining the image features of the second object based on fusion features and generating an image of the second object in a drawing area corresponding to the first object based on the image features of the second object. The image editing method according to any one of claims 2 to 5.

7. The step of determining a drawing area corresponding to the first object from a drawing target image having the same size as the input image is: The steps include: obtaining drawing image features that include multiple feature items by extracting features from a drawing target image having the same size as the input image; A step of obtaining a third degree of relevance between each feature item in each of the aforementioned drawing image features and the aforementioned fused feature, The process includes the step of obtaining the drawing region by determining a region composed of pixel points corresponding to a feature item whose third relevance is equal to or greater than a first threshold, The image editing method according to claim 5.

8. The step of obtaining the drawing region by determining a region composed of pixel points corresponding to feature items whose third relevance is equal to or greater than the first threshold is as follows: The steps include obtaining at least one drawing image feature group by clustering feature items whose relevance is equal to or greater than a first threshold, For each of the at least one group of drawing image features, Obtain a pixel region that contains at least one pixel point corresponding to each feature item in a single drawn image feature group, The process includes the step of determining the boundary points of the drawing region based on each acquired pixel region, and obtaining the drawing region from the image to be drawn by connecting each boundary point, The image editing method according to claim 7.

9. The step of obtaining a target image by merging the edited image and the input image is as follows: A step of acquiring the target image by superimposing the target image onto the input image, or, A step of extracting objects other than the first object from the input image and merging at least one other object into the target image to obtain the target image, or, The process includes the step of obtaining the target image by merging the target image with the input image from which the first object has been cut out. The image editing method according to any one of claims 1 to 8.

10. The step of obtaining the aforementioned input image is: A step of acquiring an original image and using the original image as the input image, or, The process includes the step of acquiring the input image by selecting an original image in response to an image selection command. The image editing method according to any one of claims 1 to 9.

11. A step of obtaining a fused feature located in a second feature space by dimensional transformation of a fused feature located in a first feature space, further comprising the step of the spatial dimension of the first feature space being lower than the spatial dimension of the second feature space. The image editing method according to claim 2.

12. It is an image editing device, A feature extraction unit that acquires an input image and an editing command for the input image, extracts a first image feature containing multiple feature items from the input image, and extracts an instruction text feature containing multiple feature items from the editing command, A feature mining unit that obtains a fused feature by fusing a target image feature and the instruction text feature, wherein the fused feature represents a description for executing the editing instruction on the input image, and the target image feature includes the aforementioned first image feature, The system includes an image editing unit that, based on the aforementioned fusion features, determines a first object and an editing operation for the first object from the input image, generates an edited image by performing the editing operation on the first object, and obtains a target image by merging the edited image with the input image. Image editing device.

13. A computer device comprising a processor and memory, wherein program code is stored in the memory, and when the program code is executed by the processor, the computer device causes the processor to perform the steps of the image editing method described in any one of claims 1 to 11.

14. A computer-readable storage medium containing program code, wherein when the program code is executed on a computer device, the program code is used to cause the computer device to perform the steps of the image editing method described in any one of claims 1 to 11.

15. A computer program product comprising computer instructions, wherein the computer instructions, when executed by a processor, realize the steps of the image editing method described in any one of claims 1 to 11.

Images