Image editing method and apparatus, and device and storage medium

Abstract

The present application relates to the field of artificial intelligence, and provides an image editing method and apparatus, and a device and a storage medium. The method comprises: acquiring an input image and an editing instruction with regard to the input image; extracting from the input image a first image feature which comprises a plurality of feature items; extracting from the editing instruction an instruction text feature comprising a plurality of feature items; fusing a target image feature and the instruction text feature, so as to obtain a fused feature, wherein the fused feature is used for representing a description of executing the editing instruction on the input image, and the target image feature comprises the first image feature; on the basis of the fused feature, determining a first object from the input image and an editing operation with regard to the first object; executing the editing operation on the first object, so as to generate an edited image; and merging the edited image and the input image, so as to obtain a target image.

Description

Image editing method, device, equipment and storage medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on September 15, 2023, with application number 2023112083362 and application name “Image Editing Method, Device, Equipment and Storage Medium”. The entire contents of the application are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence and provides an image editing method, apparatus, device and storage medium. Background Art

[0003] In recent years, with the rapid development of computer network technology, artificial intelligence (AI) technology has been widely used in the field of image processing, especially the application of AI-based image editing models has become increasingly popular.

[0004] For example, image editing models based on stable diffusion (SD) can provide image editing services. However, the text encoder built into SD has poor text comprehension and reasoning capabilities, which prevents the image editing model from understanding complex editing instructions and, consequently, from generating edited images that match the instructions.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide an image editing method, apparatus, device, and storage medium.

[0007] In a first aspect, an embodiment of the present application provides an image editing method, comprising:

[0008] Acquire an input image and an editing instruction for the input image;

[0009] extracting a first image feature comprising a plurality of feature items from the input image;

[0010] Extracting instruction text features including a plurality of feature items from the editing instruction;

[0011] fusing target image features and the instruction text features to obtain fused features, wherein the fused features are used to represent a description of executing the editing instruction on the input image, the target image features including the first image features;

[0012] determining a first object and an editing operation on the first object in the input image according to the fused features;

[0013] performing the editing operation on the first object to generate an edited image;

[0014] The edited image is merged with the input image to obtain a target image.

[0015] In a second aspect, an embodiment of the present application further provides an image editing device, comprising:

[0016] A feature extraction unit is configured to obtain an input image and an editing instruction for the input image; extract a first image feature including a plurality of feature items from the input image; and extract an instruction text feature including a plurality of feature items from the editing instruction;

[0017] a feature mining unit, configured to fuse target image features with the instruction text features to obtain fused features, wherein the fused features are used to represent a description of executing the editing instruction on the input image, the target image features including the first image features;

[0018] An image editing unit is configured to determine a first object and an editing operation on the first object in the input image based on the fusion feature; perform the editing operation on the first object to generate an edited image; and merge the edited image with the input image to obtain a target image.

[0019] In a third aspect, an embodiment of the present application further provides a computer device comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any one of the above-mentioned image editing methods.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium comprising a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of any one of the above-mentioned image editing methods.

[0021] In a fifth aspect, an embodiment of the present application further provides a computer program product, comprising computer instructions, which are executed by a processor to perform the steps of any one of the above-mentioned image editing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] FIG1A is a schematic diagram of the structure of an image editing model provided in an embodiment of the present application;

[0024] FIG1B is an optional schematic diagram of an application scenario in an embodiment of the present application;

[0025] FIG1C is a schematic diagram showing a comparison of images before and after editing according to an embodiment of the present application;

[0026] FIG2A is a schematic diagram of a process for training an image editing model according to an embodiment of the present application;

[0027] FIG2B is a logic diagram of a training image editing model provided in an embodiment of the present application;

[0028] FIG2C is a logic diagram of generating a preset target image corresponding to a training sample image for an editing function provided by an embodiment of the present application;

[0029] FIG2D is a logical diagram of a preset target image corresponding to a training sample image provided in an embodiment of the present application;

[0030] FIG3A is a schematic diagram of a process of performing an image editing operation on an input image according to an embodiment of the present application;

[0031] FIG3B is a logic diagram of performing an image editing operation on an input image according to an embodiment of the present application;

[0032] FIG3C is a schematic diagram of a process for obtaining fusion features according to an embodiment of the present application;

[0033] FIG3D is a logic diagram of obtaining fusion features according to an embodiment of the present application;

[0034] FIG3E is a schematic diagram of a process for generating a drawing area according to an embodiment of the present application;

[0035] FIG3F is a logic diagram of generating a drawing area according to an embodiment of the present application;

[0036] FIG3G is a logic diagram of generating a drawing area according to an embodiment of the present application;

[0037] FIG3H is a logic diagram of drawing a blue beverage according to an embodiment of the present application;

[0038] FIG3I is a logic diagram of drawing a cat provided in an embodiment of the present application;

[0039] FIG3J is a logical diagram of obtaining a target image using the first image fusion method according to an embodiment of the present application;

[0040] FIG3K is a logic diagram of obtaining a target image using the second image fusion method provided by an embodiment of the present application;

[0041] FIG3L is a logic diagram of obtaining a target image using the third image fusion method according to an embodiment of the present application;

[0042] FIG4A is a schematic diagram of a process for editing an image of a first object according to a second object in a reference image according to an embodiment of the present application;

[0043] FIG4B is a logical diagram of image editing of a first object according to a second object in a reference image according to an embodiment of the present application;

[0044] FIG4C is a logic diagram of drawing glasses in a reference image according to an embodiment of the present application;

[0045] FIG5 is a comparison diagram of the editing effects of the related technology provided in the embodiment of the present application and the present application;

[0046] FIG6A is an image editing model for a landing scenario provided by an embodiment of the present application;

[0047] FIG6B is a target image of a girl wearing glasses generated by editing an image according to an embodiment of the present application;

[0048] FIG6C is a logic diagram of editing an input image to obtain a target image according to an embodiment of the present application;

[0049] FIG6D is a schematic diagram of an interface for uploading attachments in batches according to an embodiment of the present application;

[0050] FIG6E is a schematic diagram of an interface for batch uploading packages to be edited according to an embodiment of the present application;

[0051] FIG7A is an image editing model for a landing scenario provided by an embodiment of the present application;

[0052] FIG7B is a target image in which a cat in the bushes is replaced with a dog in the reference image, provided by an embodiment of the present application;

[0053] FIG7C is a logic diagram of editing an input image to obtain a target image according to an embodiment of the present application;

[0054] FIG8 is a schematic structural diagram of an image editing device provided in an embodiment of the present application;

[0055] FIG9 is a schematic diagram of a hardware structure of a computer device using an embodiment of the present application;

[0056] FIG10 is a schematic diagram of a hardware structure of another computer device to which an embodiment of the present application is applied. DETAILED DESCRIPTION

[0057] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of the technical solutions of this application, but not all of them. Based on the embodiments described in this application document, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the technical solutions of this application.

[0058] The following explains some of the terms used in the embodiments of the present application to facilitate understanding by those skilled in the art.

[0059] Large Language Model (LLM): A deep learning model trained using large amounts of text data through unsupervised or semi-supervised learning. In the field of natural language processing, LLMs can automatically learn language patterns, generate natural language text, or understand the meaning of text. LLMs can handle a variety of natural language tasks, such as text classification, question answering, and conversation, and are an important path to artificial intelligence.

[0060] SD (Stable Diffusion): Diffusion models for text-to-image generation, which can generate high-quality images. These models gradually recover the target image from a noisy image given a text description. Stable Diffusion is an open-source text-to-image model.

[0061] Q-Former, short for Querying Transformer, is a lightweight Transformer architecture designed for vision-language alignment. It introduces a learnable set of query vectors between a frozen visual model and a large language model, enabling efficient visual feature extraction and language representation learning. Q-Former consists of two Transformer submodules: an image transformer and a text transformer. These two submodules share the same self-attention layer for efficient computation and information sharing.

[0062] Image Transformer: Responsible for interacting with the frozen image encoder to extract visual features through a set of learnable query vectors. These query vectors interact not only with themselves, but also with the output of the image encoder through a cross-attention layer to extract the most relevant visual representation of the text.

[0063] Text Converter: This can be used as both a text encoder and a text decoder. During the representation learning phase, it primarily acts as a text encoder, sharing the self-attention layer with the image converter. During the generation learning phase, it acts as a text decoder, responsible for generating text that matches the visual representation.

[0064] The following is a brief introduction to the design concept of the embodiment of this application:

[0065] In some embodiments, an SD-based image editing model can provide image editing services. However, the text understanding and reasoning capabilities of SD's built-in text encoder are poor, resulting in the image editing model being unable to understand complex editing instructions and, consequently, unable to generate target images that match the instructions.

[0066] To improve the model's text comprehension and reasoning capabilities, an LLM can be introduced to understand the content of editing instructions. The image editing model processes the input image and editing instructions through the LLM, obtaining one-dimensional text features that indicate the editing operation to be performed on the image. The model then calls the image editing plug-in that performs the corresponding operation, which then draws the target image based on the one-dimensional text features.

[0067] Although image editing models in related technologies offer multiple image editing functions, each function is implemented by a different image editing plug-in. Because the underlying frameworks of these plug-ins are not interoperable, the model can only call one of these plug-ins to process a task at a time.

[0068] When the input image contains two or more objects of the same type, and the editing instruction only performs image editing operations on some objects of this type, the image editing model mistakenly identifies all objects of this type in the image as the first object and performs local editing operations on them, resulting in a discrepancy between the generated target image and the editing instruction.

[0069] Therefore, in order to solve the above problems, the present application proposes a new image editing method. The method includes: obtaining an input image and an editing instruction for the input image; extracting a first image feature including multiple feature items from the input image; extracting an instruction text feature including multiple feature items from the editing instruction; fusing a target image feature with the instruction text feature to obtain a fused feature, wherein the fused feature is used to represent a description of executing the editing instruction on the input image, wherein the target image feature includes the first image feature; determining a first object and an editing operation on the first object in the input image based on the fused feature; executing the editing operation on the first object to generate an edited image; and merging the edited image with the input image to obtain a target image.

[0070] In summary, the embodiments of the present application can improve the ability to understand the instructions by extracting instruction text features including multiple feature items from the editing instructions; fusing the target image features with the instruction text features to obtain fused features. In this way, even if the input image contains multiple objects of the same type, it is possible to accurately determine which objects are the first objects and which are other objects that do not require image editing operations. This avoids identifying all objects of the same type in the image as the first object, reduces the object false detection rate, and thus improves the accuracy of image editing.

[0071] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application can be combined with each other if there is no conflict.

[0072] The image editing method provided in the embodiment of the present application is implemented through an image editing model, which adopts an end-to-end integrated structural design. As shown in Figure 1A, the image editing model includes a command understanding module and an image editing module.

[0073] The instruction understanding module is configured to understand the content of the editing instruction and integrate the understood instruction text features into the image features of the input image to obtain fused features. The fused features can assist the image editing module in determining the first object and the editing operation on the first object in the input image, and obtaining the drawing area of ​​the first object in the image to be drawn. In this application, the instruction understanding module can be an LLM or other large language model, which is not limited here.

[0074] The image editing module is used to draw a second object obtained based on the editing instruction in the drawing area of ​​the image to be drawn to obtain an edited image, and fuse the edited image with the input image to obtain a target image.

[0075] Sometimes the image size is too large or the number of elements in the image is too large. In order to reduce the computational pressure of the image editing model, an image encoding module and an image selection module are added. The former is used to convert the image in pixel form into data that the model can recognize, and the latter responds to the image selection instruction to select the required image.

[0076] In addition, in order to further improve the image editing effect, a dimension conversion module is added to the model to convert the spatial dimension of the fused features into a spatial dimension compatible with the image editing module.

[0077] The image editing model can be applied to various application scenarios requiring image editing, including image processing, browsers, and social platforms. When using products based on the image editing model, users input natural language editing instructions and an input image, triggering the model to perform image editing operations on the input image combined with a mark-preserving function, outputting the corresponding target image. Compared to cryptic instructions, natural language-based editing instructions are more in line with user habits, lowering the learning curve and difficulty of using the product, and providing users with a convenient and fast method for image editing.

[0078] FIG1B shows one application scenario, which includes two terminal devices 110 and a server 130 . A wired network or a wireless network is used to establish a communication connection between each terminal device 110 and the server 130 .

[0079] As a machine learning model, the image editing model can be deployed in the terminal device 110 and provide the user with image editing functions by local calling. It can also be deployed in the server 130, other independent servers in the network, server clusters in the network, or distributed systems in the network and provide the user with image editing functions by network calling.

[0080] Among them, the terminal device 110 includes but is not limited to: mobile phones, computers (such as tablets, laptops, desktop computers, etc.), smart home appliances, smart voice interaction devices (such as smart watches, smart speakers, etc.), vehicle terminals, aircraft, etc.

[0081] Server 130 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This application does not impose any restrictions on this.

[0082] Terminal device 110 receives the user-entered editing instruction "Change the cat in the picture to a dog" and the input image through image editing interface 120, and transmits the data to server 130 via a pre-established communication link. Server 130 invokes the image editing model and, in response to the editing instruction, performs the image editing operation on the input image, obtaining the target image shown in FIG1C .

[0083] During the model training phase, the image editing model is trained based on sample images. Editing commands include global and local image editing operations. LLM's powerful ability to understand and reason about commands and input images, along with its ability to learn from large amounts of data and integrate diverse information, allows for generalization across diverse functions and accurate image editing of specific objects, saving model training costs.

[0084] 2A and 2B , the image editing model is trained by performing the following operations:

[0085] S201: adopting a cyclic iteration method, based on each training sample image used to train the editing function, train an untrained image editing model until the iterative training stops, and obtains the image editing model that has been trained for the first time.

[0086] Before formal training, a preset target image must be generated for each training sample. Assuming the editing instruction is "add a cat on the chair," a cat is drawn on the training sample image to obtain the preset target image shown in Figure 2C.

[0087] Each iteration includes:

[0088] Based on at least one sample editing instruction, performing an editing operation on the corresponding training sample image to obtain a respective target image;

[0089] Obtain the preset target images of the corresponding training sample images, obtain the image editing loss value generated by this round of iteration based on the target image and the corresponding preset target image, and optimize the parameters of each module in the model in turn based on the image editing loss value.

[0090] Here, if the parameters of the instruction understanding module and the image editing module in the image editing model are optimized separately, the instruction understanding module will not be able to optimize the parameters based on the rich image supervision signals in the image editing module during the parameter adjustment process, and the image editing module will not be able to optimize the parameter adjustment effect based on the text supervision signals of the instruction understanding module during the parameter adjustment process.

[0091] In order to solve this problem, this application adopts a joint optimization method. First, based on the image editing loss value generated by this iteration, the image editing module is optimized; then the image supervision signal and the image editing loss value generated in the image editing module are sent back to the dimension conversion module to optimize the dimension conversion module; then the image supervision signal, the image editing loss value and the corresponding supervision signal generated by the dimension conversion module are sent back to the instruction understanding module to optimize the instruction understanding module, and so on, to complete the parameter optimization of the image selection module and the image encoding module.

[0092] The conditions for stopping the iteration include: (1) the difference between the image editing loss value generated by this iteration and the image editing loss value generated by the previous iteration does not exceed the preset threshold value; (2) the image editing loss value generated by this iteration does not exceed the preset threshold value; (3) the round of this iteration has reached the preset round threshold.

[0093] When any of the above iteration stopping conditions is met, it means that the internal parameters of the model have stabilized after multiple rounds of iteration. Therefore, the model after the round of parameter adjustment that meets the iteration stopping condition is used as the image editing model that has completed the first training. If all the iteration stopping conditions are not met, the next batch of training sample images is read to continue training the model.

[0094] S202: Using a cyclic iteration method, the image editing model that has been trained for the first time is trained based on each training sample image used to train the detection object function until the iterative training stops, and the image editing model that has been trained for the second time is output.

[0095] Before formal training, a preset target image must be generated for each training sample. Assuming the editing instruction is "detect the sparrow on the left side of the image," a detection box is drawn on the training sample image to obtain the preset target image shown in Figure 2D.

[0096] Each iteration includes:

[0097] Based on the at least one editing instruction, performing a detection operation on the corresponding training sample image to obtain a respective target image;

[0098] Obtain the preset target images of the corresponding training sample images, obtain the loss value generated by this round of iteration based on the target image and the corresponding preset target image, and based on the loss value, optimize the parameters of each functional module in the model in turn.

[0099] 3A and 3B , the target image is obtained by performing the following operations:

[0100] S301: Acquire an input image and an editing instruction for the input image.

[0101] S302: Extracting a first image feature comprising multiple feature items from the input image. For example, an image coding model may be used to extract the first image feature representing a visual feature from the input image. Each feature item corresponds to at least one pixel of the input image. The first image feature may be, for example, a feature vector comprising multiple feature items.

[0102] S303: Extracting instruction text features including multiple feature items from the editing instruction. The instruction text feature is, for example, a one-dimensional text vector.

[0103] This application supports the following two ways to obtain input images:

[0104] Method 1: obtaining an original image input by a user, and using the original image as the input image;

[0105] Method 2: In response to an image selection instruction, image selection is performed on the original image input by the user to obtain an input image.

[0106] In the second method of obtaining the input image, the original image is characterized through the image coding model, and then the features of each original image and the image selection instruction are input into the image selection module together, and the features of the original image are aligned with the instruction text features of the image selection instruction, and the image content corresponding to the features of the original image with a high degree of correlation with the instruction text features is obtained to obtain the input image.

[0107] The model's instruction understanding module is used to extract features from the input image to obtain first image features, thereby converting the image in pixel form into data that can be recognized by the instruction understanding module; and the instruction understanding module is used to extract features from the editing instructions to obtain instruction text features, thereby converting the editing instructions in the form of text or strings into data that can be recognized by the model.

[0108] S304: Fusing target image features and the instruction text features to obtain fused features, where the fused features are used to represent a description of executing the editing instruction on the input image, and the target image features include the first image features.

[0109] In some embodiments, a first correlation degree between each feature item in the target image feature and each feature item in the instruction text feature is obtained, and then the target image feature and the instruction text feature are fused using the first correlation degree to obtain the fused feature.

[0110] In some embodiments, an attention mechanism may be used to obtain the fusion features.

[0111] For example, referring to the schematic diagrams shown in FIG. 3C and FIG. 3D , the process of fusing the first image feature and the instruction text feature is as follows:

[0112] S3041: Combine the first image feature with the linear transformation matrix W Q Multiply to get the query vector matrix Q, and combine the first image feature with the linear transformation matrix W V Multiply to get the value vector matrix V, and combine the instruction text features with the linear transformation matrix W K Multiply them together to get the key vector matrix K. Among them, the linear transformation matrix W Q, linear transformation matrix W V Multiplication and linear transformation matrix W K is a trainable matrix.

[0113] S3042: Multiply the query vector matrix Q by the key vector matrix K to obtain the attention weight of the instruction text feature to each target image feature. Here, the attention weight can represent the degree of correlation between the feature item in the first image feature and the feature item in the instruction text feature.

[0114] S3043: Multiply the value vector matrix by the attention weight matrix to obtain the fusion feature.

[0115] S305: Determine a first object and an editing operation on the first object in the input image according to the fusion feature.

[0116] S306: Perform the editing operation on the first object to generate an edited image.

[0117] S307: Merge the edited image with the input image to obtain a target image.

[0118] In summary, the embodiments of the present application can improve the ability to understand the instructions by extracting instruction text features including multiple feature items from the editing instructions; fusing the target image features with the instruction text features to obtain fused features. In this way, even if the input image contains multiple objects of the same type, it is possible to accurately determine which objects are the first objects and which are other objects that do not require image editing operations. This avoids identifying all objects of the same type in the image as the first object, reduces the object false detection rate, and thus improves the accuracy of image editing.

[0119] In some embodiments, in order to further improve the image editing effect, a dimension conversion module is added to the model to perform dimension conversion on the fusion features located in the first feature space to obtain the fusion features located in the second feature space. The spatial dimensions of the converted fusion features are more compatible with the image editing module; wherein the spatial dimension of the first feature space is lower than the spatial dimension of the second feature space.

[0120] In some embodiments, the image editing module adopts an attention mechanism to obtain a second degree of correlation between each feature item in the first image feature of the input image and the fusion feature, and determines the first object and the corresponding editing operation in the input image based on the feature items with a higher degree of correlation.

[0121] In some embodiments, S305 includes: determining a second degree of association between each feature item in the first image feature and the fused feature; based on the second degree of association, determining feature items in the first image feature whose degree of association with the fused feature reaches a first threshold to obtain a first feature group; and determining, in the input image, a region consisting of pixels corresponding to the first feature group to determine the first object corresponding to the region. The first threshold is a preset value, for example, 0.8.

[0122] In some embodiments, S306 includes: determining a drawing area corresponding to the first object in an image to be drawn that has the same size as the input image; when the editing operation is an operation of modifying the properties of the first object, modifying the properties of the first object in the drawing area according to the editing operation to obtain the edited image; when the editing operation is an operation of replacing the first object with a second object, and no reference image for generating the second object is specified, selecting a second object from a preset object library, and drawing the second object in the drawing area to obtain the edited image.

[0123] In some embodiments, S306 includes: when the editing operation is an operation of replacing a first object with a second object, and a reference image for generating the second object is specified, determining the image features of the second object based on the fusion features, and generating the image of the second object in the drawing area corresponding to the first object based on the image features of the second object to obtain an edited image.

[0124] In some embodiments, the degree of correlation between each feature item in the drawing image features of the image to be drawn and the fusion feature is obtained. Based on the feature items with the highest degree of correlation, a drawing area corresponding to the first object is obtained in the image to be drawn that has the same size as the input image. The method for obtaining the degree of correlation between features is similar to the step shown in step S3042 of Figure 3C.

[0125] In some embodiments, determining the drawing area corresponding to the first object in the image to be drawn having the same size as the input image includes: performing feature extraction on the image to be drawn having the same size as the input image to obtain drawing image features including a plurality of feature items; obtaining a third degree of correlation between each feature item in the drawing image features and the fusion feature; and determining an area consisting of pixel points corresponding to feature items whose third degree of correlation is not less than a first threshold value to obtain the drawing area.

[0126] 3E to 3F , the specific process of obtaining the drawing area corresponding to the first object is as follows:

[0127] S3061: Extract features of the image to be drawn that has the same size as the input image to obtain corresponding drawing image features;

[0128] S3062: Obtain a third correlation degree between each feature item in the drawing image feature and the fusion feature, and determine an area consisting of pixel points corresponding to feature items whose third correlation degree is not less than a first threshold to obtain the drawing area.

[0129] The specific implementation process of step 3062 is shown in FIG3G :

[0130] First, clustering the feature items whose correlation degree is not less than a first threshold to obtain at least one drawn image feature group;

[0131] Then, for each of the at least one drawn image feature groups, perform the following operations:

[0132] Obtaining pixel regions corresponding to each feature item in a drawn image feature group, where each pixel region includes at least one pixel point;

[0133] Based on the obtained pixel areas, boundary points of the drawing area are determined, and the drawing area is obtained in the image to be drawn by connecting the boundary points.

[0134] In the embodiment of the present application, the image to be drawn is a noise image. However, due to the limitations of the mapping requirements and in order to clearly display the pixels and the drawing area, the image to be drawn in Figures 3B, 3H and 3I is an image with a white background. These figures are only simple schematic diagrams for illustrating the corresponding steps.

[0135] If the input image is not input into the image editing module, the image editing module will not edit based on the input image, but will regenerate a new target image completely according to the one-dimensional text features output by the LLM. This will make it difficult to maintain the consistency of local structure and local texture between other objects other than the first object in the target image and other objects other than the first object in the input image.

[0136] In order to solve this problem, the present application inputs the input image, the noisy image to be drawn, and the fusion features into the image editing module, draws the second object obtained based on the editing operation of the first object in the image to be drawn, obtains the edited image, and then obtains the target image by merging the edited image with the input image. Therefore, other objects in the target image that have not been edited originate from the input image, so that the images before and after editing remain consistent in the local structure and local texture of the image.

[0137] In this application, image modification tasks can be further divided into two categories. One is to directly edit the first object itself, such as adjusting the image color temperature, adjusting the image contrast, adjusting the color of the object and other attribute editing operations; the other is to edit the first object based on a pre-built object library, such as adding a kitten to the image, putting a pair of sunglasses on the person in the image, modifying the painting style of the input image, etc.

[0138] performing the editing operation on the first object to generate an edited image:

[0139] When the editing operation is an operation of modifying properties of a first object, modifying properties of the first object in the drawing area according to the editing operation to obtain the target image;

[0140] When the editing operation is to replace the first object with the second object and no reference image for generating the second object is specified, the second object is selected from a preset object library and drawn in the drawing area to obtain the target image.

[0141] In some embodiments, when the editing operation is an operation of replacing a first object with a second object, and a reference image for generating the second object is specified, image features of the second object are determined based on the fusion features, and an image of the second object is generated in the drawing area corresponding to the first object based on the image features of the second object to obtain the edited image.

[0142] In some embodiments, when the editing operation is an operation to modify the properties of the first object, the editing operation indicates editing the first object itself. Based on the editing operation, the object obtained after performing the editing operation on the first object is used as the second object, and the second object is drawn in the drawing area of ​​the first object.

[0143] In some embodiments, the editing operation is an operation that replaces a first object with a second object. For example, the editing operation "Change the cat in the image to a dog" is an editing operation on the first object "cat." In the input image before the edit, the first object's position was occupied by "cat." In the edited image, the object in the original first object's position is now "dog." For another example, the editing operation "Change the brown beverage to a blue beverage" is an operation that modifies the properties of the first object "beverage." In the images before and after the edit, the object in the original first object's position is still "beverage."

[0144] For example, the editing operation is "change the coffee-colored drink into a blue drink", and the corresponding editing operation is to modify the color of the first object "a cup of coffee-colored drink". The modified object is "a cup of blue drink", then "a cup of blue drink" is drawn as the second object in the corresponding drawing area, and the editing image shown in Figure 3H is obtained.

[0145] It should be noted that the blue beverage is represented by slashes in FIG3H , and FIG3H is just a simple schematic diagram for illustrating this example.

[0146] In some embodiments, when the editing operation is to replace a first object with a second object, and no reference image for generating the second object is specified, a second object is selected from a preset object library and drawn in the drawing area to obtain the target image. For example, the second object is obtained based on objects of the same type as the second object in the preset object library, and the second object is drawn in the drawing area of ​​the first object.

[0147] Among them, the process of obtaining the second object based on the various similar objects associated with the type of the second object in the preset object library is: based on the type of the second object, obtain the similar objects associated with the type in the preset object library; then use any one of the similar objects as the second object; or, obtain the second object by fusing multiple similar objects.

[0148] For example, if the editing operation is "add a cat to the chair," the corresponding editing operation is to add a second object to the input image. The instruction information carried by this operation is "cat." A set of objects similar to "cat" is obtained from the preset image library, and any cat object in this set is used as the second object. Alternatively, multiple cat objects are merged to obtain a second object. Then, the second object "cat" is drawn in the drawing area of ​​the image to be drawn, resulting in the edited image shown in Figure 3I.

[0149] In the embodiment of the present application, the following three image merging methods are provided:

[0150] Method 1: Overlay the edited image on the input image to obtain the target image.

[0151] For example, in response to the editing operation "change the cat in the picture to a dog", an edited image is drawn, the drawing area in the edited image is an opaque layer, and the other areas in the image are transparent layers. The edited image is directly overlaid on the input image. The second object will cover the first object below, but the other areas of the edited image will not cover the input image, so that the other unedited objects in the target image shown in Figure 3J are consistent with those in the input image to avoid abnormal situations such as image distortion.

[0152] Method 2: Integrate other unedited objects in the input image into the edited image.

[0153] Objects other than the at least one first object are extracted from the input image, and the at least one other object is merged into the edited image to obtain a target image.

[0154] For example, in response to the editing operation "changing the coffee-colored drink to a blue drink", an edited image is drawn, and then the input image is decomposed into multiple images of other objects, and the multiple images of other objects are merged with the edited image to obtain the target image shown in Figure 3K.

[0155] The input image shows a cup of coffee-colored beverage, and the target image shows a cup of blue beverage. However, due to the limitations of the mapping requirements, Figure 3K represents the coffee-colored beverage in white and the blue beverage in slashes. Figure 3K is just a simple schematic diagram used to illustrate this example.

[0156] Method 3: Merge the second object in the edited image into the input image.

[0157] The target image is merged into the input image from which the first object has been removed to obtain the target image.

[0158] For example, in response to the editing operation "change the cat in the picture to a dog", an edited image is drawn, the second object "dog" in the image is extracted, and it is merged into the input image from which the first object "cat" has been removed, to obtain the target image shown in Figure 3L.

[0159] In addition, the present application can also perform image editing on the corresponding first object based on the reference image to obtain the edited second object. Referring to the schematic diagrams shown in Figures 4A and 4B, the process of modifying the input image to the second object in the reference image is as follows:

[0160] S401: Acquire an input image, an editing instruction for the input image, and a reference image.

[0161] S402: Extracting a first image feature including a plurality of feature items from the input image, and extracting a second image feature including a plurality of feature items from the reference image.

[0162] This application also supports the following two methods of obtaining reference images:

[0163] Method 1: obtaining an original reference image input by a user, and using the original reference image as the reference image;

[0164] Method 2: In response to an image selection instruction, image selection is performed on the input original reference image to obtain a reference image.

[0165] To implement the image selection function, the model incorporates two image encoding modules and two image selection modules. Image encoding module 1 and image selection module 1 are used to select the input image input by the user to obtain the input image; image encoding module 2 and image selection module 2 are used to select another original reference image input by the user to obtain the reference image.

[0166] Afterwards, the model's instruction understanding module is used to extract features from the input image to obtain features of each editing image, and feature extraction is performed on the reference image to obtain features of the second image, thereby converting the image in pixel form into data that the model can recognize; and, through the instruction understanding module, feature extraction is performed on the editing instructions to obtain instruction text features, thereby converting the editing instructions in the form of text or character strings into data that the model can recognize.

[0167] S403: Fusing target image features and the instruction text features to obtain fused features, where the fused features are used to represent a description of executing the editing instruction on the input image based on the reference image, and the target image features include the first image features and the second image features.

[0168] In some embodiments, an instruction understanding module that adopts the idea of ​​an attention mechanism integrates instruction text features into image features of the input image and the reference image based on the degree of correlation between target image features and instruction text features to obtain fused features.

[0169] The fusion feature can assist the image editing module in identifying a first object and the corresponding editing operation in the input image, obtaining the drawing area corresponding to the first object in the image to be drawn, and generating a second object based on the features of the reference image specified by the reference fusion feature. In this way, embodiments of the present application can generate a second object for the input image that references the reference image, thereby improving the accuracy of image editing.

[0170] S404: Determine, in the input image, a first object and an editing operation on the first object based on the fused features. Here, because the fused features can specify features of a reference image, embodiments of the present application can accurately execute editing instructions that specify the reference image, thereby improving the accuracy of executing complex editing instructions (e.g., editing instructions that specify a reference image).

[0171] S405: Perform the editing operation on the first object to generate an edited image. In some embodiments, S405 includes: when the editing operation is an operation of replacing the first object with a second object and a reference image for generating the second object is specified, determining the image features of the second object based on the fusion features, and generating the image of the second object in the drawing area corresponding to the first object based on the image features of the second object to obtain the edited image. In some embodiments, S405 includes: determining a drawing area corresponding to the first object in an image to be drawn having the same size as the input image; when the editing operation is an operation of modifying the properties of the first object, modifying the properties of the first object in the drawing area based on the editing operation to obtain the edited image; when the editing operation is an operation of replacing the first object with a second object and no reference image for generating the second object is specified, selecting a second object from a preset object library and drawing the second object in the drawing area to obtain the edited image.

[0172] S406: Merge the edited image with the input image to obtain a target image.

[0173] Compared with related technologies, the image editing method provided by this application can achieve local editing with higher recognition accuracy requirements, especially when performing local editing operations on input images containing multiple similar objects, its model performance is better than the image editing model disclosed under related technologies.

[0174] Assuming the editing instruction is "change the cat in the mirror to a tiger," the input image in FIG5 is edited and drawn using the related art and the image editing method provided by this application, respectively, to obtain the two target images shown in FIG5. As can be clearly seen from the figure, the target image output by the related art changes both cats in the image into tigers, while the target image output by this application retains the background image in the input image and the image of the kitten in the real world, and only changes the cat in the mirror to a tiger.

[0175] In the image editing model shown in FIG6A , the image encoder is the image encoding module, the IN-QFormer is the image selection module, the LLM is the instruction understanding module, the OUT-QFormer is the dimension conversion module, and the SD model is the image editing module.

[0176] In the image editing interface shown in FIG6B , enter the editing command “add glasses to the girl in the picture” and the image selection command “extract the entire image”, upload the attachment of the input image, and click the Next button. Then, input the above data into the image editing model shown in FIG6A . As shown in FIG6C , after multiple steps such as feature extraction and feature fusion, the target image is obtained.

[0177] The image editing model also supports batch editing. As shown in Figure 6D, users can click the Continue Add button in the image editing interface to upload additional input image attachments. The image editing model then performs the same image editing operation on multiple images.

[0178] Users can also click the Batch Edit button to enter the Batch Edit interface shown in Figure 6E , where they can package a document containing editing instructions and image selection instructions with at least one input image to create an edit package. They can then upload multiple edit packages simultaneously to the image editing module shown in Figure 6A . If the editing instructions written into each edit package are different, the image editing module will perform different image editing operations on the images in each edit package. However, the same image editing operations will be performed on images in the same edit package.

[0179] In the image editing model shown in Figure 7A, image encoder 1 and IN-QFormer1 are used to process the input image, image encoder 2 and IN-QFormer2 are used to process the reference image, LLM is used to integrate the characterized instruction information into the image features of the input image and the image features of the reference image, OUT-QFormer is used to convert the spatial dimensions of the fused features, and the SD model is used to perform image editing operations on the input image.

[0180] In the image editing interface shown in FIG7B , enter the editing command “Change the cat in the picture to the dog in the picture below”, the image selection command “Extract the entire image” and upload the attachment of the input image. Then enter the image selection command “Extract the dog in the picture” and upload the attachment of the original reference image. After clicking the Next button, the above data is input into the image editing model shown in FIG7A . As shown in FIG7C , after multiple steps such as feature extraction and feature fusion, the target image is obtained.

[0181] Similarly, the image editing model shown in FIG7A also supports batch editing. For specific implementation methods, please refer to the relevant contents of FIG6C to FIG6D , which will not be described in detail here.

[0182] In addition, it should be noted that in the specific implementation of this application, it involves object data related to obtaining input images, obtaining original reference images, etc. When the above embodiments of this application are applied to specific products or technologies, it is necessary to obtain the object's permission or consent, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0183] Based on the same inventive concept as the above method embodiment, the present application embodiment further provides an image editing device. As shown in FIG8 , the image editing device 800 may include:

[0184] The feature extraction unit 801 is configured to obtain an input image and an editing instruction for the input image; extract a first image feature including multiple feature items from the input image; and extract an instruction text feature including multiple feature items from the editing instruction.

[0185] A feature mining unit 802 is configured to fuse target image features with the instruction text features to obtain fused features, wherein the fused features are used to represent a description of executing the editing instruction on the input image, and the target image features include the first image features.

[0186] The image editing unit 803 is used to determine a first object and an editing operation on the first object in the input image based on the fusion feature; perform the editing operation on the first object to generate an edited image; and merge the edited image with the input image to obtain a target image.

[0187] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0188] After introducing the image editing method and apparatus according to an exemplary embodiment of the present application, a computer device according to another exemplary embodiment of the present application will be introduced.

[0189] Those skilled in the art will appreciate that various aspects of the present application can be implemented as systems, methods, or program products. Therefore, various aspects of the present application can be specifically implemented in the following forms: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation that combines hardware and software aspects, which may be collectively referred to herein as a "circuit," "module," or "system."

[0190] Based on the same inventive concept as the above-mentioned method embodiment, embodiments of the present application also provide a computer device. In one embodiment, the computer device may be a server, such as server 130 shown in FIG1B . In this embodiment, the structure of computer device 900 is shown in FIG9 , and may include at least a memory 901, a communication module 903, and at least one processor 902.

[0191] Memory 901 is used to store computer programs executed by processor 902. Memory 901 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and programs required for running instant messaging functions, while the data storage area may store various instant messaging messages and operating instruction sets.

[0192] Memory 901 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing a desired computer program in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 901 may be a combination of the aforementioned memories.

[0193] The processor 902 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 902 is configured to implement the above-mentioned image editing method when calling the computer program stored in the memory 901 .

[0194] The communication module 903 is used to communicate with terminal devices and other servers.

[0195] The specific connection medium between the memory 901, communication module 903, and processor 902 is not limited in the embodiments of the present application. In Figure 9, the embodiment of the present application shows that the memory 901 and processor 902 are connected via bus 904. Bus 904 is depicted as a bold line in Figure 9. The connection methods between other components are merely schematic and are not intended to be limiting. Bus 904 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Figure 9 only uses a single bold line, but does not represent a single bus or a single type of bus.

[0196] The memory 901 stores a computer storage medium, which stores computer executable instructions for implementing the image editing method of the embodiment of the present application. The processor 902 is used to execute the above-mentioned image editing method, as shown in Figure 3A.

[0197] In another embodiment, the computer device may be another computer device, such as the terminal device 110 shown in FIG1B . In this embodiment, the structure of the computer device may be as shown in FIG10 , including: a communication component 1010, a memory 1020, a display unit 1030, a camera 1040, a sensor 1050, an audio circuit 1060, a Bluetooth module 1070, a processor 1080, and other components.

[0198] The communication component 1010 is used to communicate with the server. In some embodiments, it may include a wireless fidelity (WiFi) module. The WiFi module is a short-range wireless transmission technology. The electronic device can help the object send and receive information through the WiFi module.

[0199] The memory 1020 can be used to store software programs and data. The processor 1080 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1020. The memory 1020 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state memory device. The memory 1020 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 1020 can store the operating system and various application programs, and may also store a computer program that executes the image editing method of the embodiment of the present application.

[0200] The display unit 1030 can also be used to display information input by or provided to an object, as well as a graphical user interface (GUI) for displaying various menus of the terminal device 110. Specifically, the display unit 1030 may include a display screen 1032 disposed on the front of the terminal device 110. The display screen 1032 may be configured in the form of a liquid crystal display, a light-emitting diode, or the like. The display unit 1030 can be used to display, for example, the image editing interface in the embodiments of the present application.

[0201] The display unit 1030 can also be used to receive input digital or character information and generate signal input related to the object setting and function control of the terminal device 110. Specifically, the display unit 1030 may include a touch screen 1031 set on the front of the terminal device 110, which can collect touch operations of objects on or near it, such as clicking a button, dragging a scroll box, etc.

[0202] The touch screen 1031 can be covered on the display screen 1032, or the touch screen 1031 and the display screen 1032 can be integrated to realize the input and output functions of the terminal device 110. The integrated display screen can be simply called a touch screen. In this application, the display unit 1030 can display applications and corresponding operation steps.

[0203] The camera 1040 can be used to capture static images, and the subject can publish the images captured by the camera 1040 through an application. The camera 1040 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the processor 1080 for conversion into a digital image signal.

[0204] The terminal device may further include at least one sensor 1050, such as an acceleration sensor 1051, a distance sensor 1052, a fingerprint sensor 1053, and a temperature sensor 1054. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0205] The audio circuit 1060, speaker 1061, and microphone 1062 can provide an audio interface between the object and the terminal device 110. The audio circuit 1060 can convert the received audio data into an electrical signal and transmit it to the speaker 1061, which converts it into a sound signal for output. The terminal device 110 can also be equipped with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data. The audio data is then output to the communication component 1010 for transmission to, for example, another terminal device 110, or the audio data is output to the memory 1020 for further processing.

[0206] The Bluetooth module 1070 is used to exchange information with other Bluetooth devices having a Bluetooth module through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1070 to exchange data.

[0207] The processor 1080 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing software programs stored in the memory 1020 and calling data stored in the memory 1020, it performs various functions of the terminal device and processes data. In some embodiments, the processor 1080 may include one or more processing units; the processor 1080 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the baseband processor mainly processes wireless communications. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1080. In the present application, the processor 1080 can run the operating system, application programs, user interface display and touch response, as well as the image editing method of the embodiment of the present application. In addition, the processor 1080 is coupled to the display unit 1030.

[0208] In some possible embodiments, various aspects of the method for identifying model contradiction phenomena provided in the present application can also be implemented in the form of a program product, which includes a computer program. When the program product is run on a computer device, the computer program is used to enable the computer device to execute the steps of the image editing method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device can execute the steps shown in Figure 3A.

[0209] The program product can use any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0210] The program product of the embodiment of the present application may be a portable compact disc read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with a command execution system, apparatus, or device.

[0211] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a readable computer program. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with a command execution system, apparatus, or device.

[0212] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0213] The computer program for performing the operations of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer program can be executed entirely on the user's computer device, partially on the user's computer device, as a stand-alone software package, partially on the user's computer device and partially on a remote computer device, or entirely on the remote computer device. In the case of a remote computer device, the remote computer device can be connected to the user's computer device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer device (e.g., through the Internet using an Internet service provider).

[0214] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0215] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0216] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0217] The present application is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program commands. These computer program commands can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the command executed by the processor of the computer or other programmable data processing device produces a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

Claims

1. An image editing method, executed in a computer device, comprising: Acquire an input image and an editing instruction for the input image; Extracting a first image feature including a plurality of feature items from the input image; Extracting instruction text features including a plurality of feature items from the editing instruction; fusing a target image feature and the instruction text feature to obtain a fused feature, wherein the fused feature is used to represent a description of executing the editing instruction on the input image, and the target image feature includes the first image feature; determining a first object and an editing operation on the first object in the input image according to the fusion feature; performing the editing operation on the first object to generate an edited image; The edited image is merged with the input image to obtain a target image.

2. The method of claim 1, wherein: The step of fusing the target image feature and the instruction text feature to obtain a fused feature includes: Acquire a first correlation degree between each feature item in the target image feature and each feature item in the instruction text feature; The target image feature and the instruction text feature are fused using the first association degree to obtain the fused feature.

3. The method according to claim 1 or 2, wherein: When the editing instruction specifies a reference image, the method further includes: Get a reference image; Extracting a second image feature including a plurality of feature items from the reference image; The target image feature further includes the second image feature, and the fusion feature is used to characterize the description of executing the editing instruction on the input image according to the reference image.

4. The method according to claim 2, wherein determining the first object and the editing operation on the first object in the input image according to the fusion feature comprises: Determining a second correlation degree between each feature item in the first image feature and the fusion feature; According to the second correlation degree, determining feature items in the first image features whose correlation degree with the fusion features reaches a first threshold value, so as to obtain a first feature group; In the input image, a region composed of pixel points corresponding to the first feature group is determined to determine the first object corresponding to the region.

5. The method according to any one of claims 1 to 4, wherein: The performing the editing operation on the first object to generate an edited image includes: In an image to be drawn having the same size as the input image, determining a drawing area corresponding to the first object; When the editing operation is an operation of modifying the property of the first object, modifying the property of the first object in the drawing area according to the editing operation to obtain the edited image; When the editing operation is to replace the first object with the second object and a reference image for generating the second object is not specified, a second object is selected from a preset object library and the second object is drawn in a drawing area to obtain the editing image.

6. The method according to any one of claims 2 to 5, wherein: The performing the editing operation on the first object to generate an edited image includes: when the editing operation is an operation of replacing the first object with a second object and a reference image for generating the second object is specified, determining image features of the second object according to fusion features, and generating an image of the second object in a drawing area corresponding to the first object according to the image features of the second object to obtain the edited image.

7. The method of claim 5, wherein: The step of determining a drawing area corresponding to the first object in an image to be drawn having the same size as the input image includes: Extracting features of an image to be drawn that has the same size as the input image to obtain a drawing image feature containing a plurality of feature items; Acquire a third correlation degree between each feature item in each drawn image feature and the fusion feature; An area consisting of pixel points corresponding to feature items whose third correlation degree is not less than the first threshold is determined to obtain the drawing area.

8. The method of claim 7, wherein: The step of determining an area composed of pixels corresponding to feature items whose third correlation degree is not less than a first threshold value to obtain the drawing area includes: Clustering the feature items whose correlation degree is not less than a first threshold value to obtain at least one drawn image feature group; For each of the at least one drawn image feature group, the following operations are performed: Obtaining pixel regions corresponding to each feature item in a drawn image feature group, each pixel region including at least one pixel point; Based on the obtained pixel regions, boundary points of the drawing region are determined, and the drawing region is obtained in the image to be drawn by connecting the boundary points.

9. The method according to any one of claims 1 to 8, wherein: The step of merging the edited image with the input image to obtain a target image comprises: Overlaying the target image on the input image to obtain the target image; or, extracting other objects except the first object in the input image, and merging at least one other object into the target image to obtain the target image; or, The target image is merged into the input image from which the first object has been removed to obtain the target image.

10. The method according to any one of claims 1 to 9, wherein: Obtaining the input image comprises: Acquire an original image, and use the original image as the input image; or, In response to the image selection instruction, image selection is performed on the original image to obtain the input image.

11. The method of claim 2, wherein: Also includes: Performing dimension conversion on the fused features in the first feature space to obtain the fused features in the second feature space; wherein the spatial dimension of the first feature space is lower than the spatial dimension of the second feature space.

12. An image editing device, wherein: include: A feature extraction unit, configured to obtain an input image and an editing instruction for the input image; and extract a first image feature including a plurality of feature items from the input image; Extracting instruction text features including a plurality of feature items from the editing instruction; a feature mining unit, configured to fuse target image features and the instruction text features to obtain fused features, wherein the fused features are used to represent a description of executing the editing instruction on the input image, and the target image features include the first image features; an image editing unit, configured to determine a first object and an editing operation on the first object in the input image according to the fusion feature; performing the editing operation on the first object to generate an edited image; The edited image is merged with the input image to obtain a target image.

13. A computer device, wherein: It comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium, wherein: The method comprises a program code, and when the program code is run on a computer device, the program code is used to make the computer device execute the steps of the method according to any one of claims 1 to 11.

15. A computer program product, wherein: The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images