Image dragging and editing method and system

Abstract

The invention provides an image dragging editing method and system, and belongs to the technical field of image processing, and the method comprises the steps: carrying out the intention reasoning based on a first image and the dragging point information of the first image, and obtaining N semantic intentions; the first image is an original image before dragging editing; editing the first image based on the N semantic intentions to obtain N second images; the N second images are used for indicating a user to select a target image from the N second images. According to the image dragging and editing method and system provided by the invention, the dragging and editing result conforming to the expectation of the user can be obtained.

Description

Technical Field

[0001] The present disclosure belongs to the field of image processing technology, and more specifically, to an image drag editing method and system. Background Art

[0002] Traditional image editing conditions include text conditions and image conditions, but these conditions are insufficient in representing some editing attributes (such as shape and position). To solve this problem, recent research has proposed a new task, namely image drag editing. In drag editing, the user inputs the drag points (starting point and target point) as editing conditions, and by moving the object to the target position, precise control of attributes such as shape and position is achieved.

[0003] Currently, the editing results of drag-and-drop editing may differ greatly from the user's expected results, such as Figure 1 As shown, Figure 1 The starting point of the drag point condition in the example is at the horse's head, and the target point is at the upper right corner. The user expects the horse to raise its head, but the actual editing result is that the horse's neck becomes longer. Therefore, further improvements need to be made to the drag editing method. Summary of the invention

[0004] The purpose of the present disclosure is to provide an image drag editing method and system to obtain a drag editing result that meets user expectations.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image dragging and editing method, comprising: Intention reasoning is performed based on the first image and the drag point information of the first image to obtain N semantic intentions; the first image is the original image before drag editing; The first image is edited respectively based on N semantic intentions to obtain N second images; the N second images are used to instruct the user to select a target image from the N second images.

[0006] According to a second aspect of the embodiments of the present disclosure, there is provided an image drag editing device, comprising: An intention inference module, configured to perform intention inference based on a drag point input by a user for a first image to obtain N semantic intentions; the first image is an original image before drag editing; The collaborative guidance module is used to edit the first image based on N semantic intentions respectively to obtain N second images; the N second images are used to instruct the user to select a target image from the N second images.

[0007] According to a third aspect of an embodiment of the present disclosure, there is provided an image drag editing system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the steps of the above-mentioned image drag editing method when executing the computer program.

[0008] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned image drag editing method are implemented.

[0009] The beneficial effects of the image drag editing method and system provided by the embodiments of the present disclosure are: The disclosed embodiments take into account that the same drag point condition can represent multiple semantic intents, while existing methods ignore this semantic diversity and perform drag editing according to only one semantics, resulting in the image after drag editing not being consistent with the result expected by the user. Therefore, the disclosed embodiments first perform intent inference based on the first image and the drag point information of the first image to obtain several possible intents. By explicitly introducing semantics, semantic perception is enhanced, and at the same time, the diversity of editing results is enriched, from which users can select their desired target image, thus achieving the consistency between the drag editing result and the result expected by the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 A schematic diagram of a drag point provided in an embodiment of the present disclosure; Figure 2 A flowchart of an image drag editing method provided by an embodiment of the present disclosure; Figure 3 A schematic diagram of intention inference results provided by an embodiment of the present disclosure; Figure 4 A principle block diagram of an image drag editing method provided by an embodiment of the present disclosure; Figure 5 A structural block diagram of an image drag editing device provided by an embodiment of the present disclosure; Figure 6 A schematic block diagram of an image drag editing system provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures and technologies are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present disclosure. However, it should be clear to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obstructing the description of the present disclosure with unnecessary details.

[0013] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0014] Please refer to Figure 2 , Figure 2 A flowchart of an image drag editing method provided by an embodiment of the present disclosure is provided, and the method includes: S101: Intention inference is performed based on a first image and drag point information of the first image to obtain N semantic intentions; the first image is an original image before drag editing, and N is a natural number.

[0015] In this embodiment, the first image is the original image to be edited. On the image editing interface, the user can directly click or mark the drag point (including the starting point and the target point) on the first image, and then record the coordinates of the starting point and the target point (that is, the drag point information) by listening to the mouse click or touch event.

[0016] Considering that the same drag point condition can represent multiple semantic intents, this embodiment performs intent reasoning based on the first image and the drag point information of the first image, and can obtain multiple semantic intents. Specifically, the user can pre-train a deep learning model, such as a decision tree, a support vector machine (SVM), a neural network, etc., to let the model learn the relationship between image features and drag point information and user intent, and then input the first image and the drag point information of the first image into the trained deep learning model, and the deep learning model outputs the corresponding N semantic intents.

[0017] by Figure 1 For example, based on the drag point information in the figure, the starting point is at the horse's head and the target point is at the upper right, multiple semantic intentions can be generated, such as "make the neck longer", "make the horse raise its head" or "make the two horses closer".

[0018] S102: Editing the first image based on the N semantic intentions respectively to obtain N second images; the N second images are used to instruct the user to select a target image from the N second images.

[0019] In this embodiment, on the basis of obtaining N semantic intents based on the drag point information, the first image can be edited based on the N semantic intents respectively to obtain N second images.

[0020] by Figure 3 Taking the multiple semantic intents in as an example, by editing the first image according to the multiple semantic intents respectively, multiple second images such as "two horses with long necks", "two horses with their heads raised" and "two horses very close to each other" can be generated.

[0021] The user can select an image with desired editing effect, ie, a target image, from the plurality of second images.

[0022] From the above, it can be concluded that the present embodiment takes into account that the same drag point condition can represent multiple semantic intents, while existing methods ignore this semantic diversity and only perform drag editing according to one semantics, resulting in the image after drag editing not being consistent with the result expected by the user. Therefore, the presently disclosed embodiment first performs intent inference based on the first image and the drag point information of the first image to obtain several possible intents. By explicitly introducing semantics, semantic perception is enhanced, and at the same time, the diversity of editing results is enriched, from which users can select their desired target image, thus achieving the consistency between the drag editing result and the result expected by the user.

[0023] In an embodiment of the present disclosure, the first image includes a plurality of objects, and the drag point information includes the coordinates of the drag point. Intent inference is performed based on the first image and the drag point information of the first image to obtain N semantic intents, including: The coordinates of the first image and the drag point are input into the multimodal model to generate description information of the target object; the target object is the object corresponding to the drag point in the first image.

[0024] The description information of the target object, the description information of the first image, and the coordinates of the drag point are input into the large language model to obtain N semantic intents.

[0025] In this embodiment, intent reasoning consists of two key components, namely, a locator based on a large multimodal model (LVLM) and an understander based on a large language model (LLM). The two components are introduced as follows: Locator based on multimodal large model: Given an input image (i.e., the first image), it may contain various objects located in different positions. In order to accurately identify the object of interest, this embodiment uses a pre-trained multimodal large model Osprey. Osprey uses fine-grained mask regions for training and can achieve pixel-level image understanding. For the input image , we will drag the point Input Osprey model to generate description of the manipulated part (i.e. target object) ,Right now .like Figure 4 As shown, = "a woman's nose", which is then fed into a large language model-based understander to infer the underlying intent.

[0026] Understander based on large language model: please refer to Figure 4 , Figure 4 The drag point condition in may contain various semantic intents. For example, it may represent a non-rigid operation such as "look down" or a rigid operation such as "move to the left". The large language model-based understander aims to identify potential semantic intents to achieve semantic-aware drag editing. This embodiment uses the generated description , original image description and drag point Then, the large language model is used to infer possible intention, namely , each output sample may have a different intent or complexity level. Specifically, ,in is the generated text output. , , They represent the predicted intent, the predicted description of the input image (starting cue), and the predicted description of the target image (target cue), respectively. It is The corresponding confidence probability of the generated text output can be selected by sampling according to the confidence probability. Output :

[0027] From the above, it can be concluded that this embodiment uses a multimodal large model and a large language model to perform intent reasoning on the drag point information, converts the user's drag point conditions into potential semantic intentions, and generates corresponding start prompts and target prompts, thereby solving the problem of "what to edit", which is conducive to improving the semantic perception ability of drag editing.

[0028] In one embodiment of the present disclosure, the semantic intent includes a start prompt and a target prompt, the start prompt is used to indicate the description information of the first image, and the target indication is used to indicate the description information of the second image; The first image is edited based on the N semantic intents to obtain N second images, including: For any semantic intent, The first image is denoised based on a starting cue in any semantic intent.

[0029] The first image after the noise processing is performed with denoising based on the target prompt in any semantic intention to generate a second image.

[0030] In this embodiment, the image generation method of the diffusion model can be used to edit the first image based on each semantic intent to obtain N second images. The diffusion model involves the following technologies: Stable Diffusion: The introduction of the denoising diffusion probability model (DDPM) has greatly promoted the development of the image generation field. Specifically, training is performed through a forward denoising process and a reverse denoising process. In the forward denoising process, noise with fixed parameters is continuously added to the image until the image approximates a random Gaussian noise, and in the reverse process, the training network continuously predicts the noise and restores the original image through noise reduction. After training, any sampling from the Gaussian noise can be used for image generation. Stable Diffusion additionally trains a text encoder on the basis of the denoising diffusion probability model, and trains the encoding features of the semantic encoder as conditions in the reverse process to achieve conditional generation. StableDiffusion also trains an image encoder and an image decoder, projects the image into a latent space, and trains a diffusion model on the latent space, which reduces the training cost and improves the generation quality. This embodiment edits the first image based on the Stable Diffusion model.

[0031] DDIM (Denoising Diffusion Implicit Models): Denoising diffusion implicit model, used to sample new images from noise. The model structure used in DDPM is not changed, only the sampling algorithm is improved, thereby accelerating the sampling.

[0032] DDIM Inversion (Denoising Diffusion Implicit Models Inversion): Denoising Diffusion Implicit Model Inversion (DDIM Inversion) is the reverse process of DDIM and is used to add noise to an image.

[0033] Since text conditions can convey semantic information, this embodiment uses the start prompt and target prompt generated by intention reasoning to achieve semantically aware dragging. Specifically, the start prompt is used in the noise addition process, and the target prompt is used in the denoising process to generate the second image. Due to the difference between the start prompt and the target prompt, this asymmetric text condition represents the semantic direction, thereby providing semantic guidance.

[0034] Please refer to Figure 4 In this embodiment, the noise addition process is implemented by DDIM inversion, that is, the first image is converted into noise by DDIM inversion. During the inversion process, the intermediate noise and the corresponding key of the self-attention layer Sum The keys and values ​​in the self-attention module in the DDIM inversion process can be replaced with the keys and values ​​in the self-attention module in the denoising process to ensure the content consistency between the edited second image and the input first image.

[0035] In the inversion process, this embodiment uses the starting hint to convert the input image into the corresponding noise through DDIM inversion, that is, In the denoising process, this embodiment uses the target prompt to generate the second image, that is, .

[0036] Among them, DDIM (Denoising Diffusion Implicit Models) inversion is a method that converts images into noise based on prompts. It aims to gradually restore the noise data to the original data through the reverse process, and add appropriate noise in this process to maintain the statistical characteristics of the data. The DDIM inversion process adopts a deterministic sampling method, that is, the output of each step is based on the determined result of the previous step, rather than random sampling, thereby improving the stability and consistency of the generation process.

[0037] It can be concluded from the above that this embodiment uses the start prompt for the noise addition process and the target prompt for the noise removal process, providing semantic guidance for drag editing, which is conducive to generating semantically controllable editing results.

[0038] In one embodiment of the present disclosure, the denoising process includes a plurality of denoising steps performed sequentially, and the image dragging and editing method further includes: In each denoising step, the image quality is judged, and the image quality guidance function is calculated based on the relative size of the first probability and the second probability; the first probability refers to the probability that the judgment result is a high-quality image, and the second probability refers to the probability that the judgment result is a low-quality image.

[0039] The parameters of the denoising step are adjusted based on the calculation results of the image quality guidance function.

[0040] In this embodiment, the denoising process of the diffusion model includes a plurality of denoising steps iterated in sequence, that is, the noise added in each time step is superimposed on the result after the denoising in the previous time step.

[0041] Existing drag editing methods mainly focus on dragging a specific area of ​​the image to the target location, ignoring possible distortion, which may reduce the overall image quality. In order to ensure the quality of the second image generated, this embodiment trains an image quality discriminator module that can distinguish high-quality images from low-quality images at any denoising step. Specifically, a data set can be first constructed, given a real image and its corresponding text description, and possible drag points and drag directions are randomly simulated. Then, DragonDiffusion is used to edit the image, and the edited image is scored according to methods such as image aesthetic score or image structure score. Images with aesthetic scores below 5 are classified as low-quality images, and the corresponding real images are regarded as high-quality images. These low-quality images, together with the high-quality images, constitute the training data set of the discriminator. In this embodiment, the training data set includes a total of 10,000 high-quality images and 10,000 low-quality images.

[0042] Please refer to Figure 4 , the discriminator consists of the downsampling layer, the middle layer and a linear classifier layer of the U-Net in the diffusion model. During the training process, the downsampling layer is frozen and the middle layer and the linear classification layer are fine-tuned to distinguish whether the image is high-quality or low-quality, that is, for high-quality images y=1, and for low-quality images y=0. Specifically, the input of the discriminator module is and the current time step , the output is the quality of the image The discriminator is trained by minimizing the standard discriminative loss , we can get a discriminator for judging image quality:

[0043] in, represents the expected value of all time steps, when y=1, represents a collection of high-quality images, represents a collection of low-quality images; conversely, when y=0, represents a collection of low-quality images, Represents a collection of high-quality images.

[0044] In each denoising step, based on the trained discriminator The quality of the image obtained in the denoising step can be judged, and then the image quality guidance function can be calculated based on the probability that the judgment result is a high-quality image and the probability that the judgment result is a low-quality image, and the parameters of the denoising step can be adjusted based on the calculation result of the image quality guidance function.

[0045] It can be concluded from the above that this embodiment guides the execution of the denoising step by using the image quality guidance function, which is beneficial to improving the quality of image editing.

[0046] In one embodiment of the present disclosure, calculating the image quality guidance function based on the relative size of the first probability and the second probability includes: By formula Compute the image quality guidance function.

[0047] in, represents the image quality guidance function, Represents the output of the discriminator, y=0 or 1. When y=1, represents the first probability, represents the second probability, represents the discriminator, represents the image obtained by the t-th denoising step.

[0048] In this embodiment, a specific calculation method of the image quality guidance function is given. By calculating the ratio of the first probability to the second probability, the parameters of the denoising step are adjusted with the goal of improving the first probability (that is, the probability of a high-quality image), which is conducive to improving the quality of image editing.

[0049] In one embodiment of the present disclosure, the denoising process includes a plurality of denoising steps performed sequentially, the denoising process includes a plurality of denoising steps performed sequentially, and the image dragging and editing method further includes: In each denoising step, feature extraction is performed on the obtained image to generate a first feature vector, and feature extraction is performed on the image obtained in the corresponding denoising step to generate a second feature vector.

[0050] The energy function of the drag area is calculated based on the first eigenvector, the second eigenvector, the starting drag position and the target drag position; the starting drag position is the position determined by the starting point in the drag point information, the target drag position is the position determined by the target point in the drag point information, and the drag area includes the starting drag position and the target drag position.

[0051] The energy function of the non-drag area is calculated based on the first eigenvector, the second eigenvector and the position of the non-drag area; the non-drag area is the area in the first image except the drag area.

[0052] The editing guidance function is calculated based on the energy function of the dragged area and the energy function of the non-dragged area.

[0053] The parameters of the denoising step are adjusted based on the calculation results of the edit guide function.

[0054] In this embodiment, the editing guide function is added to the denoising step with the goal of editing the target area (i.e., the dragged area) and retaining the irrelevant area (i.e., the non-dragged area). Specifically, in each denoising step, the image obtained in the denoising step is And the image obtained in the corresponding noise addition step The intermediate layer features are extracted by U-Net in the diffusion model. (also known as the first eigenvector) and (That is, the second eigenvector). On this basis, the energy function of the drag area is calculated based on the first eigenvector, the second eigenvector, the starting drag position and the target drag position, the energy function of the non-drag area is calculated based on the first eigenvector, the second eigenvector and the position of the non-drag area, and the editing guidance function is calculated based on the energy function of the drag area and the energy function of the non-drag area. The parameters of the denoising step are adjusted based on the calculation result of the editing guidance function, so that the characteristics of the starting drag position are similar to those of the target drag position, thereby achieving dragging of the object at the starting drag position to the target drag position; at the same time, the characteristics of the irrelevant areas are kept unchanged.

[0055] It can be concluded from the above that this embodiment is based on and The energy function of image editing is constructed based on the correspondence between them, which is conducive to dragging the object at the starting drag position to the target drag position while keeping the features of irrelevant areas unchanged.

[0056] In one embodiment of the present disclosure, calculating the energy function of the drag area based on the first feature vector, the second feature vector, the start drag position, and the target drag position includes: By formula Calculate the energy function of the drag area.

[0057] in, represents the energy function of the dragged area, and is the set constant, represents the first eigenvector, represents the second eigenvector, A mask indicating the starting drag position, A mask representing the target drag position, Indicates the similarity between the first feature vector corresponding to the target drag position and the second feature vector corresponding to the start drag position.

[0058] Calculating the energy function of the non-drag area based on the first eigenvector, the second eigenvector and the position of the non-drag area includes: By formula Calculate the energy function of the non-drag area.

[0059] in, represents the energy function of the non-drag area, A mask indicating the position of the non-drag area. Indicates the similarity between the first feature vector corresponding to the position of the non-drag area and the second feature vector corresponding to the position of the non-drag area.

[0060] The editing guidance function is calculated based on the energy function of the dragged area and the energy function of the non-dragged area, including: By formula Compute the edit bootstrap function.

[0061] in, Represents the edit guide function, and It represents the weight coefficient. Those skilled in the art can design the specific value of the weight coefficient according to actual needs.

[0062] In this embodiment, a specific calculation method of the editing guide function is given. When calculating the energy function of the drag area, by calculating By constraining the similarity between the first feature vector corresponding to the target drag position and the second feature vector corresponding to the starting drag position, the features of the target drag position can be made similar to those of the starting drag position, so that the object at the starting drag position can be dragged to the target drag position. The features of irrelevant areas are constrained to remain unchanged before and after editing, so that the content of irrelevant areas remains unchanged. Finally, the energy function of the dragged area and the energy function of the non-dragged area are weighted averaged to obtain the editing guidance function.

[0063] The similarity calculation can be implemented by using existing methods such as Euclidean distance and cosine similarity, which will not be described in detail here.

[0064] From the above, it can be concluded that this embodiment realizes dragging the object at the starting drag position to the target drag position while keeping the characteristics of the irrelevant areas unchanged by constraining the similarity between the first feature vector corresponding to the target drag position and the second feature vector corresponding to the starting drag position, as well as the characteristics of the irrelevant areas remaining unchanged before and after editing.

[0065] In one embodiment of the present disclosure, the denoising process includes a plurality of denoising steps performed sequentially, the denoising process includes a plurality of denoising steps performed sequentially, and the image dragging and editing method further includes: In each denoising step, the image quality is judged, and the image quality guidance function is calculated based on the relative size of the first probability and the second probability; the first probability refers to the probability that the judgment result is a high-quality image, and the second probability refers to the probability that the judgment result is a low-quality image.

[0066] In each denoising step, feature extraction is performed on the obtained image to obtain a first feature vector, and feature extraction is performed on the image obtained in the corresponding denoising step to obtain a second feature vector.

[0067] The energy function of the drag area is calculated based on the first eigenvector, the second eigenvector, the starting drag position and the target drag position; the starting drag position is the position determined by the starting point in the drag point information, and the target drag position is the position determined by the target point in the drag point information; the drag area is the area determined by the starting drag position and the target drag position. The energy function of the non-drag area is calculated based on the first eigenvector, the second eigenvector and the position of the non-drag area; the non-drag area is the area in the first image except the drag area.

[0068] The editing guidance function is calculated based on the energy function of the dragged area and the energy function of the non-dragged area.

[0069] A first bootstrap function is calculated based on the image quality bootstrap function and the editing bootstrap function.

[0070] The parameters of the denoising step are adjusted based on the calculation results of the first bootstrap function.

[0071] In this embodiment, the image quality guiding function and the editing guiding function can be combined to obtain a first guiding function, and the parameters of the denoising step can be adjusted based on the calculation result of the first guiding function. The first guiding function can be calculated using the following formula: ;in represents the first guidance function, image quality guidance function and edit the boot function The calculation process of has been introduced in detail in the above embodiments and will not be repeated here.

[0072] In summary, this embodiment proposes a semantic-aware image drag editing method based on intent reasoning. First, based on the drag point conditions input by the user, a multimodal large model is used to infer several possible intentions, and the user's drag point is converted into a potential semantic intention, and the corresponding start prompt and target prompt are generated, thereby solving the problem of "what to edit". Then, collaborative guided sampling is proposed to realize image editing, and the start prompt and target prompt generated by intent reasoning are used as text conditions for noise addition and denoising, respectively, to provide semantic guidance for generating semantically controllable results; a quality discriminator is used to provide quality guidance to improve the fidelity of the image. Semantic guidance, quality guidance and editing guidance work together to provide a new perspective on "how to edit", significantly improving editing accuracy and image quality.

[0073] Corresponding to the image dragging and editing method of the above embodiment, Figure 5This is a structural block diagram of an image drag editing device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 5 The image drag editing device 20 includes: an intention reasoning module 21 and a collaborative guidance module 22.

[0074] Among them, the intention reasoning module 21 is used to perform intention reasoning based on the drag point input by the user for the first image to obtain N semantic intentions; the first image is the original image before drag editing.

[0075] The collaborative guidance module 22 is used to edit the first image based on N semantic intentions respectively to obtain N second images; the N second images are used to instruct the user to select a target image from the N second images.

[0076] In one embodiment of the present disclosure, the semantic intent includes a start prompt and a target prompt, the start prompt is used to indicate the description information of the first image, and the target indication is used to indicate the description information of the second image; the collaborative guidance module 22 is specifically used to: For any semantic intent, The first image is denoised based on a starting cue in any semantic intent.

[0077] The first image after the noise processing is performed with denoising based on the target prompt in any semantic intention to generate a second image.

[0078] In one embodiment of the present disclosure, the denoising process includes a plurality of denoising steps performed sequentially, and the collaborative guidance module 22 is specifically used to: In each denoising step, the image quality is judged, and the image quality guidance function is calculated based on the relative size of the first probability and the second probability; the first probability refers to the probability that the judgment result is a high-quality image, and the second probability refers to the probability that the judgment result is a low-quality image.

[0079] The parameters of the denoising step are adjusted based on the calculation results of the image quality guidance function.

[0080] In one embodiment of the present disclosure, the collaborative guidance module 22 is further configured to: By formula Compute the image quality guidance function.

[0081] in, represents the image quality guidance function, Represents the output of the discriminator, y=0 or 1. When y=1, represents the first probability, represents the second probability, represents the discriminator, represents the image obtained by the t-th denoising step.

[0082] In one embodiment of the present disclosure, the denoising process includes a plurality of denoising steps performed sequentially, and the denoising process includes a plurality of denoising steps performed sequentially. The collaborative guidance module 22 is specifically used for: In each denoising step, feature extraction is performed on the obtained image to generate a first feature vector, and feature extraction is performed on the image obtained in the corresponding denoising step to generate a second feature vector.

[0083] The energy function of the drag area is calculated based on the first eigenvector, the second eigenvector, the starting drag position and the target drag position; the starting drag position is the position determined by the starting point in the drag point information, and the target drag position is the position determined by the target point in the drag point information; the drag area is the area determined by the starting drag position and the target drag position.

[0084] The energy function of the non-drag area is calculated based on the first eigenvector, the second eigenvector and the position of the non-drag area; the non-drag area is the area in the first image except the drag area.

[0085] The editing guidance function is calculated based on the energy function of the dragged area and the energy function of the non-dragged area.

[0086] The parameters of the denoising step are adjusted based on the calculation results of the edit guide function.

[0087] In one embodiment of the present disclosure, the collaborative guidance module 22 is further configured to: By formula Calculate the energy function of the drag area.

[0088] in, represents the energy function of the dragged area, and is the set constant, represents the first eigenvector, represents the second eigenvector, A mask indicating the starting drag position, A mask representing the target drag position, Indicates the similarity between the first feature vector corresponding to the target drag position and the second feature vector corresponding to the start drag position.

[0089] Calculating the energy function of the non-drag area based on the first eigenvector, the second eigenvector and the position of the non-drag area includes: By formula Calculate the energy function of the non-drag area.

[0090] in, represents the energy function of the non-drag area, A mask indicating the position of the non-drag area. Indicates the similarity between the first feature vector corresponding to the position of the non-drag area and the second feature vector corresponding to the position of the non-drag area.

[0091] The editing guidance function is calculated based on the energy function of the dragged area and the energy function of the non-dragged area, including: By formula Compute the edit bootstrap function.

[0092] in, Represents the edit guide function, and Represents the weight coefficient.

[0093] In one embodiment of the present disclosure, the denoising process includes a plurality of denoising steps performed sequentially, and the denoising process includes a plurality of denoising steps performed sequentially. The collaborative guidance module 22 is specifically used for: In each denoising step, the image quality is judged, and the image quality guidance function is calculated based on the relative size of the first probability and the second probability; the first probability refers to the probability that the judgment result is a high-quality image, and the second probability refers to the probability that the judgment result is a low-quality image.

[0094] In each denoising step, feature extraction is performed on the obtained image to obtain a first feature vector, and feature extraction is performed on the image obtained in the corresponding denoising step to obtain a second feature vector.

[0095] The energy function of the drag area is calculated based on the first eigenvector, the second eigenvector, the starting drag position and the target drag position; the starting drag position is the position determined by the starting point in the drag point information, and the target drag position is the position determined by the target point in the drag point information; the drag area is the area determined by the starting drag position and the target drag position. The energy function of the non-drag area is calculated based on the first eigenvector, the second eigenvector and the position of the non-drag area; the non-drag area is the area in the first image except the drag area.

[0096] The editing guidance function is calculated based on the energy function of the dragged area and the energy function of the non-dragged area.

[0097] A first bootstrap function is calculated based on the image quality bootstrap function and the editing bootstrap function.

[0098] The parameters of the denoising step are adjusted based on the calculation results of the first bootstrap function.

[0099] See also Figure 6 , Figure 6Schematic diagram of an image drag editing system provided by an embodiment of the present disclosure. Figure 6 The image drag editing system 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303 and one or more memories 304. The processors 301, input devices 302, output devices 303 and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of each module / unit in the above-mentioned device embodiments, such as Figure 5 The functions of modules 21 to 22 are shown.

[0100] It should be understood that in the embodiment of the present disclosure, the processor 301 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0101] The input device 302 may include a touch panel, a fingerprint collection sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 303 may include a display (LCD, etc.), a speaker, etc.

[0102] The memory 304 may include a read-only memory and a random access memory, and provide instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0103] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the image drag editing method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the image drag editing system described in the embodiments of the present disclosure, which will not be repeated here.

[0104] In another embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by the processor, all or part of the processes in the above-mentioned embodiment method are implemented, and the computer program can also be completed by instructing the relevant hardware through the computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, the steps of each of the above-mentioned method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0105] The computer-readable storage medium may be an internal storage unit of the image drag editing system of any of the aforementioned embodiments, such as a hard disk or memory of the image drag editing system. The computer-readable storage medium may also be an external storage device of the image drag editing system, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the image drag editing system. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of the image drag editing system. The computer-readable storage medium is used to store computer programs and other programs and data required by the image drag editing system. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0106] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the image drag editing system and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0108] In the several embodiments provided in the present application, it should be understood that the disclosed image drag editing system and method can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or it can be an electrical, mechanical or other form of connection.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present disclosure.

[0110] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0111] The above are only specific embodiments of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present disclosure, and these modifications or replacements should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image drag editing method, characterized in that: include: Performing intention inference based on the first image and the drag point information of the first image to obtain N semantic intentions; The first image is the original image before drag editing; The first image is edited respectively based on N semantic intentions to obtain N second images; the N second images are used to instruct the user to select a target image from the N second images.

2. The image drag editing method according to claim 1, characterized in that: The semantic intent includes a start prompt and a target prompt, the start prompt is used to indicate the description information of the first image, and the target indication is used to indicate the description information of the second image; The editing of the first image based on the N semantic intentions to obtain the N second images includes: For any semantic intent, Noising the first image based on a start hint in any one of the semantic intents; The first image after noise processing is denoised based on the target prompt in any one of the semantic intentions to generate a second image.

3. The image drag editing method according to claim 2, characterized in that: The denoising process includes a plurality of denoising steps performed sequentially, and the image dragging and editing method further includes: In each denoising step, the image quality is judged, and the image quality guidance function is calculated based on the relative size of the first probability and the second probability; the first probability refers to the probability that the judgment result is a high-quality image, and the second probability refers to the probability that the judgment result is a low-quality image; The parameters of the denoising step are adjusted based on the calculation result of the image quality guidance function.

4. The image drag editing method according to claim 3, characterized in that: The calculating of the image quality guidance function based on the relative size of the first probability and the second probability comprises: By formula Calculate image quality guidance function; in, represents the image quality guidance function, Represents the output of the discriminator, y=0 or 1. When y=1, represents the first probability, represents the second probability, represents the discriminator, represents the image obtained by the t-th denoising step.

5. The image drag editing method according to claim 2, characterized in that: The denoising process includes a plurality of denoising steps executed sequentially, the denoising process includes a plurality of denoising steps executed sequentially, and the image dragging and editing method further includes: In each denoising step, feature extraction is performed on the obtained image to generate a first feature vector, and feature extraction is performed on the image obtained in the corresponding denoising step to generate a second feature vector; An energy function of the drag area is calculated based on the first feature vector, the second feature vector, a start drag position and a target drag position; the start drag position is a position determined by a start point in the drag point information, and the target drag position is a position determined by a target point in the drag point information; the drag area is an area determined by the start drag position and the target drag position; Calculating an energy function of a non-drag area based on the first feature vector, the second feature vector, and the position of the non-drag area; the non-drag area is an area in the first image excluding the drag area; Calculating an editing guidance function based on the energy function of the drag area and the energy function of the non-drag area; The parameters of the denoising step are adjusted based on the calculation results of the editing guide function.

6. The image drag editing method according to claim 5, characterized in that: The calculating the energy function of the drag area based on the first feature vector, the second feature vector, the start drag position and the target drag position includes: By formula Calculate the energy function of the drag area; in, represents the energy function of the dragged area, and is the set constant, represents the first eigenvector, represents the second eigenvector, A mask indicating the starting drag position, A mask representing the target drag position, Represents the similarity between the first feature vector corresponding to the target drag position and the second feature vector corresponding to the start drag position; The calculating the energy function of the non-drag area based on the first feature vector, the second feature vector and the position of the non-drag area comprises: By formula Calculate the energy function of the non-drag area; in, represents the energy function of the non-drag area, A mask indicating the position of the non-drag area. Represents the similarity between the first feature vector corresponding to the position of the non-drag area and the second feature vector corresponding to the position of the non-drag area; The calculating of the editing guidance function based on the energy function of the drag area and the energy function of the non-drag area comprises: By formula Calculate the edit boot function; in, Represents the edit guide function, and Represents the weight coefficient.

7. The image drag editing method according to claim 2, characterized in that: The denoising process includes a plurality of denoising steps executed sequentially, the denoising process includes a plurality of denoising steps executed sequentially, and the image dragging and editing method further includes: In each denoising step, the image quality is judged, and the image quality guidance function is calculated based on the relative size of the first probability and the second probability; the first probability refers to the probability that the judgment result is a high-quality image, and the second probability refers to the probability that the judgment result is a low-quality image; In each denoising step, feature extraction is performed on the obtained image to obtain a first feature vector, and feature extraction is performed on the image obtained in the corresponding denoising step to obtain a second feature vector; An energy function of the drag area is calculated based on the first feature vector, the second feature vector, a start drag position and a target drag position; the start drag position is a position determined by a start point in the drag point information, and the target drag position is a position determined by a target point in the drag point information; the drag area is an area determined by the start drag position and the target drag position; Calculating an energy function of a non-drag area based on the first feature vector, the second feature vector, and the position of the non-drag area; the non-drag area is an area in the first image excluding the drag area; Calculating an editing guidance function based on the energy function of the drag area and the energy function of the non-drag area; Calculating a first guided function based on the image quality guided function and the editing guided function; The parameters of the denoising step are adjusted based on the calculation result of the first bootstrap function.

8. An image drag editing device, characterized in that: include: An intention inference module, configured to perform intention inference based on a drag point input by the user for the first image, and obtain N semantic intentions; The first image is the original image before drag editing; The collaborative guidance module is used to edit the first image based on N semantic intentions respectively to obtain N second images; the N second images are used to instruct the user to select a target image from the N second images.

9. An image drag editing system, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images