Image editing and processing method, device, electronic device and storage medium

Abstract

The present invention provides an image editing and processing method, device, electronic device, and storage medium, relating to the field of image processing technology. The method comprises: obtaining an editing request input by a user, segmenting an initial image and the coordinates of selected points of an object to be edited in editing data to obtain a target mask image; determining a cropped image based on the target mask image and the initial image; describing the style of the initial image to obtain corresponding style description text; processing text instructions to obtain content description text and task trigger words; processing based on the cropped image, target mask image, style description text, content description text, and task trigger words to obtain a cropped result image; and then merging the cropped result image with the original image based on the target mask image and the starting coordinates and width and height data of the cropping area to obtain a result image with the same resolution as the original image. This achieves high-quality editing results and meets the actual needs of users.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image editing and processing method, device, electronic device and storage medium. Background Art

[0002] Image editing refers to the use of computer algorithms and techniques to modify and adjust images. More specifically, image editing involves modifying the appearance, structure, and content of an image. This ranges from subtle changes in color and texture to creating complex image content by adding, replacing, or removing objects.

[0003] Current image editing methods edit images using an image editing model corresponding to simple text and image inputs to generate a target image corresponding to the descriptive text. Since text and images can only be edited using a single model, it is difficult to stably achieve high-quality editing results for tasks with complex image content or large content changes, and cannot meet the actual needs of users. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide an image editing and processing method, device, electronic device and storage medium to solve the problem in the prior art that it is difficult to stably achieve high-quality editing results and cannot meet the actual needs of users.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention provides an image editing method, the method comprising:

[0007] Obtaining an editing request input by a user, wherein the editing request includes an initial image, a text instruction, and editing data;

[0008] Segmenting the initial image and the coordinates of the points of the object to be edited in the editing data to obtain a target mask image;

[0009] Determine a cropped image according to the target mask image and the initial image;

[0010] Performing a style description on the initial image to obtain a corresponding style description text;

[0011] Processing the text instructions to obtain content description text and task trigger words;

[0012] Processing is performed based on the cropped image, the target mask image, the style description text, the content description text, and the task trigger word to obtain a cropped result image;

[0013] Image processing is performed according to the cropping result image, the target mask image, and the initial image to obtain a target image.

[0014] Optionally, segmenting the initial image and the coordinates of the selected points of the object to be edited in the editing data to obtain a target mask image includes:

[0015] Calling a pre-trained image segmentation model to process the initial image according to the coordinates of the point selected by the object to be edited in the editing request to obtain an initial mask image, wherein the pre-trained image segmentation model is trained based on a sample set;

[0016] In response to an adjustment request corresponding to the initial mask image, the initial mask image is adjusted based on the adjustment request to obtain a target mask image.

[0017] Optionally, determining a cropped image according to the target mask image and the initial image includes:

[0018] Calculating the position coordinates of the target mask image in the initial image and using them as the cropping area;

[0019] An image corresponding to the cropping area is cropped from the initial image to obtain a cropped image.

[0020] Optionally, the text instruction is processed to obtain a content description text and a task trigger word, including:

[0021] Extracting the text instructions to obtain a content description text corresponding to the scene of the initial image;

[0022] Performing intent recognition on the text instruction to obtain an editing task category;

[0023] Determine a task trigger word corresponding to the editing task category.

[0024] Optionally, performing command extraction on the text command to obtain content description text corresponding to the scene of the initial image includes:

[0025] A target text instruction is obtained by removing text irrelevant to the content description from the text instruction using a preset large language model, wherein the preset large language model is trained based on multiple historical text instructions, an initial image corresponding to the text instruction, and content description text corresponding to the scene;

[0026] The target text instruction is expanded and enhanced to determine a content description text corresponding to the scene of the initial image.

[0027] Optionally, processing is performed based on the cropped image, the target mask image, the style description text, the content description text, and the task trigger word to obtain a cropped result image, including:

[0028] Triggering a corresponding editing task based on the task trigger word to load a corresponding image diffusion model and a corresponding pre-trained editing task;

[0029] Adjusting model parameters of the image diffusion model based on the pre-trained editing task to obtain an adjusted image diffusion model;

[0030] encoding the cropped image using the adjusted image encoder in the image diffusion model, and using the encoded cropped image as an image latent variable;

[0031] adding noise in the mask area corresponding to the image latent variable according to the target mask image using the adjusted image diffusion model to obtain the image latent variable after adding noise;

[0032] Encoding the merged style description text and the content description text using the adjusted image encoder in the image diffusion model to obtain text latent variables;

[0033] The combined text latent variables and the image latent variables after adding noise are processed based on the adjusted image diffusion model to obtain a cropped result image.

[0034] Optionally, performing image processing according to the cropped result image, the target mask image, and the initial image to obtain the target image includes:

[0035] Expanding the mask area in the target mask image and performing fuzzy processing on the expanded part to obtain a fuzzy mask image;

[0036] The cropped result image corresponding to the masked area in the blurred mask image is merged with the initial image to obtain the target image.

[0037] A second aspect of an embodiment of the present invention provides an image editing and processing device, the device comprising:

[0038] An input module, configured to obtain an editing request input by a user, wherein the editing request includes an initial image, a text instruction, and editing data;

[0039] a mask generation processing module, configured to segment the initial image and the coordinates of the selected points of the object to be edited in the editing data to obtain a target mask image; and determine a cropped image based on the target mask image and the initial image;

[0040] A style inference module, configured to describe the style of the initial image and obtain a corresponding style description text;

[0041] A text processing module is used to process the text instructions to obtain content description text and task trigger words;

[0042] An image redrawing module is used to process the cropped image, the target mask image, the style description text, the content description text, and the task trigger word to obtain a cropped result image;

[0043] The post-processing module is used to perform image processing according to the cropping result image, the target mask image, and the initial image to obtain a target image.

[0044] The third aspect of an embodiment of the present invention shows an electronic device, which includes a processor and a memory, wherein the memory is used to store program code and data generated by data, and the processor is used to call the program instructions in the memory to execute the image editing processing method described in the first aspect of the embodiment of the present invention.

[0045] A fourth aspect of an embodiment of the present invention shows a storage medium, which includes a stored program, wherein when the program runs, the device where the storage medium is located is controlled to execute the image editing processing method shown in the first aspect of the embodiment of the present invention.

[0046] Based on the above-mentioned embodiments of the present invention, an image editing processing method, device, electronic device and storage medium are provided, and the method includes: obtaining an editing request input by a user, the editing request including an initial image, text instructions and editing data; segmenting the point coordinates of the object to be edited in the initial image and the editing data to obtain a target mask map; determining a cropped image based on the target mask map and the initial image; performing a style description on the initial image to obtain a corresponding style description text; processing the text instructions to obtain a content description text and a task trigger word; performing processing based on the cropped image, the target mask map, the style description text, the content description text, and the task trigger word to obtain a cropped result image; performing image processing based on the cropped result image, the target mask map and the initial image to obtain a target image. In an embodiment of the present invention, it is first necessary to segment the target mask image corresponding to the editing request, and then crop the cropped image corresponding to the target mask image from the initial image; then, the style description of the initial image is performed, and the content description text and task trigger words corresponding to the file instruction are determined; then, the cropped image, target mask image, style description text, content description text, and task trigger words are processed to obtain a cropped result image; then, based on the target mask image and the starting coordinate point and width and height data of the cropping area, the cropped result image is merged with the original image to obtain a result image with the same resolution as the original image. This achieves precise positioning of the object to be edited under the editing request, maintains the overall consistency of the image style before and after editing, has wide applicability and extremely high editing quality, thereby achieving high-quality editing results and meeting the actual needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0048] Figure 1 A schematic flow chart of an image editing method according to an embodiment of the present invention;

[0049] Figure 2 An example diagram of mask cropping and image cropping shown in an embodiment of the present invention;

[0050] Figure 3 This is an example diagram of the diffusion process (noise addition) according to an embodiment of the present invention;

[0051] Figure 4 This is an example diagram of the reverse diffusion process (denoising) according to an embodiment of the present invention;

[0052] Figure 5 This is a diagram showing the image editing effect according to an embodiment of the present invention;

[0053] Figure 6 This is a visualization example diagram of the image editing process shown in an embodiment of the present invention;

[0054] Figure 7 A schematic structural diagram of an image editing and processing device according to an embodiment of the present invention;

[0055] Figure 8 This is a schematic diagram of the processing of each module shown in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0057] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] It should be noted that the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0059] In this application, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0060] See also Figure 1 , which is a flow chart of an image editing method according to an embodiment of the present invention, the method includes:

[0061] Step S101: obtaining an editing request input by a user, wherein the editing request includes an initial image, text instructions, and editing data;

[0062] In the specific implementation of step S101, the user's input operation is responded to to obtain the image input by the user and use it as the initial image; then, the initial image is displayed in a visual interface so that the user can select the coordinates of the object to be segmented, that is, the point coordinates of the object to be edited, and obtain the text instructions input by the user.

[0063] It should be noted that the number of the selected coordinates of the object to be edited can be one or more.

[0064] The point selection coordinates include two sets of coordinates: a positive feedback coordinate set and a negative feedback coordinate set. A positive feedback coordinate set is a set of one or more coordinate points, indicating that the object to be edited contains this coordinate point. A negative feedback coordinate set is a set of zero or more coordinate points, indicating that the object to be edited does not contain this coordinate point.

[0065] The text instruction includes the editing intention input by the user and the text description of the object to be edited.

[0066] The editing data includes the point coordinates of the object to be edited.

[0067] Step S102: performing segmentation processing on the initial image and the coordinates of the point selected by the object to be edited in the editing data to obtain a target mask image.

[0068] It should be noted that the specific implementation of step S102 includes the following steps:

[0069] Step S11: calling a pre-trained image segmentation model to process the initial image according to the point coordinates of the object to be edited in the editing request to obtain an initial mask image.

[0070] The pre-trained image segmentation model is trained based on a sample set.

[0071] It should be noted that the pre-training process of the specific image segmentation model includes:

[0072] Collect different historical images and point coordinates of different objects selected by the user;

[0073] The model is trained based on the point coordinates and historical images of different objects, and the trained model is used to test the test set to obtain a training mask map; if the training mask map meets the expectations, it is determined that the trained image segmentation model is obtained; if not, the point coordinates and historical images of different objects are used to retrain until the output training mask map meets the expectations.

[0074] It should be noted that the test set refers to some historical images during training and the point coordinates of different objects.

[0075] It is expected that technicians will set it up in advance based on historical images and point coordinates of different objects.

[0076] Step S12: responding to an adjustment request corresponding to the initial mask image, adjusting the initial mask image based on the adjustment request to obtain a target mask image.

[0077] In the specific implementation of step S12, the initial mask image is displayed to the user, allowing the user to manually modify and adjust the mask to obtain an object mask consistent with the user's expectations; in response to the point coordinates of the mask image adjusted by the user, the image is cropped to obtain the target mask image, and it is used as the first input.

[0078] Step S103: determining a cropped image according to the target mask image and the initial image.

[0079] It should be noted that the specific implementation of step S103 includes the following steps:

[0080] Step S21: Calculate the position coordinates of the target mask image in the initial image and use them as the cropping area.

[0081] In the process of implementing step S103, the adjusted initial mask image, that is, the position coordinates x, y and width and height of the target mask image in the initial image, is calculated, and N times the maximum value of the width, height and width is taken as the width and height of the cropping area; the center point of the target mask image is used as the center point of the cropping area, and the cropping area is drawn on the initial image with the center point of the cropping area and the width and height of the cropping area.

[0082] It should be noted that N is set in advance by technicians based on multiple experiments, and can be set to a positive number.

[0083] Step S22: cropping the image corresponding to the cropping area from the initial image to obtain a cropped image.

[0084] In the process of implementing step S22, the cropping area drawn on the initial image is cropped to obtain a cropped image corresponding to the object to be edited, and the cropped image is used as the second input to reduce the size of the redrawn image area and increase the editing speed. The starting coordinate point and width and height data of the cropping area are retained for subsequent processing after editing.

[0085] It should be noted that, in the process of implementing step S102 and step S103, Figure 2 Provide explanation.

[0086] First, a pre-trained image segmentation model is called to segment and mask the initial image according to the coordinates of the point selected by the object to be edited in the editing request to obtain an initial mask map; then, in response to an adjustment request corresponding to the initial mask map, the cropping area corresponding to the initial mask map is expanded to obtain a target mask map, and mask cropping is performed according to the target mask map;

[0087] Next, an image corresponding to the cropping area is cropped from the initial image according to the target mask image to obtain a cropped image.

[0088] Step S104: performing a style description on the initial image to obtain a corresponding style description text.

[0089] Since the text instruction does not contain descriptions related to the image style, and most non-professional users are unable to accurately describe the image style in detail; therefore, in the specific implementation process of step S104, the initial image is input into a preset multimodal large model so that the multimodal large model can reversely infer the style-related descriptions such as the artistic category, color, brightness, depth of field, etc. of the initial image to obtain a style description text, and use it as the third input.

[0090] It should be noted that the style description text is only used to constrain the editing image style and does not include text descriptions of scenes, characters, etc. that are not related to the style; the length of the style description text is not limited, but in order to ensure the detailed description, the length is usually set to more than 256 characters.

[0091] The preset multimodal large model is based on a variety of historical images and the style-related descriptive texts such as the artistic category, color, brightness, depth of field, etc. corresponding to each image as a sample set; and the sample set is divided into a training set and a test set, and the model is trained using the training set to obtain a multimodal large model; then, the multimodal large model is tested using the test set to obtain a test result. If the test result is consistent with the style-related descriptive text in the test set, if so, it is determined that the training of the multimodal large model is completed; if not, the model is re-trained based on the training set until the test result is consistent with the style-related descriptive text in the test set.

[0092] Step S105: Process the text instruction to obtain content description text and task trigger words.

[0093] It should be noted that the specific implementation of step S105 includes the following steps:

[0094] Step S31: extracting the text instructions to obtain a content description text corresponding to the scene of the initial image.

[0095] It should be noted that the specific implementation of step S31 includes the following steps.

[0096] Step S41: using a preset large language model to remove text irrelevant to the content description in the text instruction to obtain a target text instruction;

[0097] Since text instructions usually contain text that is irrelevant to the editing target, directly inputting it into the image diffusion model as editing information will reduce the generation quality; in the specific implementation process of step S41, the large language model is used to filter the input text instructions, remove text that is irrelevant to the content description, and only retain the content description text, that is, the target text instructions are obtained.

[0098] Step S42: using a preset large language model to expand and enhance the target text instruction, and determining a content description text corresponding to the scene of the initial image;

[0099] In the specific implementation of step S42, the target text instruction is expanded and enhanced using the preset large language model to obtain a content description text that is more detailed and fits the scene of the initial image, and the content description text is used as the fourth input.

[0100] In order to better understand the contents shown in step S41 and step S42, an example is given below.

[0101] For example, using the preset large language model to extract the target text instruction "I want to change the girl's top into overalls", after the instruction is extracted, it is obtained as "a girl wearing overalls", and then expanded to "a girl wearing blue overalls with a rose pattern on it."

[0102] It should be noted that the preset large language model here is trained based on multiple historical text instructions, the initial image corresponding to the text instructions, and the content description text corresponding to the scene.

[0103] The specific training process is to take multiple historical text instructions and corresponding content description texts, as well as the initial images corresponding to the text instructions as sample sets; divide the sample sets to obtain training sets and test sets; use the training set to train the large language model, and use the test set to test the large language model. If the output target text instructions are consistent with the corresponding content description text in the test set, and the output description text is consistent with the description text corresponding to the scene in the test set, if they are consistent, it is determined that the trained large language model is obtained; if any one is inconsistent, the training set is used to continue model training until the trained large language model is obtained.

[0104] Step S32: performing intent recognition on the text instruction to obtain an editing task category.

[0105] It's important to note that text commands are unpredictable, and a single, rigid task assignment strategy struggles to adapt to complex and varied user input. Therefore, in addition to the aforementioned training, the pre-built large language model also needs to be trained on text commands and their corresponding editing task categories. Specifically, key features from the text commands are extracted, and a mapping is established between different key features and editing task categories.

[0106] The editing task categories include, but are not limited to, adding new objects, replacing objects, removing objects, and detail enhancement.

[0107] Step S33: Determine the task trigger word corresponding to the editing task category.

[0108] It should be noted that the editing task categories and the task trigger words do not need to be exactly the same, and only need to be mapped one-to-one. That is, a one-to-one mapping relationship between the editing task categories and the task trigger words is preset.

[0109] In the process of implementing step S32, a large language model is used to identify the intent of the text instructions input by the user, and intelligently analyze the categories of enhancements and the like that the user wants to perform. Specifically, the large language model extracts the key features of the text instructions, and searches for the editing task categories corresponding to the key features. The task trigger words corresponding to the editing task categories are then used as the next input for assigning editing tasks, which can effectively improve the applicability of the editing method, and the task categories are used as the fifth input.

[0110] For example: the text instruction is "I want to change the girl's top into overalls". The large language model recognizes the key feature of the text instruction as "change" through intent, and searches for the editing task category corresponding to the key feature "change" as the replacement object, and then determines that the task trigger word corresponding to the editing task category is "replacement object".

[0111] The task category name and the trigger word name do not need to be exactly the same, and only need to be mapped one to one.

[0112] Step S106: Processing is performed based on the cropped image, the target mask image, the style description text, the content description text, and the task trigger words to obtain a cropped result image.

[0113] It should be noted that the specific implementation of step S106 includes the following steps:

[0114] Step S51: triggering a corresponding editing task based on the task trigger word to load a corresponding image diffusion model and a corresponding pre-trained editing task lora;

[0115] It should be noted that the correspondence between different task trigger words and editing tasks is pre-set, and each editing task includes an image diffusion model and a corresponding pre-trained editing task lora.

[0116] Low-Rank Adaptation of Large Language Models (LORA) is a low-rank adaptation technique for fine-tuning large models. It fine-tunes the model by training a low-rank matrix and then injecting these parameters into the original model while maintaining the original capabilities of the pre-trained model.

[0117] The construction of an image diffusion model primarily involves a forward diffusion process and a reverse denoising process. The forward diffusion process gradually adds noise to the original image until it becomes pure noise. The reverse denoising process starts with pure noise and gradually removes it to restore the image. The specific model requires learning parameters through a neural network so that the predicted noise and the actual noise are as close to 1 as possible, thus achieving a trained image diffusion model.

[0118] There are multiple image diffusion models, and each image diffusion model corresponds to a task trigger word.

[0119] Step S52: adjusting the model parameters of the image diffusion model based on the pre-trained editing task lora to obtain an adjusted image diffusion model.

[0120] In the specific implementation process of step S52, the pre-trained editing task lora uses its own training low-rank matrix, and then injects these parameters into the image diffusion model to fine-tune the model and obtain the adjusted image diffusion model.

[0121] Step S53: Encode the cropped image using the adjusted image encoder in the image diffusion model, and use the encoded cropped image as an image latent variable;

[0122] In the specific implementation of step S53 , the second input is encoded using the built-in variational autoencoder of the diffusion model to obtain a cropped image encoding, which is used as an image latent variable.

[0123] Step S54: using the adjusted image diffusion model to add noise in the mask area corresponding to the image latent variable according to the target mask image, to obtain the image latent variable after adding noise.

[0124] In the specific implementation of step S54, the image diffusion model determines that the cropping mask corresponds to a mask area in the image latent variable, and adds noise in the mask area, such as Figure 3 As shown in , at this time, the mask area image becomes a noise image after noise addition, that is, the image potential variable after adding noise is obtained.

[0125] The noise distribution conforms to the Gaussian distribution.

[0126] Step S55: using the adjusted image encoder in the image diffusion model to encode the merged style description text and the content description text to obtain text latent variables.

[0127] In the specific implementation of step S55 , the style description text and the content description text are merged, and the merged style description text and content description text are encoded using an image diffusion model embedded text encoder to obtain text latent variables.

[0128] Step S56: processing the merged text latent variables and the image latent variables after adding noise based on the adjusted image diffusion model to obtain a cropped result image.

[0129] In the specific implementation process of step S56, the image latent variables after adding noise and the text latent variables are merged in the preset latent space to obtain latent variables; then the merged latent variables are gradually denoised by the reverse diffusion process to generate denoised image latent variables, and the denoised image latent variables are decoded into the cropped result image using the image decoder embedded in the image diffusion model.

[0130] It should be noted that the process of performing reverse diffusion on the merged latent variables to gradually denoise and generate the denoised image latent variables can be as follows: Figure 4 shown.

[0131] Step S107: performing image processing according to the cropping result image, the target mask image, and the initial image to obtain a target image.

[0132] It should be noted that the specific process of implementing step S107 includes the following steps:

[0133] Step S61: expanding the mask area in the target mask image and performing fuzzy processing on the expanded part to obtain a fuzzy mask image;

[0134] In the specific implementation of step S61, the mask area in the target mask image is expanded by M pixels, and then the portion of the mask area expanded by M pixels is blurred to obtain a blurred mask image;

[0135] It should be noted that M is set in advance by technicians based on multiple experiments or experience.

[0136] The mask's expansion range is not limited and can be changed with the mask size.

[0137] Step S62: merging the cropped result image corresponding to the masked area in the blurred mask image with the initial image to obtain a target image.

[0138] In the specific implementation of step S62 , a mask area in the cropped result image corresponding to the mask area in the blurred mask image is determined, and the mask area is merged into a corresponding mask position in the initial image to obtain a target image.

[0139] The initial image, that is, the input image Q, will output different target images when different file instructions are executed, such as Figure 5 shown.

[0140] Assume that text instruction 1 is to add a desk and a schoolbag. After processing from step S101 to step S107, the corresponding target image output is Q1. Assume that text instruction 2 is to help me change the boy's shirt into overalls. After processing from step S101 to step S107, the corresponding target image output is Q2. Assume that text instruction 3 is to remove the girl. After processing from step S101 to step S107, the corresponding target image output is Q3. Assume that text instruction 4 is detail enhancement. After processing from step S101 to step S107, the corresponding target image output is Q4.

[0141] In order to better understand the image editing process shown in the above embodiment of the present invention, Figure 6 As shown, an example is given below to illustrate.

[0142] Obtaining an edit request input by a user, the edit request including an initial image a, point coordinates, and a text instruction, wherein the text instruction is "help me change the boy's shirt into overalls";

[0143] The pre-trained image segmentation model is invoked to process the initial image a according to the coordinates of the selected object to be edited in the edit request. After processing in step S102, a target mask image a1 is obtained and used as the first input. An image corresponding to the heated cropping area of ​​the target mask image a1 is cropped from the initial image. After processing in step S103, a cropped image a2 is obtained and used as the second input.

[0144] The initial image is risk-inferred using a preset multimodal large model, that is, after processing in step S104, a style description text "This painting is in the style of Xin Haicheng animation, with a simple background" is obtained and used as the third input.

[0145] The text instruction is extracted using a preset large language model to obtain the content description text "In a classroom, a boy wearing overalls", which is processed in step S105 and used as the fourth input.

[0146] The preset large language model is used to perform intent recognition on the text instruction, and then the corresponding task trigger word is determined to be "replacement object", that is, after the processing of step S105, it is used as the fifth input.

[0147] The first input, the second input, the third input, the fourth input and the fifth input are assigned tasks to perform image redrawing through corresponding image diffusion models, that is, after the processing of step S106 , a cropping result image a3 is obtained.

[0148] Finally, the target mask image a1, the cropped result image a3 and the initial image a are subjected to image mask merging, that is, after the processing in step S107, a result image, that is, the target image, is output.

[0149] In an embodiment of the present invention, an image segmentation model is first used to segment a target mask corresponding to the edit request, and then a cropped image corresponding to the target mask is cropped from the initial image. A multimodal large model then describes the style of the initial image, and a large language model is used to determine the content description text and task trigger words corresponding to the file instruction. The image diffusion model is then used to process the cropped image, target mask, style description text, content description text, and task trigger words obtained from the above processes to obtain a cropped result image. The cropped result image is then merged with the original image based on the target mask and the starting coordinates and width and height data of the cropped region to obtain a result image with the same resolution as the original image. The present invention combines an image segmentation model, a multimodal large model, a large language model, and an image diffusion model to accurately locate the object to be edited under the edit request, maintaining the overall consistency of the image style before and after editing. This approach has wide applicability and extremely high editing quality, thereby achieving high-quality editing results that meet the actual needs of users. Furthermore, the present invention is widely applicable to various image editing tasks such as image appearance, structure, and content modification, and can stably generate high-quality editing results.

[0150] Based on the image editing method shown in the above embodiment of the present invention, the embodiment of the present invention also shows a flow chart of an image editing device, as shown in FIG. Figure 7 As shown, the device includes:

[0151] An input module 701 is configured to obtain an editing request input by a user, wherein the editing request includes an initial image, a text instruction, and editing data;

[0152] The mask generation processing module 702 is used to segment the initial image and the coordinates of the point selected by the object to be edited in the editing data to obtain a target mask image; and determine a cropped image based on the target mask image and the initial image;

[0153] A style inference module 703 is used to describe the style of the initial image and obtain a corresponding style description text;

[0154] A text processing module 704 is used to process the text instruction to obtain a content description text and a task trigger word;

[0155] An image redrawing module 705 is configured to process the cropped image, the target mask image, the style description text, the content description text, and the task trigger word to obtain a cropped result image;

[0156] The post-processing module 706 is configured to perform image processing based on the cropped image, the target mask image, and the initial image to obtain a target image.

[0157] The specific principles and execution processes of each unit in the image editing and processing device disclosed in the above embodiment of the present invention are the same as the corresponding contents in the image editing and processing method provided in the above embodiment of the present invention. Please refer to the corresponding parts of the image editing and processing method disclosed in the above embodiment of the present invention, and no further details will be given here.

[0158] This invention combines an image segmentation model, a large multimodal model, a large language model, and an image diffusion model to precisely locate the object being edited in response to an edit request, maintaining the overall consistency of the image style before and after editing. This approach offers broad applicability and exceptionally high editing quality, resulting in high-quality editing results that meet user needs. Furthermore, it is widely applicable to diverse image editing tasks, such as modifying image appearance, structure, and content, and consistently produces high-quality editing results.

[0159] Optionally, based on the image editing processing device shown in the embodiment of the present invention, a mask generation processing module 702 performs segmentation processing on the initial image and the coordinates of the point selected by the object to be edited in the editing data to obtain a target mask image, specifically for:

[0160] Calling a pre-trained image segmentation model to process the initial image according to the coordinates of the point selected by the object to be edited in the editing request to obtain an initial mask image, wherein the pre-trained image segmentation model is trained based on a sample set;

[0161] In response to an adjustment request corresponding to the initial mask image, the initial mask image is adjusted based on the adjustment request to obtain a target mask image.

[0162] Optionally, based on the image editing processing device shown in the embodiment of the present invention, the mask generation processing module 702 for determining the cropped image according to the target mask image and the initial image is specifically configured to:

[0163] Calculating the position coordinates of the target mask image in the initial image and using them as the cropping area;

[0164] An image corresponding to the cropping area is cropped from the initial image to obtain a cropped image.

[0165] Optionally, based on the image editing and processing device shown in the above embodiment of the present invention, the text processing module 704 includes an instruction extraction module and an intention recognition module;

[0166] An instruction extraction module, configured to extract the text instruction to obtain a content description text corresponding to the scene of the initial image;

[0167] The intention recognition module is used to perform intention recognition on the text instruction to obtain an editing task category; and determine a task trigger word corresponding to the editing task category.

[0168] Optionally, based on the image editing and processing device shown in the above embodiment of the present invention, the image redrawing module 705 is specifically configured to:

[0169] Using a preset large language model to remove text irrelevant to the content description in the text instruction to obtain a target text instruction, the preset large language model being trained based on multiple historical text instructions;

[0170] The target text instruction is expanded and enhanced to determine a content description text corresponding to the scene of the initial image.

[0171] Optionally, based on the image editing and processing device shown in the above embodiment of the present invention, the post-processing module 706 is specifically configured to:

[0172] Processing is performed based on the cropped image, the target mask image, the style description text, the content description text, and the task trigger word to obtain a cropped result image, including:

[0173] Triggering a corresponding editing task based on the task trigger word to load a corresponding image diffusion model and a corresponding pre-trained editing task;

[0174] Adjusting model parameters of the image diffusion model based on the pre-trained editing task to obtain an adjusted image diffusion model;

[0175] encoding the cropped image using the adjusted image encoder in the image diffusion model, and using the encoded cropped image as an image latent variable;

[0176] adding noise in the mask area corresponding to the image latent variable according to the target mask image using the adjusted image diffusion model to obtain the image latent variable after adding noise;

[0177] Encoding the merged style description text and the content description text using the adjusted image encoder in the image diffusion model to obtain text latent variables;

[0178] The combined text latent variables and the image latent variables after adding noise are processed based on the adjusted image diffusion model to obtain a cropped result image.

[0179] The step of performing image processing according to the cropped image, the target mask image, and the initial image to obtain the target image includes:

[0180] Expanding the mask area in the target mask image and performing fuzzy processing on the expanded part to obtain a fuzzy mask image;

[0181] The cropped result image corresponding to the masked area in the blurred mask image is merged with the initial image to obtain the target image.

[0182] Optionally, the process of implementing image editing processing based on the modules shown in the above embodiment of the present invention can be as follows: Figure 8 shown.

[0183] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store image editing processing program code and data, and the processor is used to call the program instructions in the memory to execute the steps shown in the image editing processing method in the above embodiment.

[0184] An embodiment of the present invention provides a storage medium, which includes the electronic device provided by the above-mentioned embodiment of the present application, and the electronic device is used to execute the image editing processing method disclosed in the embodiment of the present application.

[0185] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple. For relevant parts, refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0186] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0187] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image editing method, characterized in that: The method comprises: Obtaining an editing request input by a user, wherein the editing request includes an initial image, a text instruction, and editing data; Segmenting the initial image and the coordinates of the points of the object to be edited in the editing data to obtain a target mask image; Determine a cropped image according to the target mask image and the initial image; Inputting the initial image into a preset multimodal macromodel, so that the multimodal macromodel reversely infers the artistic category, color, brightness, and depth of field of the initial image to obtain a style description text; Processing the text instruction to determine a content description text and a task trigger word corresponding to the scene of the initial image; Triggering a corresponding editing task based on the task trigger word to load a corresponding image diffusion model and a corresponding pre-trained editing task; Adjusting model parameters of the image diffusion model based on the pre-trained editing task to obtain an adjusted image diffusion model; encoding the cropped image using the adjusted image encoder in the image diffusion model, and using the encoded cropped image as an image latent variable; adding noise in the mask area corresponding to the image latent variable according to the target mask image using the adjusted image diffusion model to obtain the image latent variable after adding noise; Encoding the merged style description text and the content description text using the adjusted image encoder in the image diffusion model to obtain text latent variables; The image latent variable after adding noise is merged with the text latent variable in a preset latent space to obtain the latent variable; the merged latent variable is then subjected to a reverse diffusion process to gradually denoise the generated denoised image latent variable, and the denoised image latent variable is decoded into a cropped result image using an embedded image decoder in the image diffusion model; Image processing is performed according to the cropping result image, the target mask image, and the initial image to obtain a target image.

2. The method according to claim 1, characterized in that Segmenting the initial image and the coordinates of the points of the object to be edited in the editing data to obtain a target mask image includes: Calling a pre-trained image segmentation model to process the initial image according to the coordinates of the point selected by the object to be edited in the editing request to obtain an initial mask image, wherein the pre-trained image segmentation model is trained based on a sample set; In response to an adjustment request corresponding to the initial mask image, the initial mask image is adjusted based on the adjustment request to obtain a target mask image.

3. The method according to claim 1, characterized in that Determining a cropped image according to the target mask image and the initial image includes: Calculating the position coordinates of the target mask image in the initial image and using them as the cropping area; An image corresponding to the cropping area is cropped from the initial image to obtain a cropped image.

4. The method according to claim 1, wherein The text instructions are processed to obtain content description text and task trigger words, including: Extracting the text instructions to obtain a content description text corresponding to the scene of the initial image; Performing intent recognition on the text instruction to obtain an editing task category; Determine a task trigger word corresponding to the editing task category.

5. The method according to claim 4, characterized in that Extracting the text instructions to obtain a content description text corresponding to the scene of the initial image includes: A target text instruction is obtained by removing text irrelevant to the content description from the text instruction using a preset large language model, wherein the preset large language model is trained based on multiple historical text instructions, an initial image corresponding to the text instruction, and content description text corresponding to the scene; The target text instruction is expanded and enhanced to determine a content description text corresponding to the scene of the initial image.

6. The method according to claim 1, characterized in that Performing image processing according to the cropped result image, the target mask image, and the initial image to obtain a target image includes: Expanding the mask area in the target mask image and performing fuzzy processing on the expanded part to obtain a fuzzy mask image; The cropped result image corresponding to the masked area in the blurred mask image is merged with the initial image to obtain the target image.

7. An image editing and processing device, characterized in that: The device comprises: An input module, configured to obtain an editing request input by a user, wherein the editing request includes an initial image, a text instruction, and editing data; a mask generation processing module, configured to segment the initial image and the coordinates of the selected points of the object to be edited in the editing data to obtain a target mask image; and determine a cropped image based on the target mask image and the initial image; A style inference module is configured to input the initial image into a preset multimodal large model so that the multimodal large model can infer the artistic category, color, brightness, and depth of field of the initial image to obtain a style description text; A text processing module, configured to process the text instruction and determine a content description text and a task trigger word corresponding to the scene of the initial image; An image redrawing module is used to trigger a corresponding editing task based on the task trigger word to load a corresponding image diffusion model and a corresponding pre-trained editing task; adjust the model parameters of the image diffusion model based on the pre-trained editing task to obtain an adjusted image diffusion model; encode the cropped image using the image encoder in the adjusted image diffusion model, and use the encoded cropped image as an image latent variable; add noise to the mask area corresponding to the image latent variable according to the target mask map using the adjusted image diffusion model to obtain a noisy image latent variable; encode the merged style description text and the content description text using the image encoder in the adjusted image diffusion model to obtain a text latent variable; merge the noisy image latent variable and the text latent variable in a preset latent space to obtain a latent variable; then perform a reverse diffusion process on the merged latent variable to gradually denoise it to generate a denoised image latent variable, and use the image decoder embedded in the image diffusion model to decode the denoised image latent variable into a cropped result image; The post-processing module is used to perform image processing according to the cropping result image, the target mask image, and the initial image to obtain a target image.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, the memory is used to store program code and data generated by data, and the processor is used to call program instructions in the memory to execute the image editing processing method according to any one of claims 1-6.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the image editing method according to any one of claims 1 to 6.

Images