Image editing and generating method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image editing and generating method and device, electronic equipment and a storage medium, and relates to the technical field of image generation, and the method comprises the steps: obtaining a to-be-processed image and a target instruction, constructing an image description of the to-be-processed image according to a scene in the to-be-processed image, converting the image description into a text mark, and storing the text mark in the target instruction; the method comprises the steps of obtaining a to-be-processed image, embedding a text mark into a high-dimensional space through an embedding model to obtain text embedding, obtaining an element vector of the to-be-processed image according to the to-be-processed image and the text embedding through a diffusion model, designing an editing operator according to a target instruction, updating the element vector by using the editing operator, and generating a result image according to the updated element vector through the diffusion model. According to the method, efficient, accurate and flexible editing of the image is realized by combining the diffusion model and the editing operator, and the problems that the expression of the model on a specific target category cannot be improved, the problem of long tail distribution cannot be solved and data similar to a real scene cannot be generated in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of image generation technology, and in particular to an image editing and generating method, device, electronic equipment and storage medium. Background Art

[0002] In the field of deep learning, in order to improve the generalization ability and performance of the model, data enhancement technology is widely used to increase the diversity and quantity of data. It mainly includes color enhancement, quality enhancement, target enhancement and synthetic / simulated data. Using simulation platforms (such as Omniverse) or generating diffusion models, synthetic data can be generated according to the description of scenes and targets. A large amount of data with specific attributes can be generated for training models.

[0003] However, existing technologies mainly focus on the overall changes of images. Although they can enhance the robustness of the model, they cannot directly improve the performance of the model on specific target categories. In addition, the mapping method is often too simple and cannot form an effective semantic association with the background, resulting in the generated images being semantically unnatural. The source of mapping targets is limited, and the problem of few target categories cannot be fundamentally solved.

[0004] In addition, simulation data generation requires building models for specific scenarios, which is costly and has limited usage scenarios. Although synthetic data can be generated in batches, it is difficult to directly generate data that meets the requirements through a simple prompt. At the same time, there are often large data domain differences between synthetic / simulated data and real scenarios, which will affect the performance of the model in real scenarios.

[0005] In summary, existing data augmentation techniques have achieved certain results in improving data diversity and quantity, but they are still insufficient in improving the performance of models on specific target categories, solving the long-tail distribution problem, and generating data close to real scenarios. Therefore, it is necessary to explore more refined and efficient data augmentation methods to further improve the performance of deep learning models.

[0006] Therefore, there is an urgent need for a more refined and efficient image editing and generation method that can generate data close to real scenes. Summary of the invention

[0007] The embodiments of the present invention provide an image editing and generation method to solve the problems that the prior art cannot improve the performance of the model on a specific target category, cannot solve the long-tail distribution problem, and cannot generate data close to the real scene. The technical solution is as follows:

[0008] According to one aspect of the present invention, an image editing and generation method includes: obtaining an image to be processed and a target instruction, constructing an image description of the image to be processed according to a scene in the image to be processed; converting the image description into a text tag, embedding the text tag into a high-dimensional space through an embedding model to obtain a text embedding; obtaining a latent vector of the image to be processed according to the image to be processed and the text embedding through a diffusion model, and designing an editing operator according to the target instruction; updating the latent vector using the editing operator to obtain an updated latent vector, and generating a result image according to the updated latent vector through a diffusion model.

[0009] In one of the embodiments, constructing an image description of the image to be processed based on the information and scene in the image to be processed is achieved by the following steps: using a labeling tool to label the image to be processed to obtain a target label; the target label includes a detection box and a category; constructing an image description based on the scene and target label of the image to be processed; the image description includes a scene description, location, attributes and category.

[0010] In one of the embodiments, converting the image description into text tags and embedding the text tags into a high-dimensional space through an embedding model to obtain text embedding is achieved through the following steps: converting the image description into text tags through a text tagger; converting the text tags into corresponding embedding vectors through an embedding model; the embedding vectors are used to represent the semantic relationship between text tags; all the embedding vectors are combined to obtain text embedding; and the text embedding is used to represent the overall semantic information of the image to be processed.

[0011] In one embodiment, the diffusion model includes a generator with a U-net architecture and a text encoder, and the network layers of the cross-attention mechanism in the generator are labeled.

[0012] In one of the embodiments, obtaining the latent vector of the image to be processed according to the image to be processed and the text embedding by using the diffusion model is achieved by the following steps: selecting a real image as a target image and a noise image according to the image to be processed, and configuring the parameters of the diffusion model and the DDIM algorithm; using the diffusion model and the DDIM algorithm to perform inverse deduction according to the real image and the text embedding to obtain the latent vector.

[0013] In one of the embodiments, an editing operator is designed according to the target instruction, and the editing operator is used to update the latent vector to obtain an updated latent vector through the following steps: editing information is obtained according to the target instruction, and an editing operator is designed according to the editing information; the editing information includes editing type, editing features, editing parameters and editing loss calculation method; a denoised image is generated according to the latent vector through the diffusion model, and the editing loss in the denoised image generation process is calculated using the editing operator; the editing loss is used to quantify the difference between the target instruction and the denoised image; and the latent vector is updated according to the editing loss through a back propagation algorithm until a set condition is met to obtain an updated latent vector.

[0014] In one of the embodiments, after generating the result image, the method further comprises the following steps: post-processing and quality assessment of the result image; the post-processing comprises color correction and sharpening; the quality assessment comprises subjective visual inspection and objective quantitative indicators.

[0015] According to one aspect of the present invention, an image editing and generating device includes: an image processing module, used to obtain an image to be processed and a target instruction, and construct an image description of the image to be processed according to the scene in the image to be processed; a text embedding module, used to convert the image description into a text tag, and embed the text tag into a high-dimensional space through an embedding model to obtain a text embedding; an image editing module, used to obtain a latent vector of the image to be processed according to the image to be processed and the text embedding through a diffusion model, and design an editing operator according to the target instruction; an image generating module, used to update the latent vector using the editing operator to obtain an updated latent vector, and generate a result image according to the updated latent vector through the diffusion model.

[0016] According to one aspect of the present invention, an electronic device includes at least one processor and at least one memory, wherein the memory stores computer-readable instructions; the computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the image editing and generation method as described above.

[0017] According to one aspect of the present invention, a storage medium stores computer-readable instructions thereon, wherein the computer-readable instructions are executed by one or more processors to implement the image editing and generating method as described above.

[0018] The beneficial effects brought by the technical solution provided by the present invention are:

[0019] In the above technical scheme, the present invention first obtains the image to be processed and the target instruction, constructs the image description of the image to be processed according to the scene in the image to be processed, converts the image description into a text tag, embeds the text tag into a high-dimensional space through an embedding model to obtain a text embedding, obtains the latent vector of the image to be processed according to the image to be processed and the text embedding through a diffusion model, designs an editing operator according to the target instruction, updates the latent vector using the editing operator to obtain an updated latent vector, generates a result image according to the updated latent vector through a diffusion model, and realizes efficient, accurate and flexible editing of the image by combining image description, text embedding, diffusion model and editing operator, while improving the user interaction experience and the diversity of image generation, avoiding the overfitting phenomenon caused by single data in specific scenes, and at the same time, target editing based on real images can reduce the difference between synthetic and simulated data and real scenes, and can realize flexible editing and enhancement of real scene images, meet diverse image processing needs, and thus can effectively solve the problems that the prior art cannot improve the performance of the model on specific target categories, cannot solve the long-tail distribution problem, and cannot generate data close to the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0021] Figure 1 is a flow chart of an image editing and generating method according to an exemplary embodiment;

[0022] Figure 2 It is a flowchart of an image editing and generating method in an application scenario;

[0023] Figure 3 is a block diagram of an image editing and generating device according to an exemplary embodiment;

[0024] Figure 4 is a hardware structure diagram of an electronic device according to an exemplary embodiment;

[0025] Figure 5 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0027] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present disclosure refers to the presence of the features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0028] The present invention provides an image editing and generating method, which realizes effective monitoring and early warning processing of arc faults in power grids through real-time collection of power data, data preprocessing and fusion, construction of advanced image editing and generating models, and timely early warning processing and model optimization, etc., and solves the problems that the existing technology cannot cope with complex practical application scenarios, and the image editing and generating range, detection accuracy and response capability are insufficient. The image editing and generating method is suitable for an image editing and generating device, which can be an electronic device. The image editing and generating method in the embodiment of the present invention can be applied to a variety of scenarios, such as image editing and generating, etc.

[0029] See also Figure 1 , an embodiment of the present invention provides an image editing and generating method, which is applicable to electronic equipment.

[0030] In the following method embodiments, for ease of description, the execution subject of each step of the method is taken as an electronic device as an example for illustration, but this does not constitute a specific limitation.

[0031] like Figure 1 As shown, the method may include the following steps:

[0032] Step 110, obtaining the image to be processed and the target instruction, and constructing an image description of the image to be processed according to the scene in the image to be processed.

[0033] In a possible implementation, an annotation tool is used to annotate the image to be processed to obtain a target annotation, and an image description is constructed according to the scene of the image to be processed and the target annotation.

[0034] Among them, the target annotation includes detection box and category, and the image description includes scene description, location, attributes and category.

[0035] Step 130, convert the image description into text tags, and embed the text tags into a high-dimensional space through an embedding model to obtain text embedding.

[0036] In one possible implementation, the image description is converted into text tags by a text tagger, the text tags are converted into corresponding embedding vectors by an embedding model, and all the embedding vectors are combined to obtain text embedding.

[0037] Among them, the embedding vector is used to represent the semantic relationship between text tags, and the text embedding is used to represent the overall semantic information of the image to be processed.

[0038] Step 150, obtain the latent vector of the image to be processed according to the image to be processed and the text embedding through the diffusion model, and design the editing operator according to the target instruction.

[0039] In one possible implementation, the diffusion model includes a U-net architecture generator and a text encoder, labeling the network layers of the cross-attention mechanism in the generator.

[0040] In a possible implementation, a real image is selected as a target image and a noise image according to the image to be processed, and the parameters of the diffusion model and the DDIM algorithm are configured. The diffusion model and the DDIM algorithm are used to perform inverse inference based on the real image and text embedding to obtain a latent vector.

[0041] Step 170: Use the editing operator to update the latent vector to obtain an updated latent vector, and generate a result image according to the updated latent vector through the diffusion model.

[0042] In one possible implementation, editing information is obtained according to a target instruction, an editing operator is designed according to the editing information, a denoised image is generated according to a latent vector through a diffusion model, the editing operator is used to calculate the editing loss in the denoised image generation process, and the latent vector is updated according to the editing loss through a back propagation algorithm until a set condition is met to obtain an updated latent vector.

[0043] Among them, the editing information includes editing type, editing features, editing parameters and editing loss calculation method. The editing loss is used to quantify the difference between the target instruction and the denoised image.

[0044] In one possible implementation, the result image is post-processed and quality assessed.

[0045] Among them, post-processing includes color correction and sharpening, and quality assessment includes subjective visual inspection and objective quantitative indicators.

[0046] Through the above process, the present invention first obtains the image to be processed and the target instruction, constructs the image description of the image to be processed according to the scene in the image to be processed, converts the image description into a text tag, embeds the text tag into a high-dimensional space through an embedding model to obtain a text embedding, obtains the latent vector of the image to be processed according to the image to be processed and the text embedding through a diffusion model, designs an editing operator according to the target instruction, updates the latent vector using the editing operator to obtain an updated latent vector, generates a result image according to the updated latent vector through a diffusion model, and realizes efficient, accurate and flexible editing of the image by combining image description, text embedding, diffusion model and editing operator, while improving the user interaction experience and the diversity of image generation, avoiding the overfitting phenomenon caused by single data in specific scenes, and at the same time, target editing based on real images can reduce the difference between synthetic and simulated data and real scenes, and can realize flexible editing and enhancement of real scene images, meet diverse image processing needs, and thus can effectively solve the problems that the prior art cannot improve the performance of the model on specific target categories, cannot solve the long-tail distribution problem, and cannot generate data close to the real scene.

[0047] In one application scenario, Figure 2 A flowchart of an image editing and generation method is shown.

[0048] like Figure 2 As shown, the image editing and generation method using a stable diffusion model may include the following steps:

[0049] Step S1: input and annotation.

[0050] Specifically, first read in a real image, which needs to be detected using traditional tools (such as image annotation software) to mark the bounding boxes and categories of the target area (such as people, objects, etc.).

[0051] Step S2, generating prompt words.

[0052] Specifically, a descriptive prompt word Prompt is generated based on the annotated image content and scene. This Prompt is used to subsequently guide the model to understand and generate image content similar to the original image.

[0053] Among them, prompt is a prompt word for instructing to generate a large model to generate an image, which can define the scene of generating the image, the target to be generated, the attributes of the target, and other descriptions.

[0054] Step S3: tokenization and embedding.

[0055] Specifically, Prompt is tokenized by Tokenizer and converted into a token sequence that the model can understand. Then, the embedding model is used to convert these token sequences into embedded vectors prompt_embeddings, which contain the semantic information of Prompt.

[0056] Step S4, model initialization.

[0057] Specifically, the Stable Diffusion model is initialized, with special attention paid to the cross-attention layers in the Unet module. The cross-attention layers in the Unet module in the model are marked as attn_features. These layers play a key role in the model, enabling the model to focus on specific cues and image content when generating images.

[0058] Step S5, Inversion process.

[0059] Specifically, the DDIM (Denoising Diffusion Implicit Model) method is used to perform Inversion processing on the real image and the Prompt embedding vector (prompt_embeds). The purpose of this process is to find an initial latent vector that can generate an image as similar as possible to the original image when decoded by the Stable Diffusion model.

[0060] Step S6, diffusion and editing.

[0061] Specifically, during the diffusion process, the model generates noise predictions (noise_pred) at each micro-stage (timesteps) through the Unet module. These predictions gradually approach the real image content. At this time, custom editing operators are introduced. These operators calculate the loss before and after editing for the features related to the target layer in attn_features. This loss is used to guide the model to consider editing requirements when updating the latent vector.

[0062] Step S7, update and iteration.

[0063] Specifically, according to the calculated loss, the noise_pred is updated through the back propagation algorithm, and then the latent vector is updated. This process is repeated in all diffusion stages (timestamps stages) until all iterations are completed.

[0064] Step S8, outputting the edited image.

[0065] Specifically, after completing all diffusion stages, the final latent vector is fed into the output module of the Stable Diffusion model and decoded into an edited image, which retains the basic information of the original image and is modified accordingly according to the guidance of the prompt and custom editing operator.

[0066] Furthermore, by designing other loss functions on the feature map channel, editing methods such as target contour, appearance, and multi-image target fusion can be completed.

[0067] Through the above steps, the embodiment of the present invention demonstrates how to edit and generate images through the Stable Diffusion model. By accurately controlling the update process of the latent vector, fine adjustment of the image content is achieved. The image targets are edited based on the generative diffusion model, and the number of targets with small data volumes can be supplemented to alleviate the long-tail distribution problem. The diversity of target data can be increased to avoid overfitting caused by single data in specific scenarios. At the same time, target editing based on real images avoids the gap between synthetic / simulated data and real scenes, which is more conducive to model training.

[0068] In another application scenario, the Stable Diffusion model is used to edit and generate real images, including image annotation, prompt generation, model initialization, latent vector acquisition, custom editing operator design, and final image editing. The specific steps include:

[0069] The first step is image annotation and prompt generation.

[0070] Image annotation: First, select a real scene image and use annotation tools (such as labelme) to annotate the target detection box (bboxes) and category (classes). For example, annotate the position and category of a red cup on the table.

[0071] Prompt generation: Generate a descriptive text prompt based on the image scene and target annotations. The format is "In the scene of {scene description}, there is {class} of {attribute set} at the {bbox} position, ...".

[0072] The second step is to initialize the Stable Diffusion model.

[0073] Initialize the Stable Diffusion model, which is a model based on the diffusion generation algorithm for gradually generating images from noise.

[0074] Find the cross attention related unet layers (attn_features) in the model, which are responsible for generating image content.

[0075] The third step is to embed Prompt and obtain Latent vector.

[0076] Convert prompt to tokenizer and use Embedding model to extract corresponding prompt_embeds.

[0077] Use the real image and prompt_embeds to perform Inversion using the DDIM algorithm in the Stable Diffusion model to obtain the initial latent vector that can generate the image.

[0078] Step 4: Customize the editing operator design.

[0079] Design custom editing operators to define the image editing type (such as position, size, etc.), editing features, editing parameters, and editing loss calculation method.

[0080] Position editing operator: The target is translated by specifying the distance (sx, sy) to be moved and calculating the loss.

[0081] Size editing operator: adjusts the target size by specifying the target size ratio (r) and calculating the loss.

[0082] Step 5: Edit the image using custom operators.

[0083] Using the initial latent vector and the custom editing operator, image editing is performed at each denoising stage (timestamp) of the Stable Diffusion model.

[0084] At each timestamp, unet inference is used to get the current denoised image (noise_pred_t).

[0085] For each custom edit operator, find the corresponding feature map (attn_feats) and calculate the loss before and after editing (edit_loss).

[0086] Use edit_loss to calculate the gradient (edit_grad) and update noise_pred_t and latent_t.

[0087] Repeat the above process until all timestamps are processed and the final latent vector is obtained.

[0088] Step 6: Generate the final edited image

[0089] The final latent vector is converted into an image in the Stable Diffusion model, which is the edited image.

[0090] Through the above process, the embodiment of the present invention realizes the precise editing and enhancement of real images by combining image annotation, text prompt, StableDiffusion model, custom editing operator and gradient optimization, while improving the user interaction experience and the diversity of image generation, avoiding the overfitting phenomenon caused by single data in specific scenarios, and at the same time, target editing based on real images can reduce the difference between synthetic and simulated data and real scenes, and can realize flexible editing and enhancement of real scene images to meet diverse image processing needs, thereby effectively solving the problems that the existing technology cannot improve the performance of the model on specific target categories, cannot solve the long-tail distribution problem, and cannot generate data close to the real scene.

[0091] The following is an embodiment of the device of the present invention, which can be used to perform the image editing and generating method involved in the present invention. For details not disclosed in the embodiment of the device of the present invention, please refer to the method embodiment of the image editing and generating method involved in the present invention.

[0092] See also Figure 3 , an image editing and generating device 800 is provided in an embodiment of the present invention.

[0093] The image editing and generating device 800 includes but is not limited to: an image processing module 810 , a text embedding module 830 , an image editing module 850 and an image generating module 870 .

[0094] The image processing module 810 is used to obtain the image to be processed and the target instruction, and construct an image description of the image to be processed according to the scene in the image to be processed.

[0095] The text embedding module 830 is used to convert the image description into text tags, and embed the text tags into a high-dimensional space through an embedding model to obtain text embedding.

[0096] The image editing module 850 is used to obtain the latent vector of the image to be processed according to the image to be processed and the text embedding through the diffusion model, and design the editing operator according to the target instruction.

[0097] The image generation module 870 is used to update the latent vector using the editing operator to obtain an updated latent vector, and generate a result image according to the updated latent vector through a diffusion model.

[0098] It should be noted that the image editing and generation provided in the above embodiments are only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the image editing and generation device will be divided into different functional modules to complete all or part of the functions described above.

[0099] In addition, the embodiments of the image editing and generating device and the image editing and generating method provided in the above embodiments belong to the same concept, and the specific manner in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0100] Figure 4 The present invention schematically illustrates the structure of an electronic device according to an exemplary embodiment.

[0101] It should be noted that the electronic device is only an example adapted to the present invention and cannot be considered to provide any limitation on the scope of use of the present invention. The electronic device cannot be interpreted as requiring dependence on or having Figure 4 One or more components of exemplary electronic device 2000 are shown.

[0102] The hardware structure of the electronic device 2000 may vary greatly due to differences in configuration or performance, such as Figure 4 As shown, the electronic device 2000 includes: a power supply 210 , an interface 230 , at least one memory 250 , and at least one central processing unit (CPU) 270 .

[0103] Specifically, the power supply 210 is used to provide operating voltage for each hardware device on the electronic device 2000 .

[0104] The interface 230 includes at least one wired or wireless network interface 231 for interacting with external devices. Of course, in other examples adapted by the present invention, the interface 230 may further include at least one serial-to-parallel conversion interface 233, at least one input-output interface 235, and at least one USB interface 237, etc. Figure 4 As shown, this is not intended to be a specific limitation.

[0105] The memory 250 is a carrier for storing resources, which may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 251, an application 253 and data 255, etc. The storage method may be temporary storage or permanent storage.

[0106] Among them, the operating system 251 is used to manage and control various hardware devices and application programs 253 on the electronic device 2000 to enable the central processor 270 to calculate and process the massive data 255 in the memory 250. It can be WindowsServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0107] The application program 253 is a computer-readable instruction that performs at least one specific task based on the operating system 251, and may include at least one module ( Figure 4 (not shown), each module may respectively contain computer-readable instructions for the electronic device 2000. For example, the image editing and generating device may be regarded as an application 253 deployed on the electronic device 2000.

[0108] The data 255 may be signal information, etc., stored in the memory 250 .

[0109] The central processor 270 may include one or more processors and is configured to communicate with the memory 250 through at least one communication bus to read the computer-readable instructions stored in the memory 250, thereby realizing the operation and processing of the mass data 255 in the memory 250. For example, the image editing and generation method is completed by the central processor 270 reading a series of computer-readable instructions stored in the memory 250.

[0110] In addition, the present invention can also be implemented by hardware circuits or hardware circuits combined with software. Therefore, the implementation of the present invention is not limited to any specific hardware circuits, software, or a combination of the two.

[0111] See also Figure 5 In an embodiment of the present invention, an electronic device 4000 is provided. The electronic device 4000 may include: a desktop computer, a laptop computer, a server, etc. with sensor recognition capability.

[0112] exist Figure 5 In the embodiment, the electronic device 4000 includes at least one processor 4001 and at least one memory 4003.

[0113] The data interaction between the processor 4001 and the memory 4003 can be realized through at least one communication bus 4002. The communication bus 4002 may include a path for transmitting data between the processor 4001 and the memory 4003. The communication bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The communication bus 4002 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0114] Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception, etc. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0115] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0116] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program instructions or codes in the form of instructions or data structures and can be accessed by the electronic device 4000, but is not limited to this.

[0117] Computer-readable instructions are stored in the memory 4003 , and the processor 4001 can read the computer-readable instructions stored in the memory 4003 through the communication bus 4002 .

[0118] The computer-readable instructions are executed by one or more processors 4001 to implement the image editing and generating methods in the above-mentioned embodiments.

[0119] In addition, an embodiment of the present invention provides a storage medium on which computer-readable instructions are stored. The computer-readable instructions are executed by one or more processors to implement the image editing and generating method as described above.

[0120] A computer program product is provided in an embodiment of the present invention. The computer program product includes computer-readable instructions. The computer-readable instructions are stored in a storage medium. One or more processors of an electronic device read the computer-readable instructions from the storage medium, load and execute the computer-readable instructions, so that the electronic device implements the image editing and generation method as described above.

[0121] Compared with the related art, the present invention has the following beneficial effects:

[0122] 1. The present invention first obtains an image to be processed and a target instruction, constructs an image description of the image to be processed according to the scene in the image to be processed, converts the image description into a text tag, embeds the text tag into a high-dimensional space through an embedding model to obtain a text embedding, obtains a latent vector of the image to be processed according to the image to be processed and the text embedding through a diffusion model, designs an editing operator according to the target instruction, updates the latent vector using the editing operator to obtain an updated latent vector, generates a result image according to the updated latent vector through a diffusion model, and realizes efficient, accurate and flexible editing of images by combining image description, text embedding, diffusion model and editing operator, while improving the user interaction experience and the diversity of image generation, avoiding the overfitting phenomenon caused by single data in specific scenes, and performing target editing based on real images can reduce the difference between synthetic and simulated data and real scenes, and can realize flexible editing and enhancement of real scene images, meet diverse image processing needs, and thus can effectively solve the problems that the prior art cannot improve the performance of the model on specific target categories, cannot solve the long-tail distribution problem, and cannot generate data close to the real scene.

[0123] 2. The present invention is highly flexible and accurate: through custom editing operators, users can edit objects in images very flexibly, including adjusting characteristics such as position and size. This editing method is not only accurate, but also can maintain the overall structure and details of the image, avoiding distortion or damage that may be caused by traditional image editing methods.

[0124] 3. The present invention has a strong semantic understanding capability: by using the crossattention mechanism of the Stable Diffusion model, the solution can deeply understand the semantic information in the image and edit the image in accordance with human intentions according to the guidance of the text prompt and the editing operator. This capability makes the editing results more natural and in line with expectations.

[0125] 4. The present invention can improve editing efficiency: By editing on latent vectors in high-dimensional space instead of directly operating on the pixel level, the solution significantly reduces the amount of calculation and resource requirements. At the same time, the application of fast inversion algorithms such as DDIM also further improves the efficiency of the editing process.

[0126] 5. The present invention has diversified editing effects: Since the Stable Diffusion model has a powerful generation capability, the solution can generate diversified editing effects. Even for the same original image and editing instructions, multiple different but satisfactory editing results may be generated, thus satisfying the diverse needs of users for image editing.

[0127] 6. The present invention can lower the technical threshold: through the intuitive text prompt and custom editing operator, the technical threshold of image editing is lowered. Users do not need to have professional image editing skills, and can achieve complex image editing effects by simply providing simple text descriptions and editing instructions. This ease of use makes the solution more popular and practical.

[0128] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0129] The above descriptions are only some embodiments of the present invention. It should be pointed out that, for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for editing and generating an image, characterized in that: The method comprises: Acquire an image to be processed and a target instruction, and construct an image description of the image to be processed according to a scene in the image to be processed; Converting the image description into text tags, and embedding the text tags into a high-dimensional space through an embedding model to obtain text embedding; Obtaining a content vector of the image to be processed according to the image to be processed and text embedding by a diffusion model, and designing an editing operator according to the target instruction; The editing operator is used to update the latent vector to obtain an updated latent vector, and a result image is generated according to the updated latent vector through a diffusion model.

2. The image editing and generating method according to claim 1, characterized in that: The step of constructing an image description of the image to be processed according to the information and scene in the image to be processed includes: Annotate the image to be processed using an annotation tool to obtain a target annotation; the target annotation includes a detection frame and a category; An image description is constructed according to the scene and target annotations of the image to be processed; the image description includes scene description, location, attributes and category.

3. The image editing and generating method according to claim 1, characterized in that: The step of converting the image description into a text tag and embedding the text tag into a high-dimensional space through an embedding model to obtain text embedding includes: converting the image description into text tags by a text tagger; The text tags are converted into corresponding embedding vectors through an embedding model; the embedding vectors are used to represent the semantic relationship between the text tags; All the embedding vectors are combined to obtain text embedding; the text embedding is used to represent the overall semantic information of the image to be processed.

4. The image editing and generating method according to claim 1, characterized in that: The diffusion model includes a generator with a U-net architecture and a text encoder, and the network layers of the cross-attention mechanism in the generator are marked.

5. The image editing and generating method according to claim 1, characterized in that: The step of obtaining a latent vector of the image to be processed according to the image to be processed and text embedding by using a diffusion model includes: Selecting a real image as a target image and a noise image according to the image to be processed, and configuring parameters of a diffusion model and a DDIM algorithm; The diffusion model and the DDIM algorithm are used to perform inverse inference based on the real image and the text embedding to obtain a content vector.

6. The image editing and generating method according to claim 1, characterized in that: The step of designing an editing operator according to the target instruction and using the editing operator to update the l atent vector to obtain an updated l atent vector includes: Obtaining editing information according to the target instruction, and designing an editing operator according to the editing information; the editing information includes editing type, editing feature, editing parameter and editing loss calculation method; Generate a denoised image according to the light vector by using the diffusion model, and calculate the editing loss in the process of generating the denoised image by using the editing operator; the editing loss is used to quantify the difference between the target instruction and the denoised image; The l atent vector is updated according to the editing loss through a back propagation algorithm until a set condition is met to obtain an updated l atent vector.

7. The image editing and generating method according to claim 1, characterized in that: After generating the result image, the method further includes: The result image is post-processed and quality assessed; the post-processing includes color correction and sharpening; the quality assessment includes subjective visual inspection and objective quantitative indicators.

8. An image editing and generating device, characterized in that: The device comprises: An image processing module is used to obtain an image to be processed and a target instruction, and construct an image description of the image to be processed according to a scene in the image to be processed; The text embedding module is used to convert image descriptions into text tags and embed the text tags into a high-dimensional space through an embedding model to obtain text embedding; The image editing module is used to obtain the latent vector of the image to be processed according to the image to be processed and the text embedding through the diffusion model, and design the editing operator according to the target instruction; The image generation module is used to update the latent vector using the editing operator to obtain the updated latent vector, and generate the result image according to the updated latent vector through the diffusion model.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein: The memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the image editing and generating method according to any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by one or more processors to implement the image editing and generating method according to any one of claims 1 to 7.

Images