Image editing method and device, equipment, storage medium and product

Abstract

The invention provides an image editing method and device, equipment, a storage medium and a product. The method comprises the steps of obtaining a to-be-edited image and an instruction type editing text; based on the instruction type editing text, carrying out feature annotation on the image features of the to-be-edited image to obtain at least one to-be-edited region identifier of the to-be-edited image; based on the at least one to-be-edited region identifier, extracting partial image features corresponding to each to-be-edited region identifier from the image features of the to-be-edited image; determining a to-be-edited region corresponding to the instruction type editing text from the to-be-edited image according to the partial image features corresponding to each to-be-edited region identifier; and performing image editing on the to-be-edited area of the to-be-edited image according to the instruction type editing text to obtain an edited image. According to the method, the to-be-edited region of the to-be-edited image can be accurately positioned and edited according to the instruction type editing text, so that the to-be-edited image is accurately and controllably edited, and the efficiency and accuracy of image editing are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an image editing method and device, equipment, storage medium and product. BACKGROUND

[0002] With the continuous progress and popularization of digital technology, image editing has become an indispensable part of modern society. From professional photographers to ordinary consumers, people can easily edit and modify images to meet different needs and aesthetics. At the same time, with the development of artificial intelligence and machine learning technology, advanced technologies have been introduced into the field of image editing to achieve more advanced and intelligent editing functions. For example, through deep learning algorithms, image editing software can automatically identify objects, scenes and colors in images and intelligently adjust and optimize them according to user needs.

[0003] In related technologies, image editing methods mainly include manual editing and automatic generation based on deep learning algorithms. Manual editing of image editing methods relies on manual operation, which is inefficient and difficult to ensure the consistency of editing quality. Image editing methods based on deep learning algorithms lack a deep understanding of the actual scene space based on user instructions, and cannot accurately edit images according to user instructions. For complex image processing tasks, related technologies often fail to meet the requirements. That is, related technologies have technical problems such as low efficiency and poor accuracy when editing images. SUMMARY

[0004] The embodiments of the present application provide an image editing method, device, equipment, storage medium and product, which can accurately position and edit the to-be-edited region of an image according to an instruction-type editing text, realize intelligent, accurate and controllable editing of the image, and improve the efficiency and accuracy of image editing.

[0005] The technical solutions of the embodiments of the present application are as follows:

[0006] The embodiments of the present application provide an image editing method, which comprises: obtaining a to-be-edited image and an instruction-type editing text; based on the instruction-type editing text, performing feature annotation on image features of the to-be-edited image to obtain at least one to-be-edited region identifier of the to-be-edited image; based on the at least one to-be-edited region identifier, extracting part of the image features corresponding to each to-be-edited region identifier from the image features of the to-be-edited image; determining a to-be-edited region corresponding to the instruction-type editing text from the to-be-edited image according to the part of the image features corresponding to each to-be-edited region identifier; and performing image editing on the to-be-edited region of the to-be-edited image according to the instruction-type editing text to obtain an edited image.

[0007] The embodiment of the application provides an image editing device, comprising: a data acquisition module configured to acquire an image to be edited and an instruction type editing text; a mark determination module configured to perform feature annotation on image features of the image to be edited based on the instruction type editing text, to obtain at least one to-be-edited region identifier of the image to be edited; a feature determination module configured to extract part of image features corresponding to each to-be-edited region identifier from the image features of the image to be edited based on the at least one to-be-edited region identifier; a region determination module configured to determine a to-be-edited region corresponding to the instruction type editing text from the image to be edited according to the part of image features corresponding to each to-be-edited region identifier; and an image editing module configured to perform image editing on the to-be-edited region of the image to be edited according to the instruction type editing text, to obtain an edited image.

[0008] In the above scheme, the mark determination module is further configured to: perform text feature extraction on the instruction type editing text to obtain editing text features; perform image feature extraction on the image to be edited to obtain image features of the image to be edited; and perform feature annotation on the editing text features and the image features of the image to be edited by using a pre-trained multi-modal large language model, to obtain at least one to-be-edited region identifier of the image to be edited.

[0009] In the above scheme, the device further comprises a multi-modal large language model determination module configured to acquire a feature annotation sample set; wherein the feature annotation sample set comprises training text features, training image features, and training to-be-edited region identifiers corresponding to the training text features and the training image features; a low-rank module is embedded into a pre-trained large language model to obtain a low-rank multi-modal large language model; wherein the low-rank module comprises at least one low-rank matrix; and the low-rank multi-modal large language model is trained based on the feature annotation sample set to obtain the pre-trained multi-modal large language model.

[0010] In the above scheme, the multi-modal large language model determination module is further configured to: input the training text features and the training image features into the low-rank multi-modal large language model to obtain predicted to-be-edited region identifiers; determine a loss function value of the low-rank multi-modal large language model based on the predicted to-be-edited region identifiers and the training to-be-edited region identifiers; if it is determined based on the loss function value that a preset training stop condition is not reached at present, update parameters of the low-rank module in the low-rank multi-modal large language model to obtain an updated low-rank multi-modal large language model; and if it is determined based on the loss function value that the preset training stop condition is reached at present, determine the current updated low-rank multi-modal large language model as the pre-trained multi-modal large language model.

[0011] In the above scheme, the image editing module is further configured to: generate a mask for the to-be-edited image based on the to-be-edited region, to obtain an image editing mask; wherein the image editing mask is used to indicate the to-be-edited region in the to-be-edited image; input the instruction-type editing text, the to-be-edited image, and the image editing mask into an image diffusion model, and generate an edited image by using the image diffusion model to generate an image for the to-be-edited region of the to-be-edited image.

[0012] In the above scheme, the image editing module is further configured to: encode the instruction-type editing text by using a text encoder of the image diffusion model, to obtain instruction text features corresponding to the instruction-type editing text; encode the to-be-edited image by using an image encoder of the image diffusion model, to obtain hidden space features corresponding to the to-be-edited image; adjust the size of the image editing mask to a standard size of the hidden space features, to obtain a region space feature corresponding to the standard size; and decode the to-be-edited image based on the instruction text features, the hidden space features, and the region space feature, to obtain the edited image.

[0013] In the above scheme, the device further includes a self-generated sample set generation module configured to obtain a data generated sample set; wherein the data generated sample set includes an original image, an instruction-type training text, and an image training mask; analyze the original image by using an image analysis model to obtain an original image description text; determine a result image description text based on the original image description text and the instruction-type training text; generate a result image based on the original image, the image training mask, and the result image description text; determine the original image, the image training mask, and the instruction-type training text as feature data, and determine the result image as label data, to obtain a self-generated sample set; wherein any one group of the feature data in the self-generated sample set corresponds to a plurality of the label data; and train an initial diffusion model based on the self-generated sample set, to obtain the image diffusion model.

[0014] In the above scheme, the self-generated sample set generation module is further configured to input the original image into the image analysis model, extract image features of the original image by using a feature extraction module of the image analysis model, and obtain original image features; and analyze the original image features by using a feature analysis module of the image analysis model, to obtain the original image description text.

[0015] In the scheme, the self-generated sample set generation module is further configured to recombine the original image description text and the instruction type training text according to a preset text format to obtain a large language model input text; input the large language model input text into a general large language model to generate an output text to obtain the result image description text.

[0016] In the scheme, the self-generated sample set generation module is further configured to set an initial prompt for the general large language model to obtain a large language model with a semantic conversion function; the initial prompt is a semantic conversion prompt; input the large language model input text into the large language model with the semantic conversion function; and generate an output text for the large language model input text by the large language model with the semantic conversion function to obtain the result image description text.

[0017] In the scheme, the data acquisition module is further configured to acquire an instruction type editing voice; perform voice recognition on the instruction type editing voice to obtain an original editing text; and perform text revision on the original editing text by a large language model with a text revision function to obtain the instruction type editing text; wherein the large language model with the text revision function is a large language model obtained by training a general large language model with an initial prompt in which a text revision prompt is preset.

[0018] An electronic device is provided in an embodiment of the present application, and the electronic device includes a memory configured to store computer executable instructions, and a processor configured to execute the computer executable instructions stored in the memory to implement an image editing method provided in an embodiment of the present application.

[0019] A computer readable storage medium is provided in an embodiment of the present application, and the computer readable storage medium stores a computer program or computer executable instructions, and is configured to implement an image editing method provided in an embodiment of the present application when executed by a processor.

[0020] A computer program product is provided in an embodiment of the present application, and the computer program product includes a computer program or computer executable instructions, and is configured to implement an image editing method provided in an embodiment of the present application when executed by a processor.

[0021] An embodiment of the present application has the following beneficial effects:

[0022] The present application can realize accurate editing of an image by inputting concise editing instructions through the introduction of instruction type editing text. This instruction type editing method not only improves editing efficiency, but also reduces the editing threshold, so that users can operate image editing without complex training or learning, making the image editing process more intelligent and flexible. By annotating the image features of the image to be edited, the identification of the to-be-edited region is generated, and the corresponding part of the image feature is extracted based on the identification of the to-be-edited region, realizing the deep understanding and analysis of the image content, so that the to-be-edited region corresponding to the instruction type editing text can be more accurately determined. By performing specific editing operations on the to-be-edited region according to the instruction type editing text, controllable editing of the image is realized, so that users can modify and adjust the specific region of the image according to their own needs, thereby obtaining a more expected editing result. In summary, the present application accurately positions and edits the to-be-edited region of the image according to the instruction type editing text, realizes intelligent, accurate and controllable editing of the image, improves the efficiency and accuracy of image editing, and provides users with more convenient, efficient and personalized image editing experience. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a structural schematic diagram of an image editing system provided by an embodiment of the present application;

[0024] Figure 2 is a structural schematic diagram of an electronic device provided by an embodiment of the present application;

[0025] Figure 3A is a flowchart of an image editing method provided by an embodiment of the present application;

[0026] Figure 3B is a flowchart of an image editing method provided by an embodiment of the present application;

[0027] Figure 3C is another flowchart of an image editing method provided by an embodiment of the present application;

[0028] Figure 3D is another flowchart of an image editing method provided by an embodiment of the present application;

[0029] Figure 3E is another flowchart of an image editing method provided by an embodiment of the present application;

[0030] Figure 3F is a flowchart of determining a result image description text provided by an embodiment of the present application;

[0031] Figure 3Gis a flowchart of a process for determining a result image description text provided by an embodiment of the present application;

[0032] Figure 4 is another flowchart of an image editing method provided by an embodiment of the present application;

[0033] Figure 5 is a flowchart of a process for generating a result image description text provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those skilled in the art without making creative efforts fall within the scope of protection of the present application.

[0035] In the following description, "some embodiments" are described, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0036] If similar descriptions of "first / second" appear in the application file, the following description is added, in the following description, the terms "first / second / third" are only to distinguish similar objects, and do not represent a specific order of the objects, and it can be understood that "first / second / third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0037] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0038] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application, and are not intended to limit the present application.

[0039] The relevant data collection and processing in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0040] Before further detailing the embodiments of the present application, the terms and phrases involved in the embodiments of the present application are explained, and the terms and phrases involved in the embodiments of the present application are applicable to the following explanations.

[0041] 1) Diffusion Model, a deep learning framework for generating images, audio, and other types of data with high realism. Diffusion Model is a generative model, unlike traditional Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs), which generate new data samples by simulating the diffusion and denoising process of data distribution.

[0042] 2) Mask, also known as mask, is mainly used to define or mark specific areas or pixel sets in an image. Mask is usually a binary image, whose pixel values are used to distinguish different areas in the image. Specifically, the part with pixel value 0 usually represents the area that is not concerned, while the non-0 value (such as 255) represents the area of interest.

[0043] 3) Low-Rank Adaptation (LoRA), a technique used in the field of machine learning, especially in fine-tuning pre-trained models. Low-Rank Adaptation aims to reduce the number of parameters that need to be adjusted during fine-tuning, thereby reducing computational resources and storage requirements while maintaining model performance. In Low-Rank Adaptation, a low-rank module is added to certain layers of the pre-trained model, such as attention mechanisms or feedforward network layers, so that only a small number of parameters are updated during fine-tuning, rather than the entire model.

[0044] 4) Low-Rank Module, a component used in the Low-Rank Adaptation (LoRA) technique, is a special structure added to the pre-trained model to reduce the number of parameters that need to be updated during fine-tuning. The low-rank module is usually composed of one or more low-rank matrices, whose rank is much smaller than the number of rows or columns of the matrix (if the rank of a matrix is less than 10% of the number of rows or columns of the matrix, then the matrix is a "low-rank matrix"), so it contains fewer non-zero elements, greatly reducing the number of parameters. Through the low-rank module, large pre-trained models can be customized and adapted quickly and efficiently without sacrificing too much performance.

[0045] 5) Pre-training, refers to the initial training of a model on a large dataset so that the model can learn a wide range of knowledge and feature representations. Pre-training is usually done on a relatively simple task, with the goal of allowing the model to learn enough general knowledge and patterns to perform well on multiple related tasks.

[0046] 6) Fine-tuning, is a process in machine learning, particularly in deep learning models, that involves using a pre-trained model on a large dataset and further training the model on a small dataset specific to a particular task. In this application, fine-tuning generally refers to adjusting and optimizing a pre-trained model for image segmentation tasks.

[0047] 7) Large Language Models (LLM), refers to a class of natural language processing models based on deep learning. Large language models learn statistical features and patterns of language by training on large amounts of text data, allowing them to understand and generate natural language text. Large language models usually have tens of billions to hundreds of billions of parameters, capable of capturing complex relationships and patterns in language text.

[0048] 8) Multimodal Large Language Models (M-LLM), refers to a class of large language models that can handle and understand multiple types of data such as text, images, sound, etc. Multimodal large language models not only understand and generate text, but also understand and generate other types of data, or associate and convert different types of data.

[0049] 9) Prompt, refers to a piece of text input to a large language model, used to guide the large language model to generate corresponding output. The prompt is usually used to stimulate the expression of the large language model on a specific topic or task, and the prompt is a way for humans to interact with the large language model, which can be regarded as a "prompt" or "question" to the large language model.

[0050] 8) Hidden Space Features, refers to the features extracted by a neural network model during the training process through a non-linear transformation from the original input data. These features are usually located in the hidden layers of the network, hence the name. Hidden space features are a manifestation of the internal state during the model learning process and are the result of the model's processing of input data. Hidden space features are internal representations extracted by deep learning models during the training process from the original input data, reflecting the internal structure and patterns of the data. For example, in natural language processing tasks, hidden space features can capture the semantic relationships between words, helping the model perform tasks such as text classification, sentiment analysis, machine translation, etc. In computer vision tasks, hidden space features can capture key features and patterns in images, helping the model perform tasks such as image classification, object detection, image segmentation, etc.

[0051] 9) Object Detection, is an important technology in the field of computer vision, aiming to find all objects of interest in an image and determine the category and location of all objects of interest. The object detection task usually includes two main steps: object localization and object classification. In the object localization stage, the algorithm determines the location and size of the target object in the image, which is usually achieved through a bounding box or a pixel-level mask. Subsequently, in the object classification stage, the algorithm classifies the located objects to determine which category of objects they belong to.

[0052] 10) Image Segmentation, is to divide an image into several specific regions with unique properties and propose objects of interest. Image segmentation is a key step from image processing to image analysis, its main purpose is to classify pixels in an image, and pixels belonging to the same region or the same object are classified into a class, and different classes of pixels are assigned corresponding labels.

[0053] 11) Loss Function Value, is a measure of the difference between the model's predicted value and the actual value in machine learning and deep learning. The loss function value is a non-negative real number, the smaller the loss function value, the closer the model's prediction result to the true result, that is, the better the model's performance. During the training process, the goal is to minimize the value of the loss function by adjusting the model parameters, so that the model can learn the mapping relationship from input to output. The loss function is a "guide" for model learning, indicating the goodness of the model's prediction result, and is a core part of the empirical risk function and an important part of the structural risk function.

[0054] 12) Labelled Data, refers to the data in which each sample (such as an image, a piece of text, or a transaction record) has a corresponding label or target value. This label or target value is defined in advance, and is usually the result that the model is expected to predict. For example, in the task of image recognition, the image and its corresponding category (such as "cat", "dog", etc.) are label data; in the task of email classification, the email and its corresponding classification (such as "spam", "non-spam") are also label data. Label data is the basis of supervised learning, because supervised learning algorithms need to use these labels to train the model, so that it can make predictions on unlabeled data.

[0055] 13) Feature Data, refers to the data in the data set that describes all the information of each sample. In machine learning, feature data is the input variable used to predict the label. Features are the attributes extracted from the data that are meaningful to the model, and features can represent the key information in the data. For example, in a house price prediction model, the feature data may include the area of the house, the number of rooms, the construction year, etc.; in a recommendation system, the feature data may include the user's historical purchase records, browsing history, ratings, etc. Selecting and constructing effective features is crucial to the performance of the model.

[0056] In related technologies, image editing methods mainly include manual editing and automatic generation based on deep learning algorithms. The image editing method of manual editing relies on manual operation, which is inefficient and difficult to ensure the consistency of editing quality; the image editing method based on deep learning algorithm lacks deep understanding of the actual scene space based on user instructions, and cannot accurately edit images according to user instructions. For complex image processing tasks, related technologies are often difficult to perform. That is, related technologies have technical problems such as low efficiency and poor accuracy when editing images.

[0057] To solve at least one of the above problems existing in the related art, embodiments of the present application provide an image editing method. By introducing instruction-type editing text, a user can input concise editing instructions to achieve accurate editing of an image. This instruction-type editing method not only improves editing efficiency, but also reduces the editing threshold, so that the user can operate the image editing without complex training or learning, making the image editing process more intelligent and flexible. By performing feature annotation on the image features of the image to be edited, the image feature to be edited is generated, and based on the image feature to be edited, the corresponding part of the image feature is extracted, realizing deep understanding and analysis of the image content, so that the image editing area corresponding to the instruction-type editing text can be more accurately determined. By performing specific editing operations on the image editing area according to the instruction-type editing text, controllable editing of the image is realized, so that the user can finely modify and adjust the specific area of the image according to his own needs, thereby obtaining a more expected editing result. In summary, the present application accurately positions and edits the image editing area according to the instruction-type editing text, realizes intelligent, accurate and controllable editing of the image, improves the efficiency and accuracy of image editing, and provides a more convenient, efficient and personalized image editing experience for the user.

[0058] Before explaining the image editing method of the embodiments of the present application, here, first, an exemplary application of the image editing device of the embodiments of the present application is described, which is an electronic device for implementing the image editing method. In an implementation manner, the image editing device (i.e. electronic device) provided by the embodiments of the present application can be implemented as a server. The server can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. Multiple servers can be directly or indirectly connected through wired or wireless communication, which is not limited in the embodiments of the present application. The electronic device provided by the embodiments of the present application can also be implemented as a terminal device. The terminal device can be a notebook computer, a tablet computer, a desktop computer, a smart phone, a smart watch, a smart television, a vehicle-mounted terminal, and various types of terminals. The electronic device provided by the embodiments of the present application can also be implemented as a combination of a terminal device and a server.

[0059] Referring to Figure 1 , Figure 1This is a schematic diagram of the architecture of the image editing system 10 provided in this application embodiment. The terminal 100 and the server 200 can connect and interact via a network 300. The terminal 100 can be used to provide a user interface, receive user-instructed editing text, and present the image editing results to the user. The server 200 can be used for training and applying various models in this application. The network 300 can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0060] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device 400 provided in the embodiments of this application. Figure 2 The illustrated electronic device 400 includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0061] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0064] The memory 450 includes volatile memory or nonvolatile memory, and can include both volatile and nonvolatile memory. The nonvolatile memory can be read only memory (ROM), and the volatile memory can be random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, which are exemplarily illustrated below.

[0066] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.

[0067] The network communication module 452 is used to communicate with other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including Bluetooth, wireless compatibility authentication (WiFi), and universal serial bus (USB), and the like.

[0068] The presentation module 453 is used to enable the presentation of information via one or more output devices 431 associated with the user interface 430 (e.g., a display screen, a speaker, and the like) (e.g., a user interface for operating peripheral devices and displaying content and information).

[0069] The input processing module 454 is used to detect and interpret one or more user inputs or interactions from one or more input devices 432.

[0070] In some embodiments, the device provided by the embodiments of the present application can be implemented in software, Figure 2 An image editing device 455 stored in the memory 450 is shown, which can be software in the form of programs and plug-ins, including the following software modules: a data acquisition module 4551, a label determination module 4552, a feature determination module 4553, a region determination module 4554, and an image editing module 4555, which are logical, and thus can be combined or further split according to the implemented functions. The functions of each module will be described below.

[0071] In some embodiments, the apparatus provided by the embodiments of the present application can be implemented in a hardware manner. For example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the image editing method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can be implemented by using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic elements.

[0072] Referring to Figure 3A , Figure 3A is an optional flowchart of the image editing method provided by the embodiments of the present application. The following will be described with reference to the steps shown in Figure 3A . For example, the execution subject of the image editing method is an electronic device, as shown in Figure 3A . The method includes the following steps 101 to 105.

[0073] Step 101: Obtain an image to be edited and an instruction type editing text.

[0074] In some embodiments, the image to be edited is an original image selected by a user and ready to be modified or enhanced. The image to be edited can have any format, such as JPEG, PNG, or RAW, etc. The image to be edited can come from various sources, such as a digital camera, a mobile phone camera, a scanner, or a network resource, etc. The image to be edited can be obtained by receiving a file uploaded by a user, reading from a storage device, or obtaining from another service through an interface, etc. The content of the image to be edited can be a landscape, a portrait, a still life, or an illustration, etc. The embodiments of the present application do not limit this.

[0075] The instruction-type editing text is a text instruction provided by a user to guide an image editing process; the instruction-type editing text is expressed in natural language, and the semantics of the instruction-type editing text explicitly indicate a specific editing task that the user wants to perform on the to-be-edited image. For example, the instruction-type editing text can be "replace the cat in front of the sofa with a dog", "there is a house beside the mountain", "change the sunny day into a cloudy day", "change the color of the vehicle tire to red", or "there is a pond beside the cow", etc. The role of the instruction-type editing text is to inform the subject performing the method of the present embodiment how to process the to-be-edited image to achieve the editing effect expected by the user.

[0076] For example, in an application scenario of an image editing software, a user can select to open a local image as a to-be-edited image, and input a text description "replace the sky with green" in the input box of the image editing software as the instruction-type editing text.

[0077] Through step 101, a necessary data basis can be provided for the subsequent image editing process, so that the user can edit the image through the text description, and the flexibility and convenience of image editing are improved.

[0078] In some embodiments, the process of obtaining the instruction-type editing text in step 101 can be implemented in the following manner: first, an instruction-type editing voice is obtained; then, the instruction-type editing voice is subjected to voice recognition to obtain an original editing text; finally, the original editing text is subjected to text revision by a large language model with a text revision function to obtain the instruction-type editing text; wherein the large language model with the text revision function is obtained by training a general large language model after presetting a text revision prompt as an initial prompt of the general large language model.

[0079] The instruction-type editing voice is a voice instruction issued by a user to guide an image editing process, and the instruction-type editing voice is a verbal description of the image editing operation required by the user. Voice recognition is a technology for converting human voice signals into computer-readable text, and the voice recognition process involves using machine learning algorithms, especially deep learning models such as recurrent neural networks or convolutional neural networks, to analyze and interpret the characteristics of voice waveforms. These deep learning models, after training, can recognize different voice units (such as phonemes), words, and phrases, and convert the recognition results into corresponding text representations, i.e., the original editing text. The original editing text is a text output generated in the voice recognition process. Due to the diversity and complexity of voice, as well as the influence of environmental noise, the original text may contain errors such as spelling errors, grammatical errors, unclear semantics, or redundant information, which are caused by unclear pronunciation, dialect, too fast speech, unclear logic, or background noise, etc. The general large language model can understand and generate natural language text, but does not have the ability to deeply understand specific fields.

[0080] The text revision prompt is a specific prompt for guiding a general large language model to perform a text revision task. The text revision prompt can be input in natural language text, instructing the general large language model to focus on correcting specific language errors, improving the fluency of the text, or ensuring that the text meets specific formatting requirements. By setting the text revision prompt as the initial prompt for the general large language model, the general large language model can be equipped with the function or ability of text revision, resulting in a large language model with text revision function. The initial prompt is an instruction provided to the large language model before processing a task. In the embodiments of the present application, the initial prompt tells the large language model the expected revision type or standard, thereby guiding the large language model to generate more accurate and user-demand-compliant revised text.

[0081] By combining speech recognition and a large language model with text revision function, the user's voice editing intention can be accurately captured and converted into accurate instruction-type editing text, providing a more convenient and efficient image editing method for the user, and improving the accuracy and reliability of image editing operations.

[0082] In step 102, based on the instruction-type editing text, image features of the image to be edited are annotated to obtain at least one to-be-edited region identifier of the image to be edited.

[0083] Feature annotation refers to the process of labeling or annotating specific attributes or content of the image to be edited. In the feature annotation process, the image features of the image to be edited can be analyzed, and the relevant regions or objects in the image to be edited can be identified and labeled according to the information provided in the instruction-type editing text. These image features can be color, shape, texture, size, and position, or any other visual information related to the editing task.

[0084] The to-be-edited region identifier is generated by feature annotation and is used to indicate which regions or objects in the image to be edited. These to-be-edited region identifiers are derived based on the understanding of the instruction-type editing text and the analysis of the image content. The to-be-edited region identifier can be a specific pixel, object contour, region division, or any other form of target indication in the image.

[0085] In some embodiments, step 102 can be implemented by: performing text feature extraction on the instruction-type editing text to obtain editing text features; performing image feature extraction on the image to be edited to obtain image features of the image to be edited; and using a pre-trained multi-modal large language model to perform feature annotation on the editing text features and the image features of the image to be edited to obtain at least one to-be-edited region identifier of the image to be edited.

[0086] In some embodiments, the implementation process of step 102 can be divided into three main links: first, text feature extraction is performed on the instruction type editing text, key words, phrases or sentence structures in the text are identified using natural language processing technology, and editing text features are formed. The editing text features reflect the editing intention and specific requirements of the user. Second, image feature extraction is performed on the image to be edited, and visual features such as color, shape and texture in the image are extracted using computer vision algorithms to form image features. These image features describe the appearance and content of the image. Finally, the editing text features and image features are fused and associated through a pre-trained multi-modal large language model to generate feature annotations. The multi-modal large language model can understand and process data in two different modalities of text and image, learn the corresponding relationship between text and image through deep learning, and thus generate the to-be-edited region identifier.

[0087] For example, in the application scenario of image editing, the user may require "replace the sky in the image with green" through the instruction type editing text. In this example, the editing text features can extract the two key information of "sky" and "green"; at the same time, the image feature extraction identifies the sky region in the image and its current color; through the multi-modal large language model, these features are associated and reasoned to generate a to-be-edited region identifier, indicating that the region identified as the sky in the image is replaced with green.

[0088] Through step 102, in-depth understanding and analysis of the instruction type editing text and the image to be edited can be achieved. Through the application of the multi-modal large language model, data in two modalities of text and image can be processed simultaneously, improving the accuracy and flexibility of the editing operation; through the generation of feature annotations and to-be-edited region identifiers, the embodiments of the present application can more accurately locate the region in the image to be edited. This provides the user with a more intelligent and efficient image editing experience.

[0089] Step 103, based on at least one to-be-edited region identifier, extracting part of the image features corresponding to each to-be-edited region identifier from the image features of the image to be edited.

[0090] In some embodiments, according to the editing intention and position information indicated by the to-be-edited region identifier, the image features can be filtered and matched through target detection or image segmentation, etc., to ensure that the extracted features match the requirements of the to-be-edited region identifier, so as to locate the part of the image features to be extracted in the image to be edited.

[0091] For example, in the application scenario of character image editing, if the to-be-edited region identifier indicates that a certain character in the image is to be replaced, step 103 needs to extract part of the image features related to the character from the image features of the to-be-edited image, such as the contour, posture, color, and other feature information of the character. Through accurate feature extraction, it can be ensured that the subsequent editing operation can be accurately applied to the target character, realizing the replacement effect.

[0092] Through step 103, the accurate application of the to-be-edited region identifier can be ensured, improving the accuracy and efficiency of image editing. By extracting part of the image features corresponding to the to-be-edited region identifier, the target region in the to-be-edited image can be more accurately located, and strong support can be provided for subsequent editing operations.

[0093] Step 104: determining the to-be-edited region corresponding to the instruction-type editing text from the to-be-edited image according to part of the image features corresponding to each to-be-edited region identifier.

[0094] In some embodiments, image feature extraction can be performed on part of the image features corresponding to each to-be-edited region identifier, and the extracted image features are matched with the semantics of the instruction-type editing text, and part of the image features that match the instruction-type editing text to a greater extent than a certain preset degree are selected, to ensure the accuracy and integrity of the to-be-edited region. The to-be-edited region refers to the region in the to-be-edited image that needs to be edited according to the instruction-type editing text; this to-be-edited region is defined according to the user's editing intention and the visual features of the image, and this to-be-edited region can be an object in the image, part of the region of a scene, or a collection of pixels with specific visual attributes.

[0095] In the embodiments of the present application, the to-be-edited region can be represented by a mask, for example, the to-be-edited region is represented as white pixels, and other regions are represented as black pixels; the to-be-edited region can also be represented by image segmentation labels, for example, in some cases, the to-be-edited region may need to be represented more finely, different objects or different parts may need different editing operations, and the to-be-edited region can be represented by image segmentation labels, the image segmentation label value of each pixel in the to-be-edited image represents the category or object to which the pixel belongs; the to-be-edited region can also be represented by a list of pixel coordinates, for example, the to-be-edited region is discrete or irregular, and the pixel coordinates of these discrete or irregular regions can be stored in a list, and each list item can be a tuple or array containing pixel position (x, y); the to-be-edited region can also be represented by dividing the region boundary, for example, for a to-be-edited region of regular shape, a bounding box or a polygonal box can be used to represent it, and when the bounding box is a rectangle, the to-be-edited region can be enclosed by the bounding box, and the bounding box can be defined by the coordinates of the upper left corner and the lower right corner.

[0096] For example, in the application scenario of image editing, assuming that the instruction type editing text requires replacing a certain object in the image with another object. In step 104, first, the image features of the to-be-replaced object, such as shape, color, and texture, can be identified according to the to-be-edited region identifier; then, these image features are used for searching and matching in the to-be-edited image, so as to determine the to-be-edited region, i.e., the position and range of the to-be-replaced object.

[0097] Through step 104, the to-be-edited region corresponding to the instruction type editing text can be accurately determined from the to-be-edited image, ensuring that the subsequent editing operation can be accurately applied to the target region, avoiding misoperation or improper editing, improving the accuracy and efficiency of editing, and providing users with a more convenient and efficient image editing experience.

[0098] In step 105, the to-be-edited region of the to-be-edited image is edited according to the instruction type editing text, and an edited image is obtained.

[0099] In some embodiments, first, the instruction-type editing text can be subjected to detailed text analysis and semantic extraction, and the semantic requirements of image editing are extracted; then, the specific region of the image to be edited is edited according to the semantic requirements of image editing. The process of image editing on the to-be-edited region of the image to be edited can adjust or replace the content such as color, brightness, contrast and shape of the to-be-edited region, and can also perform various operations such as cropping, scaling, filter effect application, text superposition, and the process of image editing depends on the content of the instruction-type editing text. After the image editing on the image to be edited is completed, the edited image, that is, the "edited image", is obtained.

[0100] For example, in the image processing process, the user can want to replace the trees in a landscape photo with trees of another color; in step 105, according to the requirements in the instruction-type editing text, the to-be-edited region where the trees are located in the photo is first identified, and then the image replacement technology is applied to replace the original trees with new trees of a specified color.

[0101] Through step 105, accurate editing operations can be ensured on the specific region (i.e., the to-be-edited region) of the image to be edited according to the requirements of the instruction-type editing text, thereby greatly improving the accuracy and efficiency of editing, reducing the cumbersome process of manual editing by the user, and meeting the personalized needs of the user.

[0102] In some embodiments, referring to Figure 3B , Figure 3A The step 105 shown in the figure can be implemented through steps 1051 to 1052, which are specifically described as follows:

[0103] Step 1051, based on the to-be-edited region, a mask generation is performed on the image to be edited to obtain an image editing mask; wherein the image editing mask is used to indicate the to-be-edited region in the image to be edited.

[0104] In some embodiments, when the mask generation is performed on the image to be edited based on the to-be-edited region, the boundary and shape of the to-be-edited region can be identified by applying an image processing algorithm according to the determined to-be-edited region, and an accurate image editing mask can be generated according to the boundary and shape of the to-be-edited region. The image editing mask is a binary image with the same size as the image to be edited, wherein the to-be-edited region is marked with a specific value (such as white), and other regions are marked with a background value (such as black), and the image editing mask can clearly indicate the to-be-edited region in the image to be edited.

[0105] For example, in a specific application of image processing, a user wants to replace a person A in a group photo with a person B. In step 1051, the instruction type editing text and the to-be-edited region identifier are used to determine the to-be-edited region of the person A in the group photo that needs to be replaced, and then a mask generation technique is used to process the to-be-edited image to generate an image editing mask that can accurately indicate the region of the person A to be replaced.

[0106] Through step 1051, an accurate image editing mask can be generated, which can indicate the to-be-edited region in the to-be-edited image, thereby providing strong support for subsequent image editing operations and ensuring the accuracy and efficiency of the editing operation.

[0107] In step 1052, the instruction type editing text, the to-be-edited image, and the image editing mask are input into an image diffusion model, and the image diffusion model is used to generate an image in the to-be-edited region of the to-be-edited image to obtain an edited image.

[0108] In some embodiments, step 1052 can be implemented in the following manner: first, the instruction type editing text, the to-be-edited image, and the generated image editing mask are input into a pre-trained image diffusion model. The image diffusion model is a deep learning-based generative model that can generate high-quality images from random noise through a diffusion process. Then, the image diffusion model combines the editing intention in the instruction type editing text and the to-be-edited region indicated by the image editing mask to generate an image in the to-be-edited region of the to-be-edited image. Through the processes of forward diffusion (adding random noise to the to-be-edited region) and reverse diffusion (transforming random noise in the to-be-edited region into a real image), the image diffusion model gradually generates new image content in the to-be-edited region that meets the user's intention, thereby obtaining an edited image in which the to-be-edited region is new image content.

[0109] For example, in a specific application of image editing, a user wants to replace the sky in a landscape painting with a sunset scene. The instruction type editing text contains a description of the sunset scene, and the image editing mask indicates the sky region in the landscape painting that needs to be replaced. In step 1052, these information (instruction type editing text, to-be-edited image, and image editing mask) are input into the image diffusion model, which generates image content that meets the characteristics of the sunset scene in the sky region of the landscape painting according to the guidance of the instruction type editing text and the mask information, thereby obtaining an edited image.

[0110] Through step 1052, the specific region of the to-be-edited image can be subjected to high-quality image generation by using the image diffusion model, the editing accuracy and efficiency are improved, more realistic and natural image content can be generated, the personalized editing demand of the user can be better met, and the user experience is improved.

[0111] In some embodiments, referring to Figure 3C , Figure 3B The step 1052 shown can be implemented through steps 10521 to 10524, which are specifically explained as follows:

[0112] Step 10521, encoding the instruction type editing text by using a text encoder of the image diffusion model to obtain instruction text features corresponding to the instruction type editing text.

[0113] In some embodiments, the text encoder can be a pre-trained natural language processing model such as a transformer (Transformer), a bidirectional encoder representation from transformers (BERT), etc.; the natural language processing model can deeply understand the semantic information of the instruction type editing text and convert the semantic information into a high-dimensional feature vector, i.e., the instruction text features; these instruction text features can effectively capture the key information in the instruction type editing text and provide necessary guidance for subsequent image editing operations.

[0114] Step 10522, encoding the to-be-edited image by using an image encoder of the image diffusion model to obtain hidden space features corresponding to the to-be-edited image.

[0115] In some embodiments, the image encoder can be a deep learning model based on a convolutional neural network or a generative adversarial network. The deep learning model can deeply analyze the to-be-edited image, extract the underlying features and high-level semantic information of the to-be-edited image, and form the hidden space features. These hidden space features not only retain the original information of the image, but also facilitate subsequent fusion with the instruction text features and the image editing mask.

[0116] Step 10523, adjusting the size of the image editing mask to a standard size based on the size of the hidden space features to obtain a region space feature corresponding to the standard size.

[0117] In some embodiments, in order to ensure the consistency of the hidden space features and the image editing mask of different sizes in subsequent processing, the size of the hidden space features needs to be determined as a standard size. The size of the hidden space features can be understood as the dimension of the hidden space features or the feature map size of the hidden space features; the standard size is the basis for the size adjustment of the image editing mask. Since the image editing mask has the same size as the image to be edited, the image editing mask needs to be adjusted in size accordingly, so that the image editing mask matches the standard size of the hidden space features, and the region space features can accurately indicate the specific region in the image to be edited, providing position information for the subsequent image generation process. The region space features are the features obtained after the size adjustment of the image editing mask.

[0118] In step 10524, the image to be edited is decoded and generated based on the instruction text features, the hidden space features, and the region space features, to obtain an edited image.

[0119] In some embodiments, an image diffusion model can be used to generate an output image based on input features (instruction text features, hidden space features, and region space features). The image diffusion model can take the instruction text features, the hidden space features, and the region space features as inputs, perform a series of operations such as recombination of image content, modification or replacement of specific regions, and finally perform decoding to generate the final edited image, to achieve the effect required by the instruction-type editing text.

[0120] Through the implementation of steps 10521 to 10524, an edited image can be generated based on the instruction-type editing text and the image editing mask through the encoding and decoding processes, fully utilizing the feature information of the text and the image, as well as the editing region indicated by the mask, to achieve accurate editing of the image. At the same time, by unifying the sizes of the hidden space features and the region space features and fusing multiple feature information, the accuracy and efficiency of the image editing operation are improved, so that the final generated edited image not only meets the user's editing intention, but also maintains the original quality and details of the image to be edited.

[0121] In some embodiments, referring to Figure 3D , Figure 3D is another optional flowchart of the image editing method provided by the embodiments of the present application, as shown in Figure 3D , Figure 3D A in the above steps 101 to 105, before step 101, the method further includes steps 1001A to 1003A, which are described in detail as follows:

[0122] Step 1001A, obtaining a feature annotation sample set; wherein the feature annotation sample set includes training text features, training image features, and training to-be-edited region identifiers corresponding to the training text features and the training image features.

[0123] In some embodiments, the feature annotation sample set is a manually annotated data set, and the feature annotation sample set includes training text features, training image features, and training to-be-edited region identifiers corresponding to the training text features and the training image features, respectively. The training text features are feature representations extracted from training texts, and are used to describe semantic content and grammatical structure of the training texts. The training image features are feature representations extracted from training images, and are used to describe content and structure of the training images. The training to-be-edited region identifiers reflect the association between the training text features and the training image features, and are a key basis for subsequent low-rank module training.

[0124] Through step 1001A, a basis is provided for subsequent low-rank module embedding and training, ensuring that the large language model can effectively learn and reason on multi-modal data.

[0125] Step 1002A, embedding a low-rank module into a pre-trained large language model to obtain a low-rank multi-modal large language model; wherein the low-rank module includes at least one low-rank matrix.

[0126] In some embodiments, the pre-trained large language model is a multi-modal large language model that has not been trained and tuned. The low-rank module is a component used in low-rank adaptation technology, and is a special structure added to the pre-trained large language model, aiming to reduce the number of parameters that need to be updated during fine-tuning. The process of embedding the low-rank module into the pre-trained large language model is to introduce the low-rank module in certain layers of the pre-trained large language model, such as attention mechanisms or feedforward network layers. The low-rank multi-modal large language model is the multi-modal large language model obtained after embedding the low-rank module into the pre-trained large language model.

[0127] Step 1003A, training the low-rank multi-modal large language model based on the feature annotation sample set to obtain a pre-trained multi-modal large language model.

[0128] In some embodiments, the training process of the low-rank multi-modal large language model involves using the feature annotated training text features, the training image features, and the training to-be-edited region labels to optimize the parameters of the low-rank module in the low-rank multi-modal large language model. By iteratively updating the parameters of the low-rank module in the low-rank multi-modal large language model, the model can better understand the correlation between multi-modal data and generate accurate reasoning results. Training based on the feature annotated sample set enables the low-rank multi-modal large language model to better adapt to the processing of multi-modal data, improving the performance and stability of the low-rank multi-modal large language model in practical applications. The pre-trained multi-modal large language model is the large language model obtained after the training process of the low-rank multi-modal large language model.

[0129] In some embodiments, step 1003A can be implemented by first inputting the training text features and the training image features into the low-rank multi-modal large language model to obtain predicted to-be-edited region labels; then, based on the predicted to-be-edited region labels and the training to-be-edited region labels, determining the loss function value of the low-rank multi-modal large language model; if it is determined based on the loss function value that the current preset training stopping condition has not been reached, updating the parameters of the low-rank module in the low-rank multi-modal large language model to obtain an updated low-rank multi-modal large language model; if it is determined based on the loss function value that the current preset training stopping condition has been reached, the current updated low-rank multi-modal large language model is determined as the pre-trained multi-modal large language model.

[0130] The predicted to-be-edited region identifier is generated by the low-rank multi-modal large language model according to the input training text features and training image features in the training process. The preset training stop condition is a threshold or standard set in the training process to determine whether the model has been able to understand the training samples accurately enough. The preset training stop condition can be determined based on the value of the loss function, the performance indicators (such as accuracy, recall rate, etc.) of the model, or the number of training iterations. When the low-rank multi-modal large language model reaches or exceeds the preset training stop condition, the training process stops. The parameters of the low-rank module are parameter values learned by the low-rank multi-modal large language model in the training process. These parameters determine how the low-rank module cooperates with the multi-modal large language model to process the input features and generate the predicted to-be-edited region identifier. By adjusting the parameters of the low-rank module, the low-rank multi-modal large language model can improve the accuracy of the predicted to-be-edited region identifier. In the training process, the training text features and training image features in the feature annotation sample set are first input into the low-rank multi-modal large language model through forward propagation. The low-rank multi-modal large language model generates a predicted to-be-edited region identifier based on these features. Then, by comparing the predicted to-be-edited region identifier with the training to-be-edited region identifier, a loss function value is calculated, which reflects the accuracy of the model prediction. If the preset training stop condition is not met, the parameters of the low-rank module are updated through the back propagation algorithm to reduce the loss function value. This process is repeated until the training stop condition is met, and the final low-rank multi-modal large language model is the pre-trained multi-modal large language model.

[0131] Through this way of training, the pre-trained multi-modal large language model can fully utilize the feature information of text and image to improve the understanding and reasoning ability on multi-modal data.

[0132] In the embodiments of the present application, after steps 1001A to 1003A are executed, steps 101 to 105 are continued to be executed. The "pre-trained multi-modal large language model" involved in steps 101 to 105 is obtained through the execution of steps 1001A to 1003A.

[0133] In some embodiments, referring to Figure 3E , Figure 3E is another optional flowchart of the image editing method provided by the embodiments of the present application, as shown in Figure 3E , Figure 3E A in the above steps 101 to 105, before step 101, the method further includes steps 1001B to 1006B, which are explained as follows:

[0134] In step 1001B, a data generation sample set is obtained; wherein the data generation sample set includes an original image, an instruction type training text and an image training mask.

[0135] In some embodiments, the acquisition of the data generation sample set can be achieved in various ways, such as extraction from an existing image dataset. Meanwhile, the image training mask can be manually annotated, generated using an automatic annotation tool, or provided in an existing image dataset.

[0136] Step 1002B, analyzing the original image through the image analysis model to obtain the original image description text.

[0137] In some embodiments, step 1002B can be implemented in the following way: first, input the original image into the image analysis model, extract the image features of the original image through the feature extraction module of the image analysis model, and obtain the original image features; then, analyze the original image features through the feature analysis module of the image analysis model to obtain the original image description text.

[0138] Here, the image analysis model is an artificial intelligence model that can understand and interpret image content, such as the Contrastive Language-Image Pre-training (CLIP) model, etc.; the image analysis model includes two modules: feature extraction and feature analysis, which are used to analyze image features and generate image description text. The feature extraction module can extract key original image features from the original image; these original image features can be basic elements of images such as color, texture, shape, and size. The feature analysis module can further analyze and interpret the original image features generated by the feature extraction module; the feature analysis module can use natural language processing techniques such as recurrent neural networks or long short-term memory networks to convert visual features into text descriptions with semantic meaning to obtain the original image description text.

[0139] Step 1003B, determining the result image description text based on the original image description text and the instruction type training text.

[0140] In some embodiments, the process of determining the result image description text involves semantic understanding and fusion of the original image description text and the instruction type training text; natural language processing techniques such as text matching, semantic analysis, etc. can be used to effectively combine the original image description text and the instruction type training text to form a complete and demand-oriented result image description text. The result image description text is a text used to describe the result image.

[0141] For example, the original image description text is "a cat sitting on a sofa", the instruction type training text is "change the cat into a dog", and the result image description text is "a dog sitting on a sofa".

[0142] Through step 1003B, it can be ensured that the generated text not only conforms to the characteristics of the original image, but also embodies the user's instructions and intentions, thereby improving the efficiency and accuracy of the entire image generation process.

[0143] In some embodiments, referring to Figure 3F , Figure 3E The step 1003B shown can be implemented through steps 1003B1 to 1003B2, which are specifically explained as follows:

[0144] Step 1003B1, recombining the original image description text and the instruction type training text according to a preset text format to obtain a large language model input text.

[0145] In some embodiments, the preset text format can include specific separators, markers or templates for recombining the original image description text and the instruction type training text and being correctly parsed in the large language model; the recombination process can involve cleaning, formatting or encoding the text to adapt the obtained large language model input text to the input requirements of the general large language model.

[0146] For example, in the application scenario of image editing, the original image description text is: "There is a cat on the sofa", and the instruction type training text is: "There is a dog next to the cat". By recombining the original image description text and the instruction type training text according to the preset format, the large language model input text obtained is: "There is a cat on the sofa, there is a dog next to the cat, and the image is described again based on this".

[0147] Step 1003B2, inputting the large language model input text into the general large language model to generate output text and obtaining a result image description text.

[0148] In some embodiments, the general large language model is a general domain large language model that has not been initially prompted or fine-tuned, and the "general domain" refers to a domain covering a wide range of topics and knowledge, including but not limited to daily life, science, technology, literature, art, entertainment, history, geography and economy, etc. When the large language model input text is input into the general large language model, the general large language model will generate corresponding output text based on the semantic representation inside the large language model input text, thereby obtaining the result image description text.

[0149] In some embodiments, referring to Figure 3G , Figure 3F The step 1003B2 shown can be implemented through steps 1003B21 to 1003B23, which are specifically explained as follows:

[0150] Step 1003B21, setting an initial prompt for the general large language model to obtain a large language model with semantic conversion function; the initial prompt is a semantic conversion prompt.

[0151] In some embodiments, the initial prompt is a prompt provided to the large language model before the large language model generates the text, and the initial prompt is used to help the large language model understand the purpose and expected content of the generated text. The semantic conversion prompt is a special initial prompt, and the semantic conversion prompt is used to guide the large language model to perform the semantic conversion task, and the semantic conversion prompt is used to instruct the large language model to convert the input text into another form or context.

[0152] Step 1003B22, inputting the large language model input text into the large language model with semantic conversion function.

[0153] Step 1003B23, generating the output text of the large language model input text by the large language model with semantic conversion function to obtain the result image description text.

[0154] In some embodiments, when the large language model input text is input into the large language model with semantic conversion function, the large language model with semantic conversion function generates corresponding output text based on the semantic representation inside the large language model input text to obtain the result image description text.

[0155] By using step 1003B23, the result image description text that is more in line with the user's demand can be generated according to the user's instruction and the original image description text, which provides a more flexible and personalized image generation method for the user, and also improves the automation degree and efficiency of the image generation process.

[0156] Continuing to refer to Figure 3E , the step 1003B is continued.

[0157] Step 1004B, generating an image based on the original image, the image training mask and the result image description text to obtain a result image.

[0158] In some embodiments, the image generation process involves editing, modifying or reconstructing the original image to conform to the description of the result image description text; the image training mask specifies the areas in the image to be edited or retained.

[0159] Step 1005B, determining the original image, the image training mask and the instruction type training text as feature data, and determining the result image as label data to obtain a self-generated sample set; wherein any set of feature data in the self-generated sample set corresponds to multiple label data.

[0160] In some embodiments, each set of feature data in the self-generated sample set can correspond to multiple label data, and for the same set of feature data, multiple different feature data can be obtained through different generation methods or parameter settings.

[0161] On the one hand, step 1005B provides rich and diverse data resources for subsequent diffusion model training; on the other hand, since each set of feature data corresponds to multiple label data, the diffusion model can learn more changes and possibilities during training, thereby improving the generalization ability of the diffusion model and the diversity of generated images.

[0162] Step 1006B, based on the self-generated sample set, trains the initial diffusion model to obtain an image diffusion model.

[0163] In some embodiments, the feature data and the corresponding label data in the self-generated sample set are taken as a training data pair, and the parameters of the initial diffusion model are updated through a back propagation algorithm and a gradient descent optimizer; during the training process, the initial diffusion model continuously learns the mapping relationship from the feature data to the label data, gradually improves the quality and accuracy of the generated images, and finally obtains the image diffusion model.

[0164] Through step 1006B, an image diffusion model capable of efficiently generating high-quality images can be obtained, and the image diffusion model can fully utilize the rich data resources in the self-generated sample set, learn more changes and details, and thereby improve the diversity and realism of the generated images.

[0165] In the embodiments of the present application, after steps 1001B to 1006B are performed, steps 101 to 105 are continued to be performed, and the "image diffusion model" involved in steps 101 to 105 is obtained through steps 1001B to 1006B.

[0166] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0167] Embodiments of the present application propose an image editing method, and the target of embodiments of the present application is to add fine and intelligent capabilities to the instruction-based image editing model. In order to achieve this goal, embodiments of the present application consider summarizing and inducing reasoning prompts from human editing knowledge, one of the most important reasoning prompts is the to-be-edited region, which is the bridge between content understanding and generation. To this end, embodiments of the present application introduce a self-generated sample set creation method to simulate different data pairs of to-be-edited region masks and result image description texts. Then, embodiments of the present application use a multi-modal large language model and a segmentation model to train an instruction-based editing region generation network to generate reasoning masks from open language knowledge. In addition, embodiments of the present application propose a mask-guided instruction-based editing network based on a text-to-image diffusion model to accept generation prompts. A large number of experiments show that the method of embodiments of the present application has good editing capabilities in benchmark tests and real-world complex cases.

[0168] Embodiments of the present application aim to edit a to-be-edited image according to a complex natural language instruction (i.e., the above instruction type editing text). Given a to-be-edited image and an instruction type editing text, the image diffusion model of embodiments of the present application infers an image editing mask by a multi-modal large language model with a low-rank module and an image segmentation model when tested / applied, edits the to-be-edited region, and generates an edited image. Unlike the global single-stage end-to-end framework or direct fine-tuning in the related art, embodiments of the present application regard the editing mask as a bridge between understanding and generation, so that embodiments of the present application can make good use of pre-training knowledge in each field.

[0169] As shown in Figure 4 In the self-generated sample set creation process, embodiments of the present application first parse the original image in the data generated sample set using an image parser to obtain an original image description text; then input the original image description text and the instruction type training text into a large language model to obtain a result image description text; then process the original image, the image training mask and the result image description text through an image generation model to finally obtain a result image; repeat the above process for the same original image to obtain multiple result images (result image 1, result image 2 and result image 3). The image generation model can be implemented by a stable diffusion XL (Stable Diffusion XL, SDXL) model.

[0170] In the training / application process of the image diffusion model, the image editing mask generation is first performed. In the implementation process, a multimodal large language model with a low-rank module and an image segmentation model can be introduced to obtain the image editing mask of the to-be-edited region. Then, the image editing based on the instructional text is performed. In the implementation process, the image editing mask can be used as an additional condition, and the image diffusion model is used to perform the image editing based on the instructional editing text, the to-be-edited image and the image editing mask, to obtain the edited image.

[0171] The steps shown in the following Figure 4 will be explained in sequence. Figure 4 The modules involved in the

[0172] Step 410, self-generated sample set creation process.

[0173] If the data in the related art is directly used to generate the sample set to train the image diffusion model, the result is relatively poor, because the data in the related art generates the sample set by only containing no more than five thousand manually annotated sample data. Therefore, the embodiment of the present application first introduces a self-generated sample set creation process for local editing tasks. Even if the to-be-edited region remains unchanged, the image editing based on the instructional training text can have different editing results. A possible solution applied by the embodiment of the present application is to use the description text to generate the repair, however, if the result is directly generated from the original image description text, the result can not match the instructional training text. Therefore, as Figure 4 shown, the instructional training text 411 in the generated sample set 41, the original image 412 and the image training mask 413 are given as input data to obtain the result picture 48. First, the original image 412 is parsed by the image parser 42 to obtain the original image description text 43; then, the original image description text 43 and the instructional training text 44 are input into the large language model 45 to obtain the result image description text 46; finally, the result image 48-1, 48-2 and 48-3 are obtained by the image generation model 47.

[0174] Figure 5 shows an example of how to use a large language model with a preset prompt to generate a result image description text, Figure 5The "Preset Prompt 501" in the example is: "I want you to do a professional image editing. Given an original image and instruction text, you need to generate a description of the resulting image in one sentence. You only need to answer using the format '**This should be the original image description**', where '**This should be the original image description**' is the description of the original image." Based on the original image description and instruction training text in "Input to the Large Language Model of Preset Prompt 502", the final content of "Output of the Large Language Model of Preset Prompt 503" is: "Two men with their hands clasped together are standing in front of a truck." In cases where the input image description is missing, this embodiment generates the original image description text through an image parsing model; then, based on the original image description text and instruction training text, it generates the resulting image description text; after generating the resulting image description text, this embodiment can input the generated resulting image description text multiple times into the image generation model to obtain different versions of the resulting image. Therefore, for each area to be edited, this embodiment obtains a more varied self-generated sample set than the data generation sample set.

[0175] Step 420, the training / application process of the image diffusion model. Step 420 can be achieved through the following steps 4201 to 4202.

[0176] Step 4201, image editing mask generation.

[0177] Image editing mask 53 is a highly relevant, specific mask that can be obtained by combining the input image to be edited 51 and the instruction-type editing text. The region to be edited indicated by image editing mask 53 differs from object-level segmentation in related technologies; image editing mask 53 provides a more detailed depiction of a specific object, or it may be a meaningless region simply for placing something. Therefore, general inference segmentation models in related technologies, such as Large Language Guided Segmentation Assistant (LISA), perform poorly in these cases. Based on this, such as Figure 4 As shown, if the desired editable image 51 is to be edited, and the instruction text is "Place a pond next to the cow," then the region to be edited is a region next to the cow, not the cow itself. Furthermore, object-level segmentation requires precise segmentation of the object, while this embodiment only needs to segment an approximate region to be edited given the input image 51, to accommodate more possibilities. Since this region to be edited is not so straightforward, this embodiment requires stronger reasoning capabilities to understand the input instruction (i.e., the instruction text). Therefore, this embodiment uses a multimodal large language model 55 with a low-rank module 54 and an image segmentation model 52 to generate the region to be edited.

[0178] In the implementation process, the parameters of the original multi-modal language model are fixed, and a low-rank module 54 is trained to generate segmented region-to-edit identification. Then, the pre-trained segmentation model, such as the Segment Anything Model (SAM), is used to extract image features from the region-to-edit identification output by the multi-modal language model with the low-rank module 54 and generate an image editing mask 53. In this stage, the parameters of the low-rank module 54 and the decoder of the pre-trained image segmentation model 52 are only trained. The parameters of the low-rank module 54 and the decoder of the pre-trained image segmentation model 52 can be trained on the training data set using a standard binary cross-entropy loss function to predict the region-to-edit and obtain the image editing mask 53.

[0179] Step 4202, image editing based on instruction type text.

[0180] After obtaining the image editing mask 53, the language-guided image editing is performed by an efficient image diffusion model. In the implementation process, the image diffusion model 56 of the present application uses a model structure including a text encoder, an image encoder, and a denoising generator, uses Stable Diffusion as the denoising generator, and trains the image diffusion model 56 for image editing by using a self-generated sample set. The contrastive language-image pre-training (CLIP) model is used as the text encoder in the present application, and the contrastive language-image pre-training model is only used to understand the textual description, without directly participating in the image editing process. The image editing mask 53 is further introduced as the input of the image diffusion model 56 for fine-tuning for local editing tasks. Unlike encoding the image-to-edit 51 through the image encoder to obtain the hidden space features corresponding to the image-to-edit 51, the image editing mask is directly adjusted to the region space features with the same dimension as the hidden space features through an adjustment function.

[0181] The embodiments of the present application also adjust the standard diffusion loss function according to the self-generated sample set used. In the implementation process, the weights of the convolutional layers of the image diffusion model 56 can be adjusted to adapt to the differences in input data. In the training phase, the real mask (manually annotated mask) is used as input because the real mask can provide more accurate guidance. In the testing phase, the method of the embodiments of the present application performs editing based on the inference result (image editing mask 53) of the multi-modal large language model 55, which makes the image diffusion model 56 of the embodiments of the present application exhibit advanced performance in quantitative results compared to related technologies. In addition, from the perspective of perceptual results, the method of the embodiments of the present application also shows significant effectiveness. Since the embodiments of the present application infer the image editing mask 53 from the multi-modal large language model 55, accurate hints for the region to be edited can be provided, which performs better than the method in related technologies. The image editing method in related technologies cannot perform editing tasks such as "delete" and "add" because the large language model used in the image editing method in related technologies cannot understand and give specific regions for editing.

[0182] The following continues to illustrate an exemplary structure of the implementation of the image editing apparatus 455 provided by the embodiments of the present application as a software module. In some embodiments, as shown in Figure 2 The software module stored in the image editing apparatus 455 of the memory 450 can include: a data acquisition module 4551 configured to acquire an image to be edited and an instruction type editing text; a label determination module 4552 configured to perform feature annotation on image features of the image to be edited based on the instruction type editing text to obtain at least one to-be-edited region identifier of the image to be edited; a feature determination module 4553 configured to extract part of the image features corresponding to each to-be-edited region identifier from the image features of the image to be edited based on the at least one to-be-edited region identifier; a region determination module 4554 configured to determine a to-be-edited region corresponding to the instruction type editing text from the image to be edited according to the part of the image features corresponding to each to-be-edited region identifier; and an image editing module 4555 configured to perform image editing on the to-be-edited region of the image to be edited according to the instruction type editing text to obtain an edited image.

[0183] In some embodiments, the label determination module 4552 is further configured to: perform text feature extraction on the instruction type editing text to obtain editing text features; perform image feature extraction on the image to be edited to obtain image features of the image to be edited; and perform feature annotation on the editing text features and the image features of the image to be edited by a pre-trained multi-modal large language model to obtain at least one to-be-edited region identifier of the image to be edited.

[0184] In some embodiments, the apparatus further comprises a multi-modal large language model determining module configured to obtain a feature annotation sample set, wherein the feature annotation sample set comprises training text features, training image features, and training to-be-edited region identifiers corresponding to the training text features and the training image features; embed a low-rank module into a pre-trained large language model to obtain a low-rank multi-modal large language model, wherein the low-rank module comprises at least one low-rank matrix; and train the low-rank multi-modal large language model based on the feature annotation sample set to obtain the pre-trained multi-modal large language model.

[0185] In some embodiments, the multi-modal large language model determining module is further configured to: input the training text features and the training image features into the low-rank multi-modal large language model to obtain predicted to-be-edited region identifiers; determine a loss function value of the low-rank multi-modal large language model based on the predicted to-be-edited region identifiers and the training to-be-edited region identifiers; if it is determined based on the loss function value that a preset training stop condition is not met, update parameters of the low-rank module in the low-rank multi-modal large language model to obtain an updated low-rank multi-modal large language model; and if it is determined based on the loss function value that the preset training stop condition is met, determine the current updated low-rank multi-modal large language model as the pre-trained multi-modal large language model.

[0186] In some embodiments, the image editing module 4555 is further configured to: generate an image editing mask based on the to-be-edited region, wherein the image editing mask is used to indicate the to-be-edited region in the to-be-edited image; input the instruction type editing text, the to-be-edited image, and the image editing mask into the image diffusion model to generate an edited image by performing image generation on the to-be-edited region of the to-be-edited image through the image diffusion model.

[0187] In some embodiments, the image editing module 4555 is further configured to: encode the instruction type editing text through a text encoder of the image diffusion model to obtain instruction text features corresponding to the instruction type editing text; encode the to-be-edited image through an image encoder of the image diffusion model to obtain hidden space features corresponding to the to-be-edited image; adjust the size of the image editing mask to a standard size of the hidden space features to obtain a region space feature corresponding to the standard size; and decode the to-be-edited image based on the instruction text features, the hidden space features, and the region space feature to obtain the edited image.

[0188] In some embodiments, the apparatus further comprises a self-generated sample set generation module configured to obtain a data generated sample set, wherein the data generated sample set comprises an original image, an instruction type training text, and an image training mask; parse the original image by using the image parsing model to obtain an original image description text; determine a result image description text based on the original image description text and the instruction type training text; generate an image based on the original image, the image training mask, and the result image description text to obtain a result image; determine the original image, the image training mask, and the instruction type training text as feature data, and determine the result image as label data to obtain the self-generated sample set; wherein any one group of feature data in the self-generated sample set corresponds to multiple label data; and train the initial diffusion model based on the self-generated sample set to obtain the image diffusion model.

[0189] In some embodiments, the self-generated sample set generation module is further configured to input the original image into the image parsing model, perform image feature extraction on the original image by using a feature extraction module of the image parsing model to obtain original image features; and perform feature analysis on the original image features by using a feature analysis module of the image parsing model to obtain the original image description text.

[0190] In some embodiments, the self-generated sample set generation module is further configured to recombine the original image description text and the instruction type training text according to a preset text format to obtain a large language model input text; input the large language model input text into the general large language model to generate an output text to obtain the result image description text.

[0191] In some embodiments, the self-generated sample set generation module is further configured to set an initial prompt for the general large language model to obtain a large language model with a semantic conversion function; the initial prompt is a semantic conversion prompt; input the large language model input text into the large language model with the semantic conversion function; and generate an output text for the large language model input text by using the large language model with the semantic conversion function to obtain the result image description text.

[0192] In some embodiments, the data acquisition module 4551 is further configured to obtain an instruction type editing voice; perform speech recognition on the instruction type editing voice to obtain an original editing text; and perform text revision on the original editing text by using the large language model with the text revision function to obtain an instruction type editing text; wherein the large language model with the text revision function is a large language model obtained by training the general large language model with the initial prompt after presetting a text revision prompt as the initial prompt of the general large language model.

[0193] The embodiment of the present application provides a computer program product, which comprises a computer program or computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the image editing method provided by the embodiment of the present application.

[0194] The embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions or computer programs. When the computer executable instructions or computer programs are executed by a processor, the processor executes the image editing method provided by the embodiment of the present application, for example, the image editing method shown in the embodiment of the present application. Figure 3A The embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions or computer programs. When the computer executable instructions or computer programs are executed by a processor, the processor executes the image editing method provided by the embodiment of the present application, for example, the image editing method shown in the embodiment of the present application.

[0195] In some embodiments, the computer readable storage medium can be RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory, and the like; or can be various devices including one or any combination of the above storage medium.

[0196] In some embodiments, the computer executable instructions can be in the form of programs, software, software modules, scripts or codes, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and can be deployed in any form, including being deployed as independent programs or being deployed as modules, components, subroutines or other units suitable for use in a computing environment.

[0197] As an example, the computer executable instructions can but not necessarily correspond to files in a file system, can be stored in a part of a file storing other programs or data, for example, stored in one or more scripts in a HyperText Markup Language (HTML) document, stored in a single file dedicated to the program in question, or stored in multiple cooperative files (for example, files storing one or more modules, subroutines or code parts).

[0198] As an example, the computer executable instructions can be deployed to be executed on one electronic device, or executed on multiple electronic devices located in one place, or executed on multiple electronic devices distributed in multiple places and interconnected through a communication network.

[0199] To sum up, by introducing the instruction type editing text, the user can input concise editing instructions to realize accurate editing of the image. This instruction type editing method not only improves the editing efficiency, but also reduces the editing threshold, so that the user can operate the image editing without complex training or learning, and the image editing process is more intelligent and flexible. By annotating the image features of the image to be edited, the to-be-edited region identifier is generated, and the corresponding part of the image features is extracted based on the to-be-edited region identifier, which realizes the deep understanding and analysis of the image content, so that the to-be-edited region corresponding to the instruction type editing text can be more accurately determined. By performing specific editing operations on the to-be-edited region according to the instruction type editing text, controllable editing of the image is realized, so that the user can finely modify and adjust the specific region of the image according to his own needs, so as to obtain a more expected editing result. In summary, the present application accurately positions and edits the to-be-edited region of the image according to the instruction type editing text, realizes intelligent, accurate and controllable editing of the image, improves the efficiency and accuracy of image editing, and provides a more convenient, efficient and personalized image editing experience for the user. To sum up, the embodiment of the present application accurately positions and edits the to-be-edited region of the image according to the instruction type editing text, realizes intelligent, accurate and controllable editing of the image, improves the efficiency and accuracy of image editing.

[0200] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image editing method, characterized in that, The method includes: Get the image to be edited and the instruction text for editing; Based on the instruction-type editing text, feature annotation is performed on the image features of the image to be edited to obtain at least one editable region identifier of the image to be edited; Based on the at least one region identifier to be edited, extract partial image features corresponding to each region identifier to be edited from the image features of the image to be edited; Based on the partial image features corresponding to each of the regions to be edited, the region to be edited corresponding to the instruction-type edit text is determined from the image to be edited; The image to be edited is obtained by editing the area of ​​the image to be edited according to the instruction-type editing text.

2. The method according to claim 1, characterized in that, The step of annotating the image features of the image to be edited based on the instruction-type editing text to obtain at least one editable region identifier of the image to be edited includes: Text features are extracted from the instruction-type edited text to obtain the edited text features; Image features are extracted from the image to be edited to obtain the image features of the image to be edited; By using a pre-trained multimodal large language model, feature annotations are performed on the features of the edited text and the image features of the image to be edited, resulting in at least one identifier of the region to be edited in the image to be edited.

3. The method according to claim 2, characterized in that, The method further includes: Obtain a feature annotation sample set; wherein, the feature annotation sample set includes training text features, training image features, and training region identifiers corresponding to the training text features and the training image features; A low-rank module is embedded into a pre-trained large language model to obtain a low-rank multimodal large language model; wherein, the low-rank module includes at least one low-rank matrix; The low-rank multimodal large language model is trained based on the feature annotation sample set to obtain the pre-trained multimodal large language model.

4. The method according to claim 3, characterized in that, The process of training the low-rank multimodal large language model based on the feature annotation sample set to obtain the pre-trained multimodal large language model includes: The training text features and the training image features are input into the low-rank multimodal large language model to obtain the predicted identifier of the region to be edited; Based on the predicted region identifier and the trained region identifier, the loss function value of the low-rank multimodal large language model is determined; If it is determined based on the loss function value that the preset training stopping condition has not been met, then the parameters of the low-rank module in the low-rank multimodal large language model are updated to obtain the updated low-rank multimodal large language model. If the preset training stopping condition is determined based on the loss function value, then the currently updated low-rank multimodal large language model is determined as the pre-trained multimodal large language model.

5. The method according to claim 1, characterized in that, The step of editing the editable area of ​​the image according to the instruction-type editing text to obtain the edited image includes: Based on the region to be edited, a mask is generated for the image to be edited to obtain an image editing mask; wherein, the image editing mask is used to indicate the region to be edited in the image to be edited; The instruction-type editing text, the image to be edited, and the image editing mask are input into the image diffusion model. The image diffusion model generates an image of the region to be edited in the image to be edited, thereby obtaining the edited image.

6. The method according to claim 5, characterized in that, The step of generating an edited image by using the image diffusion model to generate an edited region of the image to be edited includes: The text encoder of the image diffusion model encodes the instruction-type edited text to obtain the instruction text features corresponding to the instruction-type edited text; The image to be edited is encoded by the image encoder of the image diffusion model to obtain the hidden space features corresponding to the image to be edited; Using the size of the hidden spatial feature as the standard size, the image editing mask is resized to obtain the regional spatial feature corresponding to the standard size; Based on the instruction text features, the hidden space features, and the region space features, the image to be edited is decoded and generated to obtain the edited image.

7. The method according to claim 5, characterized in that, The method further includes: Acquire data to generate a sample set; wherein, the data generated sample set includes the original image, instruction-type training text, and image training mask; The original image is parsed using an image parsing model to obtain the original image description text; Based on the original image description text and the instruction-type training text, the resulting image description text is determined; Based on the original image, the image training mask, and the descriptive text of the resulting image, an image is generated to obtain the result image; The original image, the image training mask, and the instruction-type training text are determined as feature data, and the resulting image is determined as label data to obtain a self-generated sample set; wherein, any set of feature data in the self-generated sample set corresponds to multiple sets of label data; The image diffusion model is obtained by training the initial diffusion model based on the self-generated sample set.

8. The method according to claim 7, characterized in that, The step of parsing the original image using an image parsing model to obtain the original image description text includes: The original image is input into the image parsing model, and the feature extraction module of the image parsing model extracts image features from the original image to obtain the original image features; The original image features are analyzed by the feature analysis module of the image analysis model to obtain the original image description text.

9. The method according to claim 7, characterized in that, The process of determining the resulting image description text based on the original image description text and the instruction-type training text includes: The original image description text and the instruction training text are recombined according to a preset text format to obtain the input text for the large language model; The input text of the large language model is fed into the general large language model to generate the output text, resulting in the image description text.

10. The method according to claim 9, characterized in that, The step of inputting the input text of the large language model into a general large language model, generating the output text, and obtaining the resulting image description text includes: An initial prompt is set for the general-purpose large language model to obtain a large language model with semantic conversion function; the initial prompt is a semantic conversion prompt. The input text of the large language model is input into the large language model with semantic conversion function; The large language model with semantic transformation function is used to generate the output text from the input text of the large language model, and the resulting image description text is obtained.

11. The method according to any one of claims 1 to 10, characterized in that, Retrieve instruction-based editing text, including: Acquire command-based editing speech; The original edited text is obtained by performing speech recognition on the instruction-type editing speech; The original edited text is revised using a large language model with text revision capabilities to obtain the instruction-type edited text; wherein, the large language model with text revision capabilities is a large language model obtained by training a general large language model with the text revision prompts preset as the initial prompts of the general large language model.

12. An image editing device, characterized in that, The device includes: The data acquisition module is used to acquire the image to be edited and the instruction-type editing text; The marker determination module is used to annotate the image features of the image to be edited based on the instruction-type editing text, and obtain at least one identifier of the editable region of the image to be edited; A feature determination module is used to extract partial image features corresponding to each of the editable regions from the image features of the image to be edited, based on the at least one editable region identifier; The region determination module is used to determine the region to be edited corresponding to the instruction-type edit text from the image to be edited based on partial image features corresponding to each region identifier to be edited; The image editing module is used to edit the area to be edited in the image according to the instruction-type editing text, so as to obtain the edited image.

13. An electronic device, characterized in that, include: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the image editing method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the image editing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the image editing method according to any one of claims 1 to 11.

Images