Method and apparatus for image processing, device, and storage medium

Abstract

According to embodiments of the present invention, provided are a method and apparatus for image processing, a device, and a storage medium. The method comprises: acquiring a target input image and indication information for a target editing area, the target editing area being located inside or outside the input image; on the basis of the target input image and the indication information, determining a target input for image editing; and on the basis of the target input, using an image editing model to generate a target output image, compared with the target input image, the target editing area being updated in the target output image. The image editing model is trained by using reward information, wherein the reward information is associated with a training output image generated by the image editing model from a training input image. In this way, reward information can be used to train an image editing model, which facilitates improvement of the image editing efficiency and quality.

Description

Method, apparatus, device and storage medium for image processing

[0001] This application claims priority to the Chinese invention patent application entitled “Method, device, apparatus and storage medium for image processing” and application number 202410345788.3, ​​filed on March 25, 2024. The entire contents of that application are incorporated herein by reference. Technical Field

[0002] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for image processing. Background Art

[0003] With the development of Internet technology, more and more applications or platforms provide image processing functions, which brings many conveniences to users. Image editing tasks are important tasks in image processing tasks. Applications with image editing functions can obtain the user's input image, editing area (also called the area to be edited) and text, and generate an edited image corresponding to the text based on this information. Compared with the input image, the edited image only differs in the editing area, and the contents of the two in other areas outside the editing area are the same. It is expected that the quality of the edited image can be improved and the effect of image editing can be improved. Summary of the Invention

[0004] In a first aspect of the present disclosure, a method for image processing is provided. The method comprises: obtaining a target input image and information indicating a target editing region, the target editing region being located inside or outside the input image; determining a target input for image editing based on the target input image and the information indicating the target editing region; and generating a target output image based on the target input using an image editing model, wherein the target editing region is updated in the target output image compared to the target input image, wherein the image editing model is trained using reward information associated with a training output image generated by the image editing model from the training input image.

[0005] In a second aspect of the present disclosure, a device for image processing is provided. The device includes: an acquisition module configured to acquire a target input image and indication information of a target editing region, where the target editing region is located inside or outside the input image; an input determination module configured to determine a target input for image editing based on the target input image and the indication information; and an image generation module configured to generate a target output image using an image editing model based on the target input, where the target editing region is updated in the target output image compared to the target input image, wherein the image editing model is trained using reward information, and the reward information is associated with a training output image generated by the image editing model from the training input image.

[0006] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, causing the electronic device to perform the method according to the first aspect of the present disclosure.

[0007] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor implements the method according to the first aspect of the present disclosure.

[0008] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of the present disclosure is implemented.

[0009] It should be understood that the content described in this summary section is not intended to limit the key features or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0011] FIG1 shows a schematic diagram of an example environment in which embodiments of the present disclosure can be implemented;

[0012] FIG2 shows a schematic diagram of an example architecture for training an image editing model according to some embodiments of the present disclosure;

[0013] FIG3 shows a schematic diagram of an example according to some embodiments of the present disclosure;

[0014] FIG4 shows a flowchart of a process for image processing according to some embodiments of the present disclosure;

[0015] FIG5 shows a schematic structural block diagram of an apparatus for image processing according to some embodiments of the present disclosure; and

[0016] FIG6 illustrates a block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0017] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, i.e., "including but not limited to." The term "based on" should be understood as "based at least in part on." The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment." The term "some embodiments" should be understood as "at least some embodiments." Other explicit and implicit definitions may be included below.

[0019] It should be noted that the acquisition, storage and application of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0020] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0021] For example, in response to receiving a user's active request, a prompt message is sent to the user to clearly remind the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to the electronic device, application, server or storage medium and other software or hardware that performs the operation of the technical solution of the present disclosure based on the prompt message.

[0022] As an optional but non-limiting implementation, in response to receiving a user's active request, a prompt message may be sent to the user, for example, in the form of a pop-up window, in which the prompt message may be presented in text form. Furthermore, the pop-up window may also include a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0023] It is understandable that the above notification and user authorization process are merely illustrative and do not limit the embodiments of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the embodiments of the present disclosure.

[0024] As used herein, the term "model" can learn the association between corresponding inputs and outputs from training data, so that after training is completed, corresponding outputs can be generated for given inputs. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0025] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs. It typically includes an input layer, an output layer, and one or more hidden layers between the input and output layers. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes the input from the previous layer.

[0026] Generally speaking, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also known as the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also known as input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. In the application stage, the model can be used to process the actual input based on the parameter values ​​obtained through training to determine the corresponding output.

[0027] Sample Environment

[0028] FIG1 shows a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. As shown in FIG1 , the environment 100 may include an electronic device 110 .

[0029] Electronic device 110 may obtain an input image 102 and indication information 104. The input image 102 and indication information 104 may be obtained in real time or pre-acquired and stored locally. The input image 102 may be an image of any appropriate type (e.g., a grayscale image, an RGB color image, etc.), in any format, and of any size. The indication information 104 may at least indicate an editing area for the input image 102. For example, the indication information 104 may include an image 106 indicating the editing area, where the white area in image 106 is the editing area.

[0030] The electronic device 110 can generate an output image 112 based on the input image 102 and the indication information 104. Compared with the input image 102, the editing area is updated in the output image 112. That is, the input image 102 and the output image 112 are different only in the editing area. In some embodiments, the electronic device 110 can generate the output image 112 based on the input image 102 and the indication information 104 with the help of an image editing model 120. The image editing model 120 can also be simply referred to as model 120. The image editing model 120 can be deployed locally on the electronic device 110, or it can be deployed at other electronic devices (such as remote electronic devices). The image editing model 120 can, for example, include but is not limited to any appropriate model such as a Transformer model, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), etc.

[0031] In the environment 100, the electronic device 110 can be any type of device with computing capabilities, including a terminal device or a server device. The terminal device can be any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an e-book device, a gaming device, or any combination thereof, including accessories and peripherals of these devices or any combination thereof.

[0032] A server-side device can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. Server-side devices may include, for example, computing systems / servers such as mainframes, edge computing nodes, and computing devices in cloud environments.

[0033] It should be understood that the structure and functionality of environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0034] As mentioned earlier, image editing is a crucial task in image processing. Applications with image editing capabilities can obtain a user's input image, an editing area, and text, and based on this information, generate an edited image corresponding to the text. There are expectations for improving the speed and quality of edited image generation.

[0035] In view of this, an embodiment of the present disclosure proposes an improved scheme for image processing. According to various embodiments of the present disclosure, a target input image and indication information of a target editing area are obtained, and the target editing area is located inside or outside the input image. Based on the target input image and the indication information, a target input for image editing is determined. Based on the target input, an image editing model is used to generate a target output image. Compared with the target input image, the target editing area is updated in the target output image, wherein the image editing model is trained using reward information, and the reward information is associated with the training output image generated by the image editing model from the training input image.

[0036] In this way, in an embodiment of the present disclosure, reward information can be used to train an image editing model, which helps to improve the efficiency and performance of image editing.

[0037] Some example embodiments of the present disclosure will be described below with continued reference to the accompanying drawings.

[0038] Detailed process of image processing

[0039] In an embodiment of the present disclosure, the electronic device 110 may implement an image processing task with the aid of an image editing model. The image editing model may include any appropriate model. In some embodiments, the image editing model may include a diffusion model.

[0040] During the model application phase, the electronic device 110 may obtain a target input image and information indicating a target editing area. The target editing area may be located inside or outside the input image. If the target editing area is located inside the input image, the electronic device 110 may modify or erase / reconstruct the content located inside the input image at the target editing area using the image editing model. If the target editing area is located outside the input image, the electronic device 110 may expand the content located outside the input image at the target editing area using the image editing model.

[0041] The electronic device 110 may determine the target input for image editing based on the target input image and the indication information. In some embodiments, the electronic device 110 may obtain a first input image from the target input image as part of the target input by adding noise to the target editing area (any appropriate type of noise may be added in any appropriate manner). The electronic device 110 may also obtain a second input image as part of the target input by removing the target editing area from the target input image. The electronic device 110 may also generate a mask for indicating the position of the target editing area as part of the target input. It should be noted that the indication information for the target editing area may indicate the target editing area in any appropriate manner. If the indication information includes a mask for the target editing area, the electronic device 110 may directly determine the mask for indicating the position of the target editing area based on the indication information.

[0042] The electronic device 110 can then use the trained image editing model to generate a target output image based on the target input. Compared with the target input image, the target editing area is updated in the target output image. That is, the content at the target editing area of ​​the target input image is different from the content at the target editing area of ​​the target output image. In some embodiments, the electronic device 110 can also obtain target description information about the image editing target (for example, it can include descriptive text). For example, the electronic device 110 can use the trained image editing model to generate the target output image based on the target input and the target description information.

[0043] Regarding the training process of the image editing model, the image editing model can be trained using reward information. The reward information is associated with the training output image generated by the image editing model from the training input image. In some embodiments, at least a portion of the reward information is generated using a reward model, and the image editing model and the reward model are trained through an adversarial training mode. The reward model may include multiple different types of reward models, for example, a consistency reward model, a matching reward model, a quality reward model, and so on. In the case of training the image editing model and the reward model through an adversarial training mode, the image editing model can be regarded as a generator and the reward model can be regarded as a discriminator.

[0044] FIG2 shows a schematic diagram of an example architecture 200 for training an image editing model according to some embodiments of the present disclosure. The architecture 200 may be implemented at the electronic device 110. The architecture 200 is described below with reference to FIG1.

[0045] During the model training phase, the electronic device 110 may obtain a training input image 212, training instruction information 214 for a training editing region of the training input image 212, and description information 202 regarding an image editing target. The training input image 212 and the training instruction information 214 may be obtained from a dataset 201, which may be a training dataset including a large number of training input images 212 and training instruction information corresponding to the training input images 212.

[0046] Electronic device 110 may determine training input 220 for image editing based on training input image 212 and training indication information 214. For example, electronic device 110 may add noise to the training edit region to obtain a first training input image 222 from training input image 212, which serves as part of training input 220. Electronic device 110 may also remove the training edit region from training input image 212 to obtain a second training input image 224, which serves as part of training input 220. Electronic device 110 may also generate a mask 226 indicating the location of the training edit region, as part of training input 220. In some embodiments, if indication information 214 directly includes mask 226, electronic device 110 may also determine mask 226 directly based on indication information 214. Electronic device 110 may provide training input 220 and description information 202 to an image editing model to be trained (e.g., diffusion model 230 shown in the figure), so that the image editing model can be used to generate a training output image 232 for training input image 212. The diffusion model 230 can, for example, predict the added noise based on the training input 220 and learn basic image editing capabilities.

[0047] The electronic device 110 can generate at least one reward for evaluating the training output image 232 as reward information based on the training output image 232. The at least one reward can, for example, include a matching reward, which indicates the degree of matching between the training editing area in the training output image 212 and the description information 202 (the higher the matching reward, the higher the matching degree). The use of matching rewards is conducive to improving the image-text matching of image editing, which is an important aspect of image editing performance. The reward can also include a consistency reward, which indicates the visual consistency between the inside and outside of the editing area (the higher the consistency reward, the higher the visual consistency between the inside and outside of the editing area). That is, the consistency reward can evaluate whether the visual content between the inside and outside of the editing area is harmonious. The reward can also include a quality reward, which indicates the visual perception quality (e.g., aesthetic quality, realism, details, etc.) of the training output image 232 (the higher the quality reward, the higher the visual perception quality). In particular, in the following embodiment, the quality reward can be an aesthetic reward.

[0048] In some embodiments, at least a portion of the at least one reward can be generated using at least one reward model. For example, architecture 200 may further include a consistency reward model 240 for generating consistency rewards, a matching reward model 250 for generating matching rewards, and a quality reward model 260 for generating quality rewards. The consistency reward model 240 may be trained, for example, together with the diffusion model 230 using an adversarial training model. When the diffusion model 230 and the consistency reward model 240 are trained using an adversarial training model, the diffusion model 230 can be considered a generator and the consistency reward model 240 can be considered a discriminator. The matching reward model 250 and the quality reward model 260 may be pre-trained models. For example, the matching reward model 250 may be a model pre-trained using a matching dataset 251, and the quality reward model 260 may be a model pre-trained using a quality dataset 261. During the training of the image editing model, the model parameters of the matching reward model 250 and the quality reward model 260 may be fixed.

[0049] The at least one reward may be represented as a specific score, for example. That is, the at least one reward model may assign a score based on the degree of match between the training edit region in the training output image 212 and the description information 202, the visual consistency between the training input image 212 and the remaining regions of the training output image 232 excluding the training edit region, and the visual perceptual quality of the training output image 232. The scoring result is the at least one reward.

[0050] The electronic device 110 may determine the target loss for the diffusion model 230 based on at least one reward, for example. In some embodiments, the electronic device 110 may determine the total loss 205 (also represented as L) directly based on at least one reward. reward The electronic device 110 may, for example, obtain the total loss 205 by performing a weighted summation on at least one reward. The electronic device 110 may, for example, determine the total loss 205 as a target loss and use the target loss to train the diffusion model 230 to determine the model parameters of the diffusion model 230.

[0051] Alternatively or additionally, in some embodiments, in order to make the remaining areas of the output image except the edited area consistent with the input image, the electronic device 110 may further determine the image difference between the training input image 212 and the training output image 232 .

[0052] Specifically, the electronic device 110 may determine the pixel level difference 205 (also represented as L ) based on the difference between the pixels of the training input image 212 and the pixels of the training output image 232. regThe electronic device 110 may also determine a feature level difference 204 (also denoted as L ) based on the difference between the feature representation of the training input image 212 and the feature representation of the training output image 232. vgg The electronic device 110 may, for example, use the VGG network feature layer 203 to determine the difference between the feature representation of the training input image 212 and the feature representation of the training output image 232 , that is, use the VGG network feature layer 203 to determine the feature level difference 204 .

[0053] The electronic device 110 may determine a first target loss (also referred to as perceptual feedback learning L) based on the image difference (ie, the feature level difference 204 and the pixel level difference 205) and at least one reward. pefl ). The electronic device 110 may then use the first target loss to train the diffusion model 230 to determine model parameters of the diffusion model 230.

[0054] In some embodiments, when using the adversarial training mode to train the diffusion model 230 and the consistency reward model 240, the generator (ie, the diffusion model 230) needs to generate realistic images as much as possible (which can also be understood as training output images 232 that are similar to the training input image 212) to deceive the consistency reward model 240, and the discriminator (ie, the consistency reward model 240) needs to distinguish whether the input is a real image (ie, the training input image 212) or a fake image (ie, the training output image 232). The electronic device 110 can set the first target loss (ie, the perceptual feedback learning L pefl ) is determined as the generator loss (which can be called L G ), the difference between the discriminant result of the consistency reward model 240 and the true result is determined as the discriminator loss 207 (also known as L D The discriminator loss optimization goal is to increase the score of real images and reduce the score of fake images. The electronic device 110 can use the generator loss and the discriminator loss 207 to train the diffusion model 230 and the consistency reward model 240 in an adversarial training mode.

[0055] In some embodiments, in order to improve the efficiency of image processing, the electronic device 110 may include multiple stages in training the diffusion model 230. Taking the multiple stages including the first stage and the second stage as an example (it can be understood that more stages may be included), the noise addition granularity in the forward diffusion process of the first stage is finer than the noise addition granularity in the forward diffusion process of the second stage. Such a multi-stage training, in which the noise addition granularity of the earlier stage is finer than the noise addition granularity of the later stage, can also be referred to as progressive training. For example, the noise addition granularity in the forward diffusion process of the first stage (also referred to as the number of optimization steps of the diffusion model of the first stage) can be determined to be 20 (also referred to as T=20), and the noise addition granularity in the forward diffusion process of the second stage (also referred to as the number of optimization steps of the diffusion model of the second stage) can be determined to be 8 (also referred to as T=8).

[0056] The longer the number of optimization steps (that is, the finer the granularity of noise addition), the higher the quality of the image input to the reward model will generally be. Figure 3 shows a schematic diagram of example 300 according to some embodiments of the present disclosure. As shown in Figure 3, the output effect of the model after the model is trained using a multi-stage training method (that is, the bar chart corresponding to "acceleration" in the figure) and the output effect of the model after the model is trained using a traditional one-stage training method (that is, the bar chart corresponding to "no acceleration" in the figure) do not differ much in image processing effects in various image processing scenarios such as expansion, modification, and erasure-reconstruction. Therefore, a two-stage training method is adopted. In the first stage, a longer number of optimization steps is used to preliminarily improve the quality of the generated image. In the second stage, reducing the number of optimization steps promotes the acceleration effect, which can improve the efficiency of image editing without losing performance.

[0057] In summary, according to the embodiments of the present disclosure, reward information can be used to train an image editing model, which helps to improve the efficiency and quality of image processing.

[0058] Example Process

[0059] FIG4 shows a flow chart of a process 400 for image processing according to some embodiments of the present disclosure. The process 400 may be implemented at the electronic device 110. The process 400 is described below with reference to FIG1.

[0060] In block 410 , the electronic device 110 obtains a target input image and indication information of a target editing region, where the target editing region is located inside or outside the input image.

[0061] In block 420 , the electronic device 110 determines a target input for image editing based on the target input image and the indication information.

[0062] In box 430, the electronic device 110 generates a target output image using the image editing model based on the target input, and the target editing area is updated in the target output image compared to the target input image, wherein the image editing model is trained using reward information, and the reward information is associated with the training output image generated by the image editing model from the training input image.

[0063] In some embodiments, the image editing model is trained as follows: based on a training input image, a training editing region in the training input image, and descriptive information about the image editing target, a training output image is generated using the image editing model; based on the training output image, at least one reward for evaluating the training output image is generated as reward information; based on the at least one reward, a target loss for the image editing model is determined; and model parameters of the image editing model are determined using the target loss.

[0064] In some embodiments, the at least one reward includes one or more of: a matching reward indicating how well the training edit region in the training output image matches the description information, a consistency reward indicating the visual consistency between the training input image and the remaining regions in the training output image excluding the training edit region, or a quality reward indicating the visually perceived quality of the training output image.

[0065] In some embodiments, determining the target loss includes: determining an image difference between the training input image and the training output image; and obtaining a first target loss based on the image difference and the at least one reward.

[0066] In some embodiments, determining the image difference comprises at least one of determining a pixel-level difference based on a difference between pixels of a training input image and pixels of a training output image, or determining a feature-level difference based on a difference between a feature representation of the training input image and a feature representation of the training output image.

[0067] In some embodiments, at least a portion of the reward information is generated using a reward model, and the image editing model and the reward model are trained via an adversarial training scheme.

[0068] In some embodiments, the image editing model includes a diffusion model, and training of the diffusion model includes a first stage and a second stage, wherein the granularity of noise addition in the forward diffusion process of the first stage is finer than that of noise addition in the forward diffusion process of the second stage.

[0069] In some embodiments, determining a target input for image editing based on a target input image and indication information includes: obtaining a first input image from the target input image by adding noise to a target editing area as part of the target input; obtaining a second input image by removing the target editing area from the target input image as part of the target input; and generating a mask for indicating the position of the target editing area as part of the target input.

[0070] Example devices and equipment

[0071] Embodiments of the present disclosure also provide corresponding apparatuses for implementing the above-described methods or processes. FIG5 shows a schematic structural block diagram of an apparatus 500 for image processing according to some embodiments of the present disclosure. Apparatus 500 may, for example, be included in electronic device 110 in FIG1 . Each module / component in apparatus 500 may be implemented by hardware, software, firmware, or any combination thereof.

[0072] As shown in the figure, the device 500 includes an acquisition module 510, which is configured to acquire a target input image and indication information of a target editing area, where the target editing area is located inside or outside the input image. The device 500 also includes an input determination module 520, which is configured to determine a target input for image editing based on the target input image and the indication information. The device 500 also includes an image generation module 530, which is configured to generate a target output image based on the target input using an image editing model, where the target editing area is updated in the target output image compared to the target input image, wherein the image editing model is trained using reward information, and the reward information is associated with the training output image generated by the image editing model from the training input image.

[0073] In some embodiments, the device 500 also includes a model training module, which is configured to train the image editing model, and the model training module includes: a training output image generation module, which is configured to generate a training output image using the image editing model based on the training input image, the training editing area in the training input image and the descriptive information about the image editing target; a reward information generation module, which is configured to generate at least one reward for evaluating the training output image as reward information based on the training output image; a target loss determination module, which is configured to determine the target loss for the image editing model based on the at least one reward; and a model parameter determination module, which is configured to determine the model parameters of the image editing model using the target loss.

[0074] In some embodiments, the at least one reward includes one or more of: a matching reward indicating how well the training edit region in the training output image matches the description information, a consistency reward indicating the visual consistency between the training input image and the remaining regions in the training output image excluding the training edit region, or a quality reward indicating the visually perceived quality of the training output image.

[0075] In some embodiments, the target loss determination module includes: an image difference determination module configured to determine an image difference between a training input image and a training output image; and a first loss determination module configured to obtain a first target loss based on the image difference and at least one reward.

[0076] In some embodiments, the image difference determination module is configured to determine pixel-level differences based on differences between pixels of the training input image and pixels of the training output image, or is configured to determine feature-level differences based on differences between feature representations of the training input image and feature representations of the training output image.

[0077] In some embodiments, at least a portion of the reward information is generated using a reward model, and the image editing model and the reward model are trained via an adversarial training scheme.

[0078] In some embodiments, the image editing model includes a diffusion model, and training of the diffusion model includes a first stage and a second stage, wherein the granularity of noise addition in the forward diffusion process of the first stage is finer than that of noise addition in the forward diffusion process of the second stage.

[0079] In some embodiments, the input determination module 520 includes: a noise adding module, configured to obtain a first input image from a target input image by adding noise to a target editing area, as part of a target input; a second image acquisition module, configured to obtain a second input image by removing a target editing area from the target input image, as part of a target input; and a mask generating module, configured to generate a mask for indicating the location of the target editing area, as part of the target input.

[0080] The units and / or modules included in the device 500 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 500 can be implemented at least in part by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0081] It should be understood that one or more steps in the above method may be performed by an appropriate electronic device or combination of electronic devices, such as the electronic device 110 in FIG1 .

[0082] FIG6 shows a block diagram of an electronic device 600 in which one or more embodiments of the present disclosure may be implemented. It should be understood that the electronic device 600 shown in FIG6 is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. The electronic device 600 shown in FIG6 may be used to implement the electronic device 110 of FIG1 and / or the apparatus 500 of FIG5 .

[0083] As shown in FIG6 , electronic device 600 is a general-purpose electronic device. Components of electronic device 600 may include, but are not limited to, one or more processors or processing units 610, memory 620, storage device 630, one or more communication units 640, one or more input devices 650, and one or more output devices 660. Processing unit 610 may be a real or virtual processor and is capable of performing various processes according to programs stored in memory 620. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to enhance the parallel processing capabilities of electronic device 600.

[0084] The electronic device 600 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 600, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 620 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 630 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data and can be accessed within the electronic device 600.

[0085] The electronic device 600 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG6 , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 620 may include a computer program product 625 having one or more program modules configured to perform various methods or actions of various embodiments of the present disclosure.

[0086] The communication unit 640 enables communication with other electronic devices via a communication medium. Additionally, the functions of the components of the electronic device 600 can be implemented in a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 600 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0087] The input device 650 may be one or more input devices, such as a mouse, keyboard, or trackball. The output device 660 may be one or more output devices, such as a display, a speaker, or a printer. The electronic device 600 may also communicate with one or more external devices (not shown) through the communication unit 640 as needed, such as a storage device, a display device, or the like, with one or more devices that allow a user to interact with the electronic device 600, or with any device that allows the electronic device 600 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0088] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0089] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0090] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0091] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0092] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0093] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. An image processing method, comprising: Acquire a target input image and indication information of a target editing area, wherein the target editing area is located inside or outside the input image; determining a target input for image editing based on the target input image and the instruction information; as well as Based on the target input, an image editing model is used to generate a target output image, and the target editing area is updated in the target output image compared to the target input image, wherein the image editing model is trained using reward information, and the reward information is associated with the training output image generated by the image editing model from the training input image.

2. The method of claim 1, wherein the image editing model is trained as follows: generating the training output image using the image editing model based on the training input image, the training editing region in the training input image, and the description information about the image editing target; generating, based on the training output image, at least one reward for evaluating the training output image as the reward information; determining a target loss for the image editing model based on the at least one reward; as well as Model parameters of the image editing model are determined using the target loss.

3. The method of claim 2, wherein the at least one reward comprises one or more of the following: A matching reward indicating the degree of matching between the training edited region in the training output image and the description information, a consistency reward, which indicates the visual consistency between the interior and exterior of the training edit region, or A quality reward that indicates the visually perceived quality of the training output images.

4. The method of claim 2, wherein determining the target loss comprises: determining an image difference between the training input image and the training output image; as well as The first target loss is obtained based on the image difference and the at least one reward.

5. The method of claim 4 , wherein determining the image difference comprises at least one of: determining pixel-level differences based on differences between pixels of the training input image and pixels of the training output image, or A feature-level difference is determined based on a difference between the feature representation of the training input image and the feature representation of the training output image.

6. The method according to any one of claims 1 to 5, wherein at least a portion of the reward information is generated using a reward model, and the image editing model and the reward model are trained through an adversarial training mode.

7. The method according to any one of claims 1 to 5, wherein the image editing model includes a diffusion model, and the training of the diffusion model includes a first stage and a second stage, the granularity of noise addition in the forward diffusion process of the first stage being finer than the granularity of noise addition in the forward diffusion process of the second stage.

8. The method according to any one of claims 1 to 5, wherein determining the target input for image editing based on the target input image and the instruction information comprises: deriving a first input image from the target input image as a portion of the target input by adding noise to the target editing region; Obtaining a second input image as a portion of the target input by removing the target editing area from the target input image; as well as A mask is generated as part of the target input to indicate the location of the target editing region.

9. An apparatus for image processing, comprising: an acquisition module configured to acquire a target input image and indication information of a target editing area, wherein the target editing area is located inside or outside the input image; an input determination module, configured to determine a target input for image editing based on the target input image and the instruction information; as well as An image generation module is configured to generate a target output image based on the target input using an image editing model, wherein the target editing area is updated in the target output image compared to the target input image, wherein the image editing model is trained using reward information associated with a training output image generated by the image editing model from a training input image.

10. An electronic device comprising: at least one processing unit; as well as At least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions causing the electronic device to perform the method according to any one of claims 1 to 8 when executed by the at least one processing unit.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program, wherein the computer program implements the method according to any one of claims 1 to 8 when executed by a processor.

Images