Image local editing method and device, readable storage medium and program product

Abstract

The invention relates to an image local editing method and device, a readable storage medium and a program product. The method comprises the steps of obtaining a to-be-edited image, wherein the to-be-edited image comprises an editable area and a content retention area; obtaining a description text for the editable area; performing down-sampling on the to-be-edited image to obtain a down-sampled image; performing image generation according to the down-sampling image and the description text to obtain an intermediate image; the content of an image block corresponding to the position of the editable region in the intermediate image is matched with the description text; performing up-sampling on the intermediate image to the image size of the to-be-edited image to obtain an up-sampled image; and fusing an image block corresponding to the position of the editable region in the up-sampling image with the content retention region to obtain a target image. By adopting the method, the image local editing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus, computer equipment, computer-readable storage medium and computer program product for local image editing. Background Art

[0002] With the development of computer technology, the demand for image processing is becoming increasingly diverse. In some scenarios, there is a need to edit a local area in an image. In order to edit a local area in an image, an image processing application is usually used to manually edit the image. For example, in order to add a new element, such as a person or an object, to a local area in an image, an image block representing the element can be manually selected from a configured material library, or an image block representing the element can be obtained from other images by cutting out the image, and the obtained image block can be pasted into the local area of ​​the image.

[0003] However, when partially editing an image, manually editing the image using an image processing application depends on the user's image processing experience and has the problem of low processing efficiency. Summary of the invention

[0004] Based on this, the present application provides a method, apparatus, computer device, computer-readable storage medium and computer program product for local image editing, which can improve the processing efficiency during local image editing.

[0005] On the one hand, the present application provides a method for local image editing, comprising:

[0006] Acquire an image to be edited, wherein the image to be edited includes an editable area and a content reserved area;

[0007] Get the description text for the editable area;

[0008] Downsampling the image to be edited to obtain a downsampled image;

[0009] An image is generated according to the downsampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the position of the editable area in the intermediate image matches the description text;

[0010] Upsampling the intermediate image to the image size of the image to be edited to obtain an upsampled image;

[0011] The image block corresponding to the position of the editable area in the up-sampled image is merged with the content reserved area to obtain a target image.

[0012] In one embodiment, the step of generating an image according to the downsampled image and the description text to obtain an intermediate image comprises:

[0013] Obtaining a binary mask matching the image size of the downsampled image, wherein pixels in an area corresponding to the position of the editable area in the binary mask are zero pixels;

[0014] Adding noise to the downsampled image to obtain a noise-added feature;

[0015] Combining a feature obtained by performing a matrix multiplication operation on the binary mask and the noisy feature with the noisy feature and the binary mask to obtain a combined feature;

[0016] Based on the combined features and the description text, an image is generated through a trained diffusion model to obtain an intermediate image.

[0017] In one embodiment, the area corresponding to the position of the area to be edited in the downsampled image is zero pixel; the image generation based on the combined features and the description text is performed through the trained diffusion model to obtain the intermediate image, including:

[0018] Through the trained conditional control network, feature extraction is performed according to the downsampled image to obtain control features; the control features are used to instruct the diffusion model to generate an image of the area corresponding to the position of the editable area in the downsampled image;

[0019] According to the control feature, the combined feature and the description text, an image is generated through a trained diffusion model to obtain an intermediate image.

[0020] In one embodiment, obtaining the description text for the editable area includes:

[0021] In response to a voice trigger event, receiving voice input;

[0022] Convert the speech into text to obtain an initial text;

[0023] The initial text is expanded to obtain a description text for the editable area.

[0024] In one of the embodiments, the step of fusing the image block corresponding to the editable area in the upsampled image with the content reserved area to obtain the target image includes:

[0025] splicing the image block corresponding to the editable area position in the upsampled image with the content reserved area to obtain a spliced ​​image; the spliced ​​image includes multiple color channel images;

[0026] For each color channel image, performing Poisson fusion processing on an image block corresponding to the editable area position in the color channel image and an image block corresponding to the content reserved area position in the color channel image to obtain a fused image corresponding to the color channel image;

[0027] The fused images corresponding to the multiple color channels are combined to obtain a target image.

[0028] In one embodiment, obtaining the image to be edited includes:

[0029] Acquire an input original image, and in response to a region selection operation, determine a region selected by the region selection operation in the original image;

[0030] Based on the selected area and the original image, an image to be edited is determined, and a position of an editable area in the image to be edited corresponds to a position of the selected area.

[0031] In one embodiment, the method further comprises:

[0032] When the original image meets a preset resolution constraint condition and a preset noise constraint condition, and the original image does not belong to a preset negative category, obtaining direction data in metadata of the original image;

[0033] According to the direction data, rotation correction is performed on the original image to obtain a preprocessed image;

[0034] The step of determining the image to be edited based on the selected area and the original image includes:

[0035] Based on the selected area and the pre-processed image, an image to be edited is determined.

[0036] In one aspect, the present application also provides a device for partially editing an image, comprising:

[0037] An acquisition module is used to acquire an image to be edited, wherein the image to be edited includes an editable area and a content reserved area; and acquire a description text for the editable area;

[0038] A down-sampling module, used for down-sampling the image to be edited to obtain a down-sampled image;

[0039] A generating module, configured to generate an image according to the down-sampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the position of the editable area in the intermediate image matches the description text;

[0040] An upsampling module, used for upsampling the intermediate image to the image size of the image to be edited, to obtain an upsampled image;

[0041] A fusion module is used to fuse the image block corresponding to the position of the editable area in the upsampled image with the content retention area to obtain a target image.

[0042] In one aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0043] Acquire an image to be edited, wherein the image to be edited includes an editable area and a content reserved area;

[0044] Get the description text for the editable area;

[0045] Downsampling the image to be edited to obtain a downsampled image;

[0046] An image is generated according to the downsampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the position of the editable area in the intermediate image matches the description text;

[0047] Upsampling the intermediate image to the image size of the image to be edited to obtain an upsampled image;

[0048] The image block corresponding to the position of the editable area in the up-sampled image is merged with the content reserved area to obtain a target image.

[0049] In one aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0050] Acquire an image to be edited, wherein the image to be edited includes an editable area and a content reserved area;

[0051] Get the description text for the editable area;

[0052] Downsampling the image to be edited to obtain a downsampled image;

[0053] An image is generated according to the downsampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the position of the editable area in the intermediate image matches the description text;

[0054] Upsampling the intermediate image to the image size of the image to be edited to obtain an upsampled image;

[0055] The image block corresponding to the position of the editable area in the up-sampled image is merged with the content reserved area to obtain a target image.

[0056] In one aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the following steps:

[0057] Acquire an image to be edited, wherein the image to be edited includes an editable area and a content reserved area;

[0058] Get the description text for the editable area;

[0059] Downsampling the image to be edited to obtain a downsampled image;

[0060] An image is generated according to the downsampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the position of the editable area in the intermediate image matches the description text;

[0061] Upsampling the intermediate image to the image size of the image to be edited to obtain an upsampled image;

[0062] The image block corresponding to the position of the editable area in the up-sampled image is merged with the content reserved area to obtain a target image.

[0063] The above-mentioned local image editing method, device, readable storage medium and program product perform image generation based on a downsampled image obtained by downsampling the image to be edited and a description text for the editable area to obtain an intermediate image. The content of the image block corresponding to the position of the editable area in the intermediate image matches the description text, and the local editing of the image to be edited can be automatically realized as required, without the need for the user to manually edit the image, and the image editing efficiency is high; moreover, before the image generation, the image to be edited is downsampled, which can improve the generation efficiency in the image generation process, and after the image generation, the generated intermediate image is upsampled, and the image block corresponding to the position of the editable area in the upsampled image is merged with the content retention area, so that the obtained target image retains the content retention area of ​​the image to be edited to the greatest extent, while ensuring that the image resolution of the target image is close to that of the image to be edited to the greatest extent, the processing efficiency during local image editing is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of a flow chart of a method for local image editing in one embodiment;

[0066] Figure 2 A schematic diagram of a local image editing framework in one embodiment;

[0067] Figure 3 A schematic diagram of a pre-processing step in an embodiment;

[0068] Figure 4 A schematic diagram of a text processing flow in an embodiment;

[0069] Figure 5 A schematic diagram of an image generation framework in one embodiment;

[0070] Figure 6 A schematic diagram of the image fusion process in one embodiment;

[0071] Figure 7 A comparative schematic diagram of an example of partial editing of an image in one embodiment;

[0072] Figure 8 is a structural block diagram of a device for local image editing in one embodiment;

[0073] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0074] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0075] In one embodiment, Figure 1 As shown, a method for local editing of an image is provided. This embodiment takes the method applied to a computer device as an example. The computer device can be a terminal or a server. It can be understood that the method can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be a personal computer, a laptop, a smart phone, a tablet computer or other. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. In this embodiment, the method includes the following steps 102 to 112:

[0076] Step 102: Acquire the image to be edited, where the image to be edited includes an editable area and a content reserved area.

[0077] The image to be edited is an image that needs to be partially edited. The editable area is the target area that needs to be edited. The content reserved area is the area that does not need to be edited. During the image generation process, the content of the editable area can be edited, and the content of the content reserved area can be reserved.

[0078] Exemplarily, the computer device may obtain an input original image, and in response to the area selection operation, determine the area selected by the area selection operation in the original image, use the original image as the image to be edited, and determine the selected area as the editable area in the image to be edited, and determine other areas outside the editable area as the content retention area. The area selection operation is an operation of interactively specifying a local area to be edited. The area selection operation may be an automatic selection operation, a frame selection operation, a circle selection operation, a smear operation, or other interactive operations.

[0079] Step 104: Obtain description text for the editable area.

[0080] The description text is used to describe the content in the editable area after the editable area is edited.

[0081] Exemplarily, the computer device may obtain input text and determine the input text as description text for the editable area.

[0082] In one embodiment, the computer device can obtain the input text, expand the content of the input text, and obtain the description text for the editable area. The content expansion is to expand the input text with more detailed content.

[0083] Step 106: downsample the image to be edited to obtain a downsampled image.

[0084] Downsampling is a process of reducing the size of the image to be edited so as to reduce the resolution of the image to be edited.

[0085] Exemplarily, the image to be edited is downsampled to an image size supported by an image generation model to obtain a downsampled image. The image generation model is used to generate images. The image generation model can be a diffusion model (DiffusionModels), a generative adversarial network (GANs) or others. In the downsampling process, the size of the image to be edited can be proportionally reduced according to the image size supported by the image generation model.

[0086] Step 108, generating an image based on the downsampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the editable area position in the intermediate image matches the description text.

[0087] The relative position of the image block corresponding to the editable region in the intermediate image is the same as the relative position of the editable region in the image to be edited. Since the resolution of the image to be edited is usually high, after downsampling the image to be edited, image generation is performed based on the obtained downsampled image, which can reduce the consumption of computing resources in the image generation process, thereby improving generation efficiency.

[0088] Exemplarily, the computer device may generate an image based on the downsampled image and the description text by using a trained image generation model to obtain an intermediate image.

[0089] In one embodiment, the computer device may obtain a binary mask that matches the image size of the downsampled image, and generate an image using a trained image generation model based on the downsampled image, the binary mask, and the description text to obtain an intermediate image.

[0090] The binary mask is a binary image used to represent the position corresponding to the editable area. The pixels in the area corresponding to the editable area position in the binary mask are zero pixels. A zero pixel is a pixel with a pixel value of zero. The binary mask can be obtained by downsampling the above-mentioned interactive mask to the image size of the downsampled image. The relative position of the area corresponding to the editable area position in the binary mask in the binary mask is the same as the relative position of the editable area in the image to be edited.

[0091] Step 110 : up-sample the intermediate image to the image size of the image to be edited to obtain an up-sampled image.

[0092] Among them, upsampling is the process of enlarging the image size of the intermediate image.

[0093] For example, the computer device may use a preset interpolation algorithm to upsample the intermediate image to the image size of the image to be edited to obtain an upsampled image, wherein the preset interpolation algorithm may be a nearest neighbor interpolation algorithm, a bilinear interpolation algorithm or others.

[0094] Step 112: Merge the image block corresponding to the editable area in the up-sampled image with the content reserved area to obtain a target image.

[0095] The relative position of the image block corresponding to the position of the editable area in the upsampled image is the same as the relative position of the editable area in the image to be edited.

[0096] Exemplarily, the computer device may perform Poisson fusion processing on the image block corresponding to the editable area position in the upsampled image and the content retention area to obtain the target image.

[0097] In the above-mentioned local image editing method, image generation is performed based on a downsampled image obtained by downsampling the image to be edited and a description text for the editable area to obtain an intermediate image. The content of the image block corresponding to the position of the editable area in the intermediate image matches the description text, so that the local editing of the image to be edited can be automatically realized as required, without the need for the user to manually edit the image, and the image editing efficiency is high; moreover, the image to be edited is downsampled before image generation, which can improve the generation efficiency in the image generation process, and the generated intermediate image is upsampled after the image is generated, and the image block corresponding to the position of the editable area in the upsampled image is merged with the content retention area, so that the obtained target image retains the content retention area of ​​the image to be edited to the greatest extent, while ensuring that the image resolution of the target image is close to that of the image to be edited to the greatest extent, the processing efficiency during local image editing is improved.

[0098] In an exemplary embodiment, step 108 includes: obtaining a binary mask that matches the image size of the downsampled image, in which the pixels in the area corresponding to the editable area position are zero pixels; adding noise to the downsampled image to obtain noisy features; combining the features obtained by matrix multiplication of the binary mask and the noisy features with the noisy features and the binary mask to obtain combined features; and generating an image based on the combined features and the description text through a trained diffusion model to obtain an intermediate image.

[0099] Among them, the diffusion model generates data by simulating the forward diffusion process and the reverse diffusion process. The forward diffusion process starts from a clear image without adding noise, and gradually adds noise, so that the clear image gradually transforms into a noise distribution. The reverse diffusion process starts from the noise distribution transformed by the forward diffusion process, and gradually restores the original clear image or generates a new image. In the reverse diffusion process, the diffusion model learns how to restore the original clear image or generate a new image from the noisy data, which can be achieved by training the diffusion model. Adding noise to the downsampled image can be understood as a forward diffusion process, and image generation can be understood as a reverse diffusion process. The diffusion model can specifically be a latent diffusion model (Latent Diffusion Models, LDMs, the diffusion model provided in the paper High-Resolution Image Synthesis with Latent Diffusion Models).

[0100] Noise addition can be done by adding Gaussian noise. When adding noise, Gaussian noise can be added in multiple time steps. In each time step, after adding Gaussian noise, the noise-added feature z_t of the time step is obtained, z_t=sqrt(1-β_t)*z+sqrt(β_t)*ε, where ε is the standard Gaussian distribution, β_t is the noise variance at time step t, sqrt represents the square root operation, * represents the multiplication operation, and z represents the noise-added feature obtained in the previous time step.

[0101] The noised feature can be in the data form of a feature map. The combination can be a cascade process of the feature obtained by performing matrix multiplication operation on the binary mask and the noised feature, the feature obtained by performing matrix multiplication operation on the binary mask and the noised feature, and the feature obtained by performing matrix multiplication operation on the binary mask and the noised feature, which can be understood as a process of splicing features in the channel direction. The noised feature can be a 4-channel feature map, and the binary mask can be understood as a single-channel feature map. Then, the feature obtained by performing matrix multiplication operation on the binary mask and the noised feature can be a 4-channel feature map, and the combined feature obtained by combining can be a 9-channel feature map. The combined feature can be understood as the input of the diffusion model, and the description text can be understood as the prompt of the diffusion model.

[0102] In this embodiment, since the pixels in the area corresponding to the position of the editable area in the binary mask are zero pixels, the position corresponding to the editable area can be clearly identified, and the noise-added features, the binary mask, and the features obtained after matrix multiplication of the binary mask and the noise-added features are combined. The obtained combined features provide richer input features for the subsequent diffusion model to generate images, which can improve the accuracy of image generation.

[0103] In one embodiment, the computer device may input the downsampled image into an encoder in a trained autoencoder model to obtain four-channel features, and perform noise addition on the four-channel features for multiple time steps to obtain noisy features. The autoencoder model may be an AutoEncoder, including an encoder and a decoder.

[0104] In one embodiment, the computer device may input the description text into a trained text encoder to obtain text features, and based on the combined features and the text features, perform image generation through a trained diffusion model to obtain an intermediate image. The text encoder may be a CLIP (Contrastive Language-Image Pre-Training) model.

[0105] In an exemplary embodiment, the above-mentioned step of generating an image based on the combined features and the descriptive text through a trained diffusion model to obtain an intermediate image includes: extracting features based on the downsampled image through a trained conditional control network to obtain control features; the control features are used to instruct the diffusion model to generate an image in an area corresponding to the position of the editable area in the downsampled image; and generating an image based on the control features, the combined features and the descriptive text through the trained diffusion model to obtain an intermediate image.

[0106] Among them, the conditional control network is used to control the image generation process of the diffusion model. The conditional control network can be ControlNet (the network proposed in the paper Adding Conditional Control to Text-to-Image Diffusion Models). When performing feature extraction through ControlNet, the downsampled image can be used as the Condition information of ControlNet. ControlNet can extract features based on the Condition information, the Input of the diffusion model (the above-mentioned combined features), and the Prompt of the diffusion model (the above-mentioned description text), and output control features, which act on the Unet network of the diffusion model. It can be understood that the features output by ControlNet for use by the diffusion model are control features.

[0107] The Unet network of the diffusion model may include multiple Encoder Blocks and Decoder Blocks corresponding to each EncoderBlock. ControlNet may include a convolution layer with a convolution kernel size of 1*1 and multiple trainable Encoder Block copies. In ControlNet, after the Condition information is subjected to feature extraction through a 1*1 convolution layer, it is concatenated with the Input of the diffusion model and the Prompt of the diffusion model. Feature extraction is performed through multiple trainable Encoder Block copies to obtain the control features output by each Encoder Block copy. The control features can be input into the corresponding Decoder Block, thereby controlling the generation process of the diffusion model.

[0108] Since the area corresponding to the position of the area to be edited in the downsampled image is zero pixel, different from the area corresponding to the position of the content retention area, the control features obtained by feature extraction by ControlNet can guide the diffusion model to generate images in the area corresponding to the position of the area to be edited in the downsampled image.

[0109] In this embodiment, control features are extracted through a conditional control network, and the control features can instruct the diffusion model to generate images in the area corresponding to the editable area position in the downsampled image. Therefore, according to the control features, combined features and descriptive text, image generation is performed through the trained diffusion model, which can focus on image generation in the area corresponding to the editable area position, while retaining the image content corresponding to the content retention area, thereby improving the accuracy of image generation.

[0110] In one embodiment, the above-mentioned step of generating an image based on a downsampled image and a description text to obtain an intermediate image may include: a computer device may obtain a binary mask that matches the image size of the downsampled image; input the downsampled image into an encoder in a trained autoencoder model to obtain four-channel features, and perform noise addition on the four-channel features for multiple time steps to obtain noisy features; combine the features obtained by matrix multiplication of the binary mask and the noisy features with the noisy features and the binary mask to obtain combined features; input the description text into a trained text encoder to obtain text features; perform feature extraction based on the downsampled image through a trained conditional control network to obtain control features; input the control features, combined features, and text features into a trained diffusion model to generate an image to obtain output features of the diffusion model, and input the output features into a decoder in the autoencoder model to obtain an intermediate image output by the decoder.

[0111] In an exemplary embodiment, step 104 includes: receiving input voice in response to a voice trigger event; converting the voice into text to obtain initial text; and expanding the content of the initial text to obtain description text for the editable area.

[0112] Among them, the voice trigger event can be an automatic trigger event, for example, it is automatically triggered when the wake-up voice is received, and the wake-up voice can be the pronunciation of the wake-up word, for example, the wake-up word can be "voice assistant". The voice trigger event can also be a manual trigger operation, for example, a trigger operation on the voice input function control displayed in the interface. Converting speech to text can be achieved by calling a speech-to-text interface, for example, by calling a speech-to-text interface in a mobile terminal.

[0113] The initial text is the text obtained by converting speech. Due to the different description granularity of the text input by the user (the initial text obtained by speech conversion or the text input directly), a few users can provide detailed text, such as "Please make the dog's eyes brighter and more watery, slightly enlarge the pupil, so that it looks more well-behaved and cute", while most users input relatively short text, usually only indicating the category of the object to be edited, such as the text input is "dog eyes". In the process of image generation through the diffusion model, it is found that the detail of the description text will significantly affect the generation effect. The meticulous and detailed description text helps the diffusion model to accurately grasp the user's intention and generate images with higher matching and better quality. The simple description text cannot provide enough semantic clues, making the image content generated by the diffusion model lack of pertinence and the effect is relatively poor. Therefore, by expanding the content of the initial text, the description text obtained can guide the diffusion model to generate better images.

[0114] Content expansion can be achieved through a trained large language model. Specifically, the text to be expanded can be input into the trained large language model, and the description text obtained by the large language model for content expansion can be output. The large language model can be GPT-4 (Generative Pre-Training Transformer 4, a Transformer-based pre-trained language model launched by OpenAI), LLama-3 (a large language model based on transformer structure and pre-trained with large-scale corpus, released by Meta), or others. For example, inputting the text "eyes" into LLama-3 can output a richer and more detailed description text related to the text "eyes", such as "The eyes are bright, clear and expressive, with enlarged pupils that give them a cute and innocent appearance". LLama-3 generates text in an autoregressive manner and is good at extending a coherent, rich, and relevant text description based on a given input topic. By using LLama-3 as a pre-description enhancement component, the dependence on the user's description ability is reduced. Even if a concise noun input is used, it can guide the diffusion model to output high-quality image results after expansion. In order to further improve the accuracy of descriptive text, LLama-3 can be continuously trained on larger-scale domain data for actual application scenarios to further enhance the ability to generate descriptive text in the corresponding vertical field.

[0115] In this embodiment, the input voice is converted into initial text without the need for the user to manually input text. Furthermore, the content of the initial text is expanded to obtain description text for the editable area, making the content of the description text more detailed and rich, which can improve the accuracy of subsequent image generation.

[0116] In one embodiment, a computer device may obtain a response to a voice trigger event, receive input voice, convert the voice into text, and obtain initial text, or receive input text and use the received text as the initial text; perform text similarity calculation on the initial text and text in a preconfigured sensitive text library, and when it is determined that the initial text is not similar to the text in the sensitive text library, perform content expansion on the initial text to obtain description text for the editable area.

[0117] When calculating text similarity, the initial text and the text in the sensitive text library can be converted into vectors through the Bag of Words Model, and then the cosine similarity between the vector corresponding to the initial text and the vector corresponding to the text in the sensitive text library is calculated. When the calculated cosine similarity is less than the preset similarity, it can be determined that the initial text is not similar to the text in the sensitive text library. The preset similarity is, for example, 0.8.

[0118] In an exemplary embodiment, step 112 includes: splicing the image blocks corresponding to the editable area positions in the upsampled image with the content retention area to obtain a spliced ​​image; the spliced ​​image includes multiple color channel images; for each color channel image, performing Poisson fusion processing on the image blocks corresponding to the editable area positions in the color channel image and the image blocks corresponding to the content retention area positions in the color channel image to obtain a fused image corresponding to the color channel image; combining the fused images corresponding to each of the multiple color channels to obtain a target image.

[0119] Wherein, if the image block corresponding to the editable area position in the upsampled image is the same size as the editable area, the image block corresponding to the editable area position in the upsampled image is spliced ​​with the content retention area to form a spliced ​​image with the same size as the image to be edited. The spliced ​​image can also be understood as an image obtained by replacing the editable area in the image to be edited with the image block corresponding to the editable area position in the upsampled image.

[0120] The multiple color channel images may include an R (Red) channel image, a G (Green) channel image, and a B (Blue) channel image. When performing Poisson fusion processing, the fused image after Poisson fusion processing can be obtained by solving the Poisson equation. The Poisson equation can be referred to in the following formula (1).

[0121] Formula (1)

[0122] Among them, formula (1) can express that the fused image obtained after fusion satisfies And meet . It can represent the fused image to be solved. It can represent the gradient field of the color channel image. The image block corresponding to the editable area position can be represented. Characterizable The boundary conditions, Can represent surrounding the border. The Laplace operator can be characterized as the sum of second-order partial derivatives. In a two-dimensional image, the Laplace operator describes the local curvature of each pixel in the image. The divergence operator can be characterized, which describes the divergence of the gradient field.

[0123] The meaning can be that in Internal, fused image The result of the Laplace operator calculation is equal to the divergence of the gradient field of the color channel image. In other words, the Laplace operator is used to calculate the fused image. The calculation result obtained by calculating the image block corresponding to the editable area position in the color channel image is equal to the divergence of the gradient field of the image block corresponding to the editable area position in the color channel image. This makes the image block corresponding to the editable area position after fusion maintain the same gradient structure as the image block corresponding to the editable area position in the original image (the color channel image), thereby maintaining the texture characteristics of the original image in the area corresponding to the editable area position.

[0124] The meaning can be that in At, fused image Boundary conditions The pixels at the boundary are the same as the pixels at the boundary of the color channel image, that is, the surrounding fused image The pixels at the boundary of the image block corresponding to the editable area position in the image are the same as the pixels at the boundary of the image block corresponding to the editable area position in the surrounding color channel image, so that there is a smooth transition at the boundary after fusion.

[0125] In this embodiment, there are obvious stitching traces in the stitched image. The fused image corresponding to each color channel image is obtained through Poisson fusion processing, and then the fused images corresponding to multiple color channels are combined to obtain the target image, so that in the target image, the part corresponding to the position of the content retention area is the same as the content of the content retention area, and the original resolution and detail texture of the image to be edited are retained. The image block corresponding to the position of the editable area has a smooth transition with the part corresponding to the position of the content retention area in the target image while retaining the gradient structure of the image block corresponding to the position of the editable area in the up-sampled image corresponding to the generated intermediate image, and the target image is more realistic and natural.

[0126] In an exemplary embodiment, step 102 includes: obtaining an input original image, and in response to a region selection operation, determining a region selected by the region selection operation in the original image; based on the selected region and the original image, determining an image to be edited, wherein the position of the editable region in the image to be edited corresponds to the position of the selected region.

[0127] The area selection operation may be an automatic selection operation, a frame selection operation, a circle selection operation, a smear operation or other interactive operations.

[0128] In this embodiment, the user can select an area in the original image through the area selection operation, so that the editable area in the image to be edited meets the user's intention, so that subsequent editing can be performed in the editable area according to the user's intention, thereby improving the accuracy of image generation.

[0129] In one embodiment, the step of determining the image to be edited based on the selected area and the original image may include: the computer device may create an interactive mask that matches the size of the original image based on the selected area, and perform a matrix operation on the interactive mask and the original image to obtain the image to be edited.

[0130] The area selection operation may be an automatic selection operation, a box selection operation, a circle selection operation, a smear operation or other interactive operations. The automatic selection operation may be a trigger operation for the smart selection control. When the area selection operation is an automatic selection operation, the computer device may automatically segment the image and automatically select the area occupied by the instance object in the image.

[0131] The interactive mask is a binary image used to represent the position corresponding to the selected area. The matrix operation can be a matrix multiplication operation. Specifically, in the interactive mask, the first area corresponding to the selected area position can be zero pixels, and the second area outside the first area can be a non-zero preset pixel, and the preset pixel can be a pixel with a pixel value of 1. After the interactive mask is subjected to a matrix multiplication operation with the original image, an image to be edited is obtained. In the image to be edited, the area corresponding to the selected area position is used as an editable area, and the pixels in the editable area are zero pixels. The pixels in the content reserved area outside the editable area are the same as the pixels in the unselected area in the original image. The relative position of the first area in the interactive mask is the same as the relative position of the selected area in the original image.

[0132] In one embodiment, when the area selection operation is a smearing operation, the computer device can respond to the smearing operation performed on the screen where the original image is located, obtain the smearing area, adaptively identify the object to which the smearing area belongs based on the smeared area, and automatically smear the area occupied by the object on the basis of the smeared area, and respond to the area confirmation operation, determine the area in the screen where the original image is smeared as the area selected in the original image by the area selection operation. Among them, the smearing operation can be generated by the user interacting with the screen displaying the original image. The object refers to the instance category to which the area belongs, for example, a person, sky, hair, skin, eyes or others. Identifying the object to which the smeared area belongs can be achieved based on fine-grained image segmentation technology. In this embodiment, since the object to which the user's smeared area belongs can be identified and the area occupied by the object can be automatically smeared, there is no need for the user to manually outline irregular boundaries, and complex shapes can be accurately selected. Irregular areas with rich details such as hair and textures can be selected.

[0133] In an exemplary embodiment, the above-mentioned local image editing method may include the following steps: when the original image meets the preset resolution constraint conditions, the preset noise constraint conditions, and the original image does not belong to the preset negative category, the direction data in the metadata of the original image is obtained; according to the direction data, the original image is rotationally corrected to obtain a preprocessed image; the above-mentioned step of determining the image to be edited based on the selected area and the original image may include: based on the selected area and the preprocessed image, determining the image to be edited, the position of the editable area in the image to be edited corresponds to the position of the selected area.

[0134] Among them, the preset resolution constraint condition is that the resolution of the original image reaches the preset resolution. The preset resolution is, for example, 1024*1024. The preset noise constraint condition is that the noise level evaluation value of the original image meets the preset low noise condition. The noise level evaluation value can be the noise variance, and the noise level evaluation value meets the preset low noise condition, which can be that the noise level evaluation value is less than the preset noise variance, and the preset noise variance is, for example, 0.02. The noise level evaluation value can also be the PSNR (Peak Signal to Noise Ratio) value between the preset standard image and the original image, and the noise level evaluation value meets the preset low noise condition, which can be that the noise level evaluation value is greater than the preset peak signal-to-noise ratio value, and the preset peak signal-to-noise ratio value is, for example, 30dB.

[0135] The preset negative category is a category of images containing negative information. Negative information includes bad information, sensitive information, inappropriate content, etc. When determining whether an original image belongs to the preset negative category, an image recognition algorithm may be used to determine the category of the original image. Specifically, the image recognition algorithm may extract features from the original image and the image of the preset negative category, and analyze the extracted features to determine whether the original image belongs to the preset negative category.

[0136] The metadata of the original image is used to record the attribute information of the original image. The metadata of the original image may include shooting equipment, shooting parameters, time, location, direction data or other data. The metadata of the original image may specifically be Exif (Exchangeable Image File Format) metadata. Exif metadata may be extracted by an image processing library, such as PIL (Python Image Library) and ImageMagick (an open source command line tool). Specifically, the "_getexif ()" function in PIL may be used to extract Exif metadata.

[0137] The orientation data may be the value of the Orientation tag in the Exif metadata. The value of the Orientation tag may include values ​​representing normal orientation, horizontal flip, 180-degree clockwise rotation, vertical flip, 90-degree clockwise rotation and horizontal flip, 90-degree clockwise rotation, 90-degree clockwise rotation and vertical flip, and 90-degree counterclockwise rotation. Corresponding one to one to the values ​​of the above-mentioned Orientation tags, the rotation corrections required for the original image are, respectively, no correction required, horizontal flip, 180-degree clockwise rotation, vertical flip, 90-degree clockwise rotation and horizontal flip, 90-degree clockwise rotation, 90-degree clockwise rotation and vertical flip, and 90-degree counterclockwise rotation. It can be understood that when the orientation data is a value representing the normal orientation, there is no need to perform rotation correction on the original image.

[0138] In this embodiment, the user can specify the selected area in the original image through the area selection operation. Then, when the original image meets the preset resolution constraint conditions and the preset noise constraint conditions, and the original image does not belong to the preset negative category, it means that the original image has high clarity and low noise level, and will not produce negative effects, and the original image quality is high. At this time, the direction data in the metadata of the original image is obtained, and the original image is rotationally corrected according to the direction data to obtain a preprocessed image to ensure that the image is in a normal direction. Based on the selected area and the preprocessed image, the image to be edited is determined, which can ensure the quality of the image to be edited and point out the editable area to facilitate subsequent image generation.

[0139] In one embodiment, the above-mentioned step of determining the image to be edited based on the selected area and the preprocessed image may include: the computer device may create an interactive mask that matches the size of the original image based on the selected area, in which a first area corresponding to the position of the selected area is zero pixels, and a second area outside the first area is a non-zero preset pixel; the interactive mask is rotationally corrected in the same manner as the original image, and a matrix multiplication operation is performed between the corrected interactive mask and the original image to obtain the image to be edited, in which the position of the editable area in the image to be edited corresponds to the position of the selected area.

[0140] In a specific embodiment, see Figure 2 As shown in the schematic diagram of the local image editing framework, the above-mentioned local image editing method may specifically include the following steps.

[0141] See Figure 3In the schematic diagram of the preprocessing step flow shown, the terminal can obtain the input original image and perform image quality evaluation on the original image, specifically, perform resolution evaluation and noise evaluation on the original image. When the original image does not meet the preset resolution constraint, a prompt message indicating low resolution is generated to prompt the user to re-enter; when the original image does not meet the preset noise constraint, a prompt message indicating high noise is generated to prompt the user to re-enter; when the original image meets the preset resolution constraint and the preset noise constraint, the image content of the original image is reviewed, specifically, the image recognition algorithm can be used to identify whether the original image belongs to the preset negative category. When the original image belongs to the preset negative category, a prompt message indicating that negative information is contained is generated to prompt the user to re-enter, or the original image is sent to the review end for manual review by the reviewer. When the review result indicates that the original image belongs to the preset negative category, a prompt message indicating that negative information is contained is generated to prompt the user to re-enter; when the original image does not belong to the preset negative category, the direction data in the metadata of the original image is obtained. According to the direction data, the original image is rotationally corrected to obtain a preprocessed image.

[0142] The terminal can determine the area selected by the area selection operation in the original image in response to the area selection operation; based on the selected area, create an interactive mask that matches the size of the original image, in which the first area corresponding to the selected area position is zero pixels, and the second area outside the first area is a pixel with a pixel value of 1; perform rotation correction on the interactive mask in the same manner as the rotation correction of the original image, perform matrix multiplication operation on the corrected interactive mask and the original image, and obtain the image to be edited. It can be understood that the image to be edited includes an editable area and a content retention area, the editable area is zero pixels, and the content of the content retention area is the same as the content of the unselected area in the original image.

[0143] The terminal can respond to the voice trigger event, receive the input voice, convert the voice into text, and obtain the initial text, or receive the input text and use the input text as the initial text; calculate the text similarity between the initial text and the text in the pre-configured sensitive text library, and when it is determined that the initial text is not similar to the text in the sensitive text library, expand the content of the initial text to obtain the description text for the editable area. Figure 4 As shown in the text processing flow chart, when expanding the content, the initial text can be input into the LLama-3 model, and the LLama-3 model generates rich descriptive text related to the initial text based on the language instructions learned through pre-training, and the descriptive text replaces the initial text for image generation of the subsequent diffusion model.

[0144] The terminal may downsample the image to be edited to an image size supported by the trained diffusion model to obtain a downsampled image.

[0145] See Figure 5 In the image generation framework diagram shown in the figure, the terminal can obtain a binary mask matching the image size of the downsampled image, input the downsampled image into the encoder in the trained autoencoder model, and the encoder maps the downsampled image to the latent space to obtain four-channel features, and denoises the four-channel features for multiple time steps to obtain the noisy features; the features obtained by matrix multiplication of the binary mask and the noisy features are combined with the noisy features and the binary mask to obtain the combined features; the description text is input into the trained text encoder (CLIP model) to obtain the text features; the control features are extracted according to the downsampled image through the trained conditional control network (ControlNet) to obtain the control features; the control features, the combined features and the text features are input into the trained diffusion model for image generation to obtain the output features of the diffusion model, and the output features are input into the decoder in the autoencoder model to map back to the image space to obtain the intermediate image output by the decoder. The content of the image block corresponding to the editable area position in the intermediate image matches the description text.

[0146] See Figure 6 The schematic flow chart of the image fusion steps shown in the figure upsamples the intermediate image output based on the diffusion model to the image size of the image to be edited (upsamples to the original resolution of the image to be edited) to obtain an upsampled image; segments the image blocks corresponding to the editable area position from the upsampled image, segments the content retention area from the image to be edited, and splices the image blocks corresponding to the editable area position in the upsampled image with the content retention area to obtain a spliced ​​image; the spliced ​​image includes multiple color channel images (R channel image, G channel image and B channel image); for each color channel image, performs Poisson fusion processing on the image blocks corresponding to the editable area position in the color channel image and the image blocks corresponding to the content retention area position in the color channel image to obtain a fused image corresponding to the color channel image; and combines the fused images corresponding to each of the multiple color channels to obtain a target image.

[0147] See Figure 7 The following is a comparison diagram of an example of partial image editing: Figure 7 (a) can represent the original image, Figure 7 In (b), area 701 may represent an editable area, and area 702 may represent a content retention area. Figure 7 (c) The target image obtained after editing can be characterized.

[0148] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0149] Based on the same inventive concept, the embodiment of the present application also provides an image local editing device for implementing the above-mentioned image local editing method. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the image local editing device provided below can refer to the limitations of the image local editing method above, and will not be repeated here.

[0150] In an exemplary embodiment, Figure 8 As shown, a local image editing device 800 is provided, comprising: an acquisition module 810, a down-sampling module 820, a generation module 830, an up-sampling module 840 and a fusion module 850, wherein:

[0151] The acquisition module 810 is used to acquire the image to be edited, where the image to be edited includes an editable area and a content reserved area; and acquire a description text for the editable area.

[0152] The down-sampling module 820 is used to down-sample the image to be edited to obtain a down-sampled image.

[0153] The generation module 830 is used to generate an image according to the downsampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the editable area position in the intermediate image matches the description text.

[0154] The up-sampling module 840 is used to up-sample the intermediate image to the image size of the image to be edited to obtain an up-sampled image.

[0155] The fusion module 850 is used to fuse the image block corresponding to the editable area in the up-sampled image with the content reserved area to obtain a target image.

[0156] In an exemplary embodiment, the generation module 830 is also used to obtain a binary mask that matches the image size of the downsampled image, in which the pixels in the area corresponding to the editable area position are zero pixels; the downsampled image is denoised to obtain noisy features; the features obtained by matrix multiplication of the binary mask and the noisy features are combined with the noisy features and the binary mask to obtain combined features; based on the combined features and the descriptive text, an image is generated through a trained diffusion model to obtain an intermediate image.

[0157] In an exemplary embodiment, the area in the downsampled image corresponding to the position of the area to be edited is zero pixel; the generation module 830 is also used to extract features based on the downsampled image through a trained conditional control network to obtain control features; the control features are used to instruct the diffusion model to generate images in the area corresponding to the position of the editable area in the downsampled image; based on the control features, combined features and descriptive text, the image is generated through the trained diffusion model to obtain an intermediate image.

[0158] In an exemplary embodiment, the acquisition module 810 is further used to receive input voice in response to a voice trigger event; convert the voice into text to obtain initial text; and expand the content of the initial text to obtain description text for the editable area.

[0159] In an exemplary embodiment, the fusion module 850 is also used to splice the image blocks corresponding to the editable area positions in the upsampled image with the content retention area to obtain a spliced ​​image; the spliced ​​image includes multiple color channel images; for each color channel image, the image blocks corresponding to the editable area positions in the color channel image and the image blocks corresponding to the content retention area positions in the color channel image are subjected to Poisson fusion processing to obtain a fused image corresponding to the color channel image; and the fused images corresponding to each of the multiple color channels are combined to obtain a target image.

[0160] In an exemplary embodiment, the acquisition module 810 is also used to acquire the input original image, and in response to the area selection operation, determine the area selected by the area selection operation in the original image; based on the selected area and the original image, determine the image to be edited, and the position of the editable area in the image to be edited corresponds to the position of the selected area.

[0161] In an exemplary embodiment, the image local editing device 800 also includes a preprocessing module, which is used to obtain direction data in the metadata of the original image when the original image meets the preset resolution constraint conditions, the preset noise constraint conditions, and the original image does not belong to the preset negative category; according to the direction data, the original image is rotationally corrected to obtain a preprocessed image; the acquisition module 810 is also used to determine the image to be edited based on the selected area and the preprocessed image.

[0162] Each module in the above-mentioned image partial editing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module.

[0163] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for a local image editing device is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0164] Those skilled in the art will understand that Fig. 9The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0165] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0166] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0167] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0168] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0169] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0170] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0171] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for local image editing, characterized in that: The method comprises: Acquire an image to be edited, wherein the image to be edited includes an editable area and a content reserved area; Get the description text for the editable area; Downsampling the image to be edited to obtain a downsampled image; An image is generated according to the downsampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the position of the editable area in the intermediate image matches the description text; Upsampling the intermediate image to the image size of the image to be edited to obtain an upsampled image; The image block corresponding to the position of the editable area in the up-sampled image is merged with the content reserved area to obtain a target image.

2. The method according to claim 1, characterized in that The step of generating an image according to the down-sampled image and the description text to obtain an intermediate image comprises: Obtaining a binary mask matching the image size of the downsampled image, wherein pixels in an area corresponding to the position of the editable area in the binary mask are zero pixels; Adding noise to the downsampled image to obtain a noise-added feature; Combining a feature obtained by performing a matrix multiplication operation on the binary mask and the noisy feature with the noisy feature and the binary mask to obtain a combined feature; Based on the combined features and the description text, an image is generated through a trained diffusion model to obtain an intermediate image.

3. The method according to claim 2, characterized in that The area corresponding to the position of the area to be edited in the downsampled image is zero pixel; the image generation is performed based on the combined features and the description text through the trained diffusion model to obtain the intermediate image, including: Through the trained conditional control network, feature extraction is performed according to the downsampled image to obtain control features; the control features are used to instruct the diffusion model to generate an image of the area corresponding to the position of the editable area in the downsampled image; According to the control feature, the combined feature and the description text, an image is generated through a trained diffusion model to obtain an intermediate image.

4. The method according to claim 1, characterized in that The obtaining of the description text for the editable area comprises: In response to a voice trigger event, receiving voice input; Convert the speech into text to obtain an initial text; The initial text is expanded to obtain a description text for the editable area.

5. The method according to claim 1, characterized in that The step of fusing the image block corresponding to the editable area in the up-sampled image with the content reserved area to obtain a target image includes: splicing the image block corresponding to the editable area position in the upsampled image with the content reserved area to obtain a spliced ​​image; the spliced ​​image includes multiple color channel images; For each color channel image, performing Poisson fusion processing on an image block corresponding to the editable area position in the color channel image and an image block corresponding to the content reserved area position in the color channel image to obtain a fused image corresponding to the color channel image; The fused images corresponding to the multiple color channels are combined to obtain a target image.

6. The method according to any one of claims 1 to 5, characterized in that: The obtaining of the image to be edited comprises: Acquire an input original image, and in response to a region selection operation, determine a region selected by the region selection operation in the original image; Based on the selected area and the original image, an image to be edited is determined, and a position of an editable area in the image to be edited corresponds to a position of the selected area.

7. The method according to claim 6, characterized in that The method further comprises: When the original image meets a preset resolution constraint condition and a preset noise constraint condition, and the original image does not belong to a preset negative category, obtaining direction data in metadata of the original image; According to the direction data, rotation correction is performed on the original image to obtain a preprocessed image; The step of determining the image to be edited based on the selected area and the original image includes: Based on the selected area and the pre-processed image, an image to be edited is determined.

8. A device for partially editing an image, characterized in that: The device comprises: An acquisition module is used to acquire an image to be edited, wherein the image to be edited includes an editable area and a content reserved area; and acquire a description text for the editable area; A down-sampling module, used for down-sampling the image to be edited to obtain a down-sampled image; A generating module, configured to generate an image according to the down-sampled image and the description text to obtain an intermediate image; the content of the image block corresponding to the position of the editable area in the intermediate image matches the description text; An upsampling module, used for upsampling the intermediate image to the image size of the image to be edited, to obtain an upsampled image; A fusion module is used to fuse the image block corresponding to the position of the editable area in the upsampled image with the content retention area to obtain a target image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images