Image processing method, apparatus, and electronic device

Abstract

Embodiments of the present application disclose an image processing method and device and electronic equipment. The method comprises: determining a target region from a first image; obtaining first replacement content based on the first image, the target region and a first processing model, wherein the first replacement content is obtained by eliminating foreground content in first generated content from second generated content by the first processing model, the first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task; and replacing image content in the target region in the first image with the first replacement content to obtain a target image. Thus, the above method can avoid the generated first replacement content including new foreground content, so that the generated image content (first replacement content) for replacing the target region is more controllable, thereby improving the image elimination effect.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to an image processing method, apparatus, and electronic device. Background Technology

[0002] Image content removal can eliminate specific elements within an image to modify and enhance its content. One method involves replacing the content to be removed with generated content to achieve the desired effect. However, some image removal methods still suffer from unsatisfactory removal results. Summary of the Invention

[0003] In view of the above problems, this application proposes an image processing method, apparatus, and electronic device to improve the above problems.

[0004] In a first aspect, this application provides an image processing method, the method comprising: determining a target region from a first image, the target region representing a region to be processed; and obtaining first replacement content based on the first image, the target region, and a first processing model, wherein the first replacement content is obtained by the first processing model through the elimination of foreground content in the first generated content by the elimination of foreground content in the first generated content, the first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task;

[0005] The image content located in the target area of ​​the first image is replaced with the first replacement content to obtain the target image.

[0006] Secondly, this application provides an image processing apparatus, the apparatus comprising: a target region determination unit, configured to determine a target region from a first image, the target region representing a region to be processed; a content generation unit, configured to obtain first replacement content based on the first image, the target region, and a first processing model, wherein the first replacement content is obtained by the first processing model through the elimination of foreground content in the first generated content by the elimination of foreground content in the first generated content, the first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task; and an image processing unit, configured to replace the image content located in the target region in the first image with the first replacement content to obtain a target image.

[0007] Thirdly, this application provides an electronic device, which includes at least a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the above-described method.

[0008] Fourthly, this application provides a computer-readable storage medium storing program code, wherein the above-described method is executed when the program code is run by a processor.

[0009] This application provides an image processing method, apparatus, and electronic device. In this method, after determining a target region from a first image, a first replacement content can be obtained based on the first image, the target region, and a first processing model. Then, the image content located in the target region of the first image is replaced with the first replacement content to obtain a target image. The first replacement content is generated by the first processing model through a background generation task and a foreground generation task. During the generation of the first replacement content, the second generated content is used to cancel out the foreground content in the first generated content. Thus, in the process of the first processing model generating image content, in addition to performing a background generation task, a foreground generation task is also performed. This allows the second generated content obtained from the foreground generation task to eliminate the foreground content in the first generated content, thereby helping to avoid the generated first replacement content including new foreground content. This makes the generated image content (the first replacement content) used to replace the target region more controllable, improving the image removal effect. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A schematic diagram illustrating an application scenario of the image processing method proposed in this application is shown.

[0012] Figure 2 A schematic diagram illustrating another application scenario of the image processing method proposed in the embodiments of this application is shown;

[0013] Figure 3 A flowchart of an image processing method according to an embodiment of this application is shown;

[0014] Figure 4 A schematic diagram of candidate objects in the first image in an embodiment of this application is shown;

[0015] Figure 5 A schematic diagram illustrating the identification of candidate objects in an embodiment of this application is shown;

[0016] Figure 6 The illustration shows a schematic diagram of the improvements made to several improvement items in the embodiments of this application;

[0017] Figure 7 A schematic diagram of the target mask image in an embodiment of this application is shown;

[0018] Figure 8 A flowchart of an image processing method according to an embodiment of this application is shown;

[0019] Figure 9 A flowchart of an image processing method according to another embodiment of this application is shown;

[0020] Figure 10 A schematic diagram of the intersection and union regions in an embodiment of this application is shown;

[0021] Figure 11 A flowchart of an image processing method according to another embodiment of this application is shown;

[0022] Figure 12 A flowchart of an image processing method according to an embodiment of this application is shown;

[0023] Figure 13 A structural block diagram of an image processing apparatus according to an embodiment of this application is shown;

[0024] Figure 14 A structural block diagram of another electronic device for performing an image processing method according to an embodiment of the present application is shown;

[0025] Figure 15 This is a storage unit in this application embodiment for storing or carrying program code that implements the image processing method according to this application embodiment. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0027] In the field of image processing, image content removal is an important and widely applicable technique. This technique allows us to selectively remove specific elements from an image, such as blemishes, distractions, or unwanted parts, thereby modifying and enhancing the image content.

[0028] In the specific implementation of image content removal, one method is to replace the content to be removed with other content generated by a specific algorithm. This method can achieve the removal goal to a certain extent, but at the same time, some problems still exist in related image removal methods. One prominent problem is that the removal effect is not ideal. For example, in one related method, when removing elements from the original image, new image content is generated to replace the elements to be removed, in order to achieve the effect of element removal. However, the generated new image content is relatively uncontrollable. For instance, the user originally expected the element to be removed to be a certain foreground content, but the generated new image content is a different foreground content, thus failing to achieve the desired removal effect.

[0029] Therefore, after discovering the above-mentioned problems in their research, the inventors proposed the image processing method, apparatus, and electronic device described in this application, which can improve upon these problems. In this method, after determining the target region from a first image, a first replacement content can be obtained based on the first image, the target region, and a first processing model. Then, the image content located in the target region of the first image is replaced with the first replacement content to obtain the target image. The first replacement content is generated by the first processing model through a background generation task to obtain first generated content, and a foreground generation task to obtain second generated content. During the generation of the first replacement content, the second generated content is used to cancel out the foreground content in the first generated content. Thus, through the above method, in the process of generating image content by the first processing model, in addition to performing the background generation task, a foreground generation task is also performed. This allows the second generated content obtained by performing the foreground generation task to eliminate the foreground content in the first generated content. This helps to avoid the first replacement content from including new foreground content (which can be understood as foreground content other than the original foreground content in the first image). As a result, the generated image content (first replacement content) used to replace the target area is more controllable, thereby improving the image elimination effect.

[0030] Before providing a more detailed description of the embodiments of this application, an application environment related to the embodiments of this application will be introduced.

[0031] The application scenarios involved in the embodiments of this application will be introduced below.

[0032] In the embodiments of this application, the provided image processing method can be executed by an electronic device. In this manner, all steps of the image processing method provided in the embodiments of this application can be performed by the electronic device. For example, as Figure 1As shown, all steps in the image processing method provided in this application embodiment can be executed by the processor of the electronic device 100.

[0033] Alternatively, the image processing method provided in this application embodiment can also be executed by a server. Correspondingly, in this method where the method is executed by a server, the server can begin executing the steps of the image processing method provided in this application embodiment in response to a triggering instruction. This triggering instruction can be sent by an electronic device used by a user, or it can be triggered locally by the server in response to some automated event.

[0034] Furthermore, the image processing method provided in this application embodiment can also be executed collaboratively by an electronic device and a server. In this method, some steps of the image processing method provided in this application embodiment are executed by the electronic device, while other steps are executed by the server. For example, as shown... Figure 2 As shown, the electronic device 100 can perform an image processing method including: acquiring a first image and determining a target region from the first image. Then, the electronic device 100 transmits the first image and the target region determined from the first image to a server 200. After receiving the first image and the target mask image, the server 200 can perform subsequent steps to obtain the target image, and then transmit the target image back to the electronic device 100. After receiving the target image, the electronic device 100 can store, display, and share the target image. Alternatively, if the target region is represented by a target mask region, the electronic device 100 can transmit both the first image and the target mask image to the server after obtaining them. Optionally, the electronic device 100 can encode the first image and the target mask image before transmitting them to the server 200. In this case, the server 200 receives the encoded first image and the target mask image, and then needs to decode the encoded first image and the target mask image to obtain the original first image and the target mask image.

[0035] It should be noted that in this method where electronic devices and servers work together, the steps performed by the electronic devices and servers are not limited to those described in the examples above. In practical applications, the steps performed by the electronic devices and servers can be dynamically adjusted according to the actual situation.

[0036] It should be noted that the electronic equipment 100, in addition to being for Figure 1 and Figure 2Besides smartphones, the device shown can also be a tablet computer, a smart voice assistant, or other similar device. Server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud computing, cloud storage, network services, cloud communication, middleware services, CDN (Content Delivery Network), and artificial intelligence platforms. In the case where the image processing method provided in this embodiment is executed by a server cluster or distributed system composed of multiple physical servers, different steps in the image processing method can be executed by different physical servers, or can be executed in a distributed manner by servers built on a distributed system.

[0037] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0038] Please see Figure 3 This application provides an image processing method, which includes:

[0039] S110: Determine the target region from the first image, whereby the target region represents the area to be processed.

[0040] In this embodiment of the application, the first image can be understood as the image to be replaced. There are various ways to obtain the first image.

[0041] In one approach, the preview image currently displayed on the electronic device can be used as the first image. In this approach, the electronic device can capture images in real-time using a camera and display the captured images in the preview area. The user can then use the image displayed in the preview area (a single frame) as the first image. In another approach, the image obtained by the electronic device taking a picture can be used as the first image. Here, the electronic device can capture images in real-time using a camera and display the captured images in the preview area. The electronic device can also take a picture in response to a shooting command, using the captured image as the first image. In yet another approach, the electronic device can retrieve the first image from its photo album. In this approach, the electronic device can display images from the album when it is open and select an image from the album as the first image in response to the user's selection. Finally, the electronic device can use an image transmitted from another device as the first image. For example, if the user of the electronic device is chatting with a friend through an instant messaging program, and the friend sends a picture via instant messaging, the electronic device can use that sent picture as the first image.

[0042] In this context, the target region in the first image can be understood as the region used for content processing. Similarly, if the image to be processed can be understood as the image used for content replacement, then the target region can be understood as the region where the content is to be replaced.

[0043] In the embodiments of this application, there are multiple ways to determine the target area.

[0044] One approach is for the user to determine the target region in the first image. For example, if the first image includes multiple objects (e.g., multiple foreground objects), all of these objects can be considered as candidate objects. For example, such as... Figure 4 As shown, by analyzing Figure 4 To perform identification, it can Figure 4 Characters 10 and 20 are used as candidate objects, allowing the user to select one as the target object. In this embodiment, the method by which the user selects the target object from the candidate objects is not specifically limited. For example, ... Figure 5 As shown, candidate objects can be marked with dashed boxes in the first image. If a click is detected on a dashed box, the candidate object marked by the clicked dashed box can be taken as the target object. In this way, the area where the target object is located can be taken as the target area.

[0045] One approach is to determine the target region in the first image based on the current processing scenario. Optionally, the electronic device can enter a specified processing scenario in response to a user's operation. In this scenario, the region corresponding to the specified processing scenario can be used as the specified processing scenario. For example, the specified processing scenario could be a pedestrian removal scenario. In this scenario, the electronic device can treat all foreground objects (e.g., other people) in the acquired first image, except for the specified person, as target objects, and the region where the target objects are located as the target region. The electronic device can identify the specified person from the first image using facial recognition. This specified person can be the user of the electronic device or another person specified by the user. As another example, the specified processing scenario could be a person removal scenario. In this case, the region containing all people in the first image can be used as the target region.

[0046] S120: Based on the first image, the target region, and the first processing model, a first replacement content is obtained, wherein the first replacement content is obtained by the first processing model through the second generated content to eliminate the foreground content in the first generated content, the first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task.

[0047] In this embodiment, the first processing model can be used to process the input image, wherein processing the input image may include eliminating a portion of the input image. Alternatively, it may include replacing the content of a portion of the input image.

[0048] The first processing model can be a diffusion model. For example, the first processing model could be the Unet model. A diffusion model is a type of deep learning model primarily used to generate new data samples. Its basic principle is to progressively add noise to the data and then learn to recover the original data from the noise. During training, the model learns how to reverse the diffusion process, thus generating new samples similar to the original data. Diffusion models have powerful generative capabilities, capable of generating high-quality, diverse content such as images, audio, and text.

[0049] The process of obtaining the first replacement content based on the first image, the target region, and the first processing model can be understood as inputting the first image and the target region into the first processing model so that the first processing model can process the target region in the first image. Optionally, the first processing model can generate the first replacement content corresponding to the target region during the processing.

[0050] In this embodiment, the first processing model can obtain first replacement content by performing a background generation task and a foreground generation task. Specifically, the first processing model obtains first generated content by performing the background generation task and second generated content by performing the foreground generation task. The first generated content can be understood as content used as an image background; for example, the first generated content can be used as background content for a target region. The second generated content can be understood as foreground content.

[0051] It should be noted that the first processing model is an image generation model. Therefore, the generated content may have a certain degree of uncontrollability. Thus, the first generated content may still contain foreground content that the user wants to eliminate (or avoid). However, the goal of the image processing method performed by the electronic device is to eliminate foreground content within the target area. Therefore, if the second generated content is foreground content, it can be used to perform an elimination operation on the foreground content in the first generated content. The first generated content after the elimination operation can then be used as the first replacement content. This elimination operation can be understood as using the second generated content to cancel out the foreground content in the first generated content. Optionally, during the elimination operation, a content difference can be calculated between the second and first generated content to prevent new foreground content from appearing in the resulting first replacement content. This content difference can be achieved by obtaining the color difference between corresponding pixels in the second and first generated content. Corresponding pixels can include having the same coordinate position in the image and / or the same depth information.

[0052] In one approach, the first processing model can obtain the difference between the first generated content and the second generated content, and then derive the first replacement content based on this difference and the first generated content. Optionally, the first replacement content can be obtained based on the following formula:

[0053] C = A + scale * (AB)

[0054] In the above formula, A represents the first generated content, B represents the second generated content, and C represents the first replacement content. `scale` is a preset value used to control the intensity of the foreground generation task's influence on the final result. By adjusting the value of `scale`, the contributions of the background completion content (first generated content) and the foreground generated content (second generated content) in the final result (first replacement content) can be balanced, thus obtaining a more desirable output. Specifically, the final result C is based on the first generated content A, with an adjustment factor added. This adjustment factor is obtained by multiplying the difference (AB) between the second generated content B and the first generated content A by a weight (`scale`).

[0055] In one approach, the second generated content is obtained by the first processing model performing a foreground generation task based on negative prompts. The negative prompts represent foreground content to be avoided. It should be noted that the negative prompts are used to reduce the occurrence of the desired avoided foreground content in the generated second content.

[0056] Optionally, when inputting content into the first processing model, in addition to the first image and the target region, background generation flags and foreground generation flags can also be input. The background generation flag guides the first processing model to perform a background generation task, and the foreground generation flag guides it to perform a foreground generation task. In this approach, the reverse prompts can include foreground generation flags. The correspondence between the foreground generation flags and the foreground content is established during the training process of the first processing model. The foreground content corresponding to the foreground generation flag can be understood as the foreground content that is avoided during the foreground generation task. Optionally, the foreground content corresponding to the foreground generation flag can be of multiple types. The training process of the first processing model can be understood as the process of training the model to be trained to obtain the first processing model.

[0057] In one approach, the training process can be divided into two tasks: background completion (i.e., background generation) and foreground generation. The background completion task can be denoted as the Context task, and the foreground generation task as the Object task. The prompt for the Object task can be "complex picture with," while the prompt for the Context task can be "pure scene with." "Pure scene with" indicates that the generated background content (e.g., the first generated content) is relatively clean and lacks foreground content. In the foreground generation task, a mask for the foreground object is randomly selected, and the names of all foreground objects in the input image are appended to the foreground generation prompt, for example, ... Figure 6 The "Uncontrollable Generation" option shown includes "complex picture with car, person, handbag," which allows the foreground generation prompt to establish a correspondence with the foreground content. During model inference, the negative prompt can be set to the foreground generation prompt: complex picture with, to suppress uncontrollable generation.

[0058] It should be noted that the large visual model (e.g., the first processing model) is obtained through pre-training on a large number of images. Therefore, when there is a shadow around the area to be eliminated (e.g., the target area in this embodiment), the large visual model may supplement a target based on prior knowledge to make the shadow look reasonable. Therefore, as a method, this embodiment can simulate the generation of shadows in the image, such as... Figure 6The "Uncontrollable Generation Due to Shadows" item in the table shown can be obtained by first randomly placing the mask of the person in the right-hand image of the original image (the image where the person is located) in any area, then performing an affine transformation on the mask to obtain the shadow mask, then randomly initializing a shadow intensity shadow_weight, multiplying the original image and shadow_weight to obtain the actual training data (ground truth), and then pasting the mask onto the corresponding area of ​​the original image to obtain the mask image (masked_image) for training.

[0059] It should be noted that after fine-tuning the large visual model using ordinary scene images, the effect of completing human limbs may degrade. In the embodiments of this application, optimizations have been made specifically for human completion. For example... Figure 6 The mask of a person can be extracted and randomly shifted to the left or right by several pixels to cover part of the human body. This data can be used to fine-tune the model and optimize the completion of limbs.

[0060] It should be noted that, in this embodiment, the first processing model can randomly generate the first generated content, or it can generate the first generated content based on the original background content of the first image. This ensures that the generated first replacement content, after being integrated into the first image, is more consistent with the original content in the first image. For example, if the background content in the first image is identified as plants, the generated first replacement content can also be plant-related content. If the background content in the first image is identified as a street, the generated first replacement content can be content that would appear on a street.

[0061] In one approach, the first replacement content can be obtained based on a first image, a target mask image, and a first processing model. The target mask image is used to represent the position of the target region in the first image. In this embodiment, the mask image can be understood as a binary image with the same size as the original image. Specifically, when the target mask image is used to represent the position of the target region in the first image, the target mask image can be understood as a binary image with the same size as the first image.

[0062] The binary image may contain only black and white pixels. The mask image may include mask regions, which are used to identify areas in the original image that need to be processed. For example, the mask region in the target mask image identifies the location of the target region in the first image. Optionally, the pixels in the mask region of the mask image can all be black, and the pixels in the non-mask region can all be white. The non-mask region can be understood as all areas outside the mask region.

[0063] In this scenario, the target mask image corresponding to the first image can include a target mask region. This target mask region is used to identify the position of the target region in the first image, thereby identifying the area in the first image that needs to be processed. The position of the target mask region in the target mask image is the same as the position of the target region in the first image, and the outline shape of the target mask region can also be the same as the outline shape of the target region. For example... Figure 7 As shown, when the region where object 20 is located in the first image is determined to be the target region, the target mask image corresponding to the first image can be as follows: Figure 7 As shown in the image on the right. Figure 7 The target mask image shown includes black areas and white areas. The black areas can be understood as the target mask areas, and the white areas can be understood as non-mask areas.

[0064] In the case where the target region is represented by a target mask image, the target mask image and the first image can be input together into the first processing model.

[0065] S130: Replace the image content located in the target area of ​​the first image with the first replacement content to obtain the target image.

[0066] It should be noted that, in the embodiments of this application, the output of the first processing model can be either the first replacement content or the target image. When the output of the first processing model is the first replacement content, it can be understood that S130 is executed by a program outside the first processing model to obtain the target image. When the first processing model directly outputs the target image, it can be understood that the first processing model can directly complete the replacement operation to output the target image. Therefore, the aforementioned S130 can also include: replacing the image content located in the target region of the first image with the first replacement content through the first processing model, and outputting the obtained target image.

[0067] It should be noted that the image content output by the first processing model (e.g., the first replacement content) may have color differences from the original image (e.g., the first image), or may have a lower resolution than the original image. To correct these problems, the image content obtained by the first processing model can be optimized. One approach is to replace the image content located in the target region of the first image with the first replacement content to obtain a second image. The first replacement content in the second image is then optimized to use the optimized second image as the target image. The optimization process includes one or more of color optimization and resolution optimization.

[0068] As one approach, color optimization processing includes: performing color optimization processing on the first replacement content in the second image based on the color information of the image content in the second image excluding the first replacement content, and the color information of the image content in the first image excluding the target region. The color information may include information such as the standard deviation of color values ​​and the mean of color values.

[0069] Optionally, color optimization can be performed using the following formula:

[0070] image_new=(sc-s_mean)*(t_std / s_std)+t_mean

[0071] Wherein, image_new represents the result after color alignment (target image), sc represents the preliminary elimination result image (second image), s_mean represents the mean of pixel color values ​​in the non-masked region (region outside the target region) of the preliminary elimination result, s_std represents the standard deviation of pixel color values ​​in the non-masked region of the preliminary elimination result, t_mean represents the mean of pixel color values ​​in the non-masked region of the downsampled image of the original image (first image), and t_std represents the standard deviation of pixel color values ​​in the non-masked region of the downsampled image of the original image. The purpose of this step is to align the mean and standard deviation of the elimination result image with the original image, thereby completing the color transfer (color optimization processing).

[0072] As one approach, the resolution optimization step can be used to increase the resolution of the first replacement content, so that after image content replacement, the resolution of the target region in the first image is consistent with the resolution of the image content in other regions (regions other than the target region) in the first image. Optionally, differences can be evaluated first to compare the resolution of the image content in other regions with the resolution of the first replacement content, identifying the existing resolution gap. Based on the aforementioned resolution gap, a specific target for resolution optimization is set, for example, increasing the resolution of the first replacement content to be the same as the resolution of the image content in other regions.

[0073] In the embodiments of this application, there are multiple ways to improve the resolution of the first replacement content. Optionally, interpolation algorithms (such as bilinear interpolation, bicubic interpolation, or more advanced neural network model-based interpolation methods, such as deep learning super-resolution techniques) can be used to increase the number of pixels in the replacement content, thereby improving its resolution. Optionally, the most suitable interpolation algorithm can be dynamically selected based on the image content features of the first replacement content. For example, a simple interpolation method can be used in smooth areas to reduce computation, while a more complex interpolation method can be used in areas with rich texture or edges to preserve details. Furthermore, while improving resolution, image processing techniques (such as sharpening, edge enhancement, etc.) can be used to enhance the texture details of the first replacement content, making it more harmonious with the image surrounding the target area.

[0074] One approach to obtaining the first replacement content based on a first image, a target mask image, and a first processing model can be understood as follows: first, the first image is masked using the target mask image to obtain a masked first image. Then, the masked first image and the target mask image are input into the first processing model, allowing the first processing model to obtain the first replacement content based on the masked first image and the target mask image. Masking the first image can be understood as changing the color values ​​of all pixels in the target region of the first image to black (i.e., changing the pixel values ​​to 0) or white (i.e., changing all pixel values ​​to 255) to obtain the masked first image. In this case, the black or white in the masked first image can be used to help the first processing model clearly identify the area where content replacement is needed.

[0075] Alternatively, in this embodiment, the content input to the first processing model can be vectorized before being input into the first processing model. In this approach, after obtaining the first image with masking, the first image can be converted into a corresponding first vector representation, and the target mask image can be converted into a corresponding second vector representation. The first and second vector representations can then be input into the first processing model, allowing the first processing model to obtain the first replacement content based on the first and second vector representations. It should be noted that in this approach, the output of the first processing model will also be in vector form; for example, a third vector representation can be output. Optionally, the third vector representation can be obtained based on the first and second vector representations and the first processing model. After converting the third vector representation, the first replacement content or the target image can be obtained. Specifically, when the third vector representation represents the first replacement content, converting the third vector representation yields the first replacement content. When the third vector representation represents the target image, converting the third vector representation yields the target image.

[0076] Next, we will proceed through... Figure 8 The steps of an image processing method involved in this embodiment will be described as follows: Figure 8 As shown, in Figure 8 In the process shown,

[0077] like Figure 8 As shown, the first image can be masked by setting the elimination region (target region) to 0, resulting in a masked first image. This masked first image is then fed into an encoder (e.g., a VAE Encoder) to obtain a latent space masked_image. Here, masked_image is the first vector representation. This masked_image is then concatenated with the mask and randomly initialized noise, and fed into the Unet model (a first processing model) to obtain the elimination result (target image).

[0078] In the model application phase, noise input is not required. That is, during the application phase, the first and second vector representations can be concatenated and then input into the first processing model. The third vector representation output by the Unet model will then be processed by a decoder (e.g., a VAE decoder) to obtain the final noise reduction result. Figure 8 As shown, the input to UNet also includes positive and negative prompts. A positive prompt can be "pure scene with", and a negative prompt can be "complex picture with". Correspondingly, both positive and negative prompts can be encoded into vector forms using a text encoder and input into the UNet model.

[0079] This embodiment provides an image processing method in which, in the process of generating image content by the first processing model, in addition to performing a background generation task, a foreground generation task is also performed. This allows the foreground content in the first generated content to be eliminated by the second generated content obtained from the foreground generation task. This helps to avoid the first replacement content from including new foreground content, thereby making the generated image content (first replacement content) used to replace the target area more controllable and improving the image elimination effect.

[0080] Please see Figure 9 This application provides an image processing method, which includes:

[0081] S210: Determine the target region from the first image, whereby the target region represents the region to be processed.

[0082] S220: If it is detected that the processing target is not to perform text elimination processing, then based on the first image, the target region and the first processing model, the first replacement content is obtained, wherein the first replacement content is obtained by the first processing model through the second generated content to eliminate the foreground content in the first generated content, the first generated content is obtained by the first processing model to perform the background generation task, and the second generated content is obtained by the first processing model to perform the foreground generation task.

[0083] S221: Replace the image content located in the target area of ​​the first image with the first replacement content to obtain the target image.

[0084] S230: If the processing target is detected to be text elimination processing, then based on the first image, the target region and the second processing model, the second replacement content is obtained. The way the second processing model generates image content is different from the way the first processing model generates image content.

[0085] S231: Replace the image content located in the target area of ​​the first image with the second replacement content to obtain the target image.

[0086] In this embodiment, after obtaining the first image, the processing target can be detected. For example, it can be detected whether the user wants to eliminate a specific object in the first image or eliminate text in the first image. Different processing models can be used for different processing targets to improve the processing effect. In one approach, the first processing model is a diffusion model, and the second processing model is a Generative Adversarial Network (GAN) model. A GAN is a deep learning architecture consisting of a generator and a discriminator. The generator attempts to generate data similar to real data. The discriminator determines whether the input is real data or data generated by the generator. The generator and discriminator compete and learn from each other to improve the quality of the generated data. The content generated by the GAN model has a different level of detail compared to the diffusion model. However, the GAN model tends to extend edges during background completion, making it more suitable for eliminating text in images.

[0087] In the embodiments of this application, there are multiple ways to determine the user's processing intent.

[0088] One approach is to directly ask the user a question. In this method, after the user identifies the target area, a question can be displayed. This question may include multiple intent options, and the user's selected option can be used as the target option, with the intent represented by the target option becoming the processing objective. For example, the multiple intent options may include a text removal option and a foreground content removal option. If the user selects the text removal option, the processing objective is determined to be text removal. If the user selects the foreground content removal option, the processing objective is determined not to be text removal.

[0089] Alternatively, if a text region is detected in the first image, and the proportion of the intersection of the text region and the target region within the target region is less than or equal to a preset proportion threshold, then it is determined that the processing target is not for text removal processing. In this embodiment, the preset proportion threshold is not specifically limited. For example, the preset proportion threshold can be 0.3, 0.4, etc.

[0090] Optionally, if the first image contains text, a text mask image of the text region can be output. This text mask image represents the region containing the text in the first image. Then, the intersection region of the target mask image and the text mask image can be calculated. This intersection region can be understood as the intersection of the target region represented by the target mask image and the text region represented by the text mask image. Then, the proportion of the intersection region to the target region is calculated. For example, such as... Figure 10 As shown, based on Figure 10 The legend shown, Figure 10 The target region consists of region Q1 and region Q2, while the text region consists of region Q2 and region Q3. In this case, region Q2 can immediately be considered the intersection region.

[0091] As one approach, when the processing target is determined to be text removal, the union region of the target region and the text region can be obtained. Based on the first image, the union region, and the second processing model, a second replacement content is obtained. The image content in the first image located within the union region is replaced with the second replacement content to obtain the target image. In other words, when the union region is obtained, it represents the region in the first image used for text removal. This union region can include regions originally unique to the target region, regions originally unique to the text region, and the aforementioned intersection region. Regions originally unique to the target region can be understood as regions belonging to the target region but not to the text region. Regions originally unique to the text region can be understood as regions belonging to the text region but not to the target region.

[0092] For example, such as Figure 10 As shown, region Q1 can be understood as a region unique to the target region. Region Q3 can be understood as a region unique to the text region. Therefore, the resulting union region can include regions Q1, Q2, and Q3.

[0093] This embodiment provides an image processing method that, through the aforementioned method, during the image content generation process of the first processing model, in addition to performing a background generation task, also performs a foreground generation task. This allows the second generated content obtained from the foreground generation task to eliminate the foreground content in the first generated content, thereby helping to avoid the generation of new foreground content in the first replacement content. This makes the generated image content (the first replacement content) used to replace the target area more controllable, thus improving the image elimination effect. Furthermore, in this embodiment, after acquiring the image to be processed, it first detects whether the image contains text regions to determine the user's intention to process the image. Based on different user intentions, different processing models can be adopted, improving the flexibility and adaptability of the processing.

[0094] Please see Figure 11 This application provides an image processing method, which includes:

[0095] S310: Determine the target region from the first image, whereby the target region represents the region to be processed.

[0096] S320: Based on the first image, the target region, and the first processing model, a first replacement content is obtained, wherein the first replacement content is obtained by the first processing model through the second generated content to eliminate the foreground content in the first generated content, the first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task;

[0097] S330: If there is no foreground content to avoid generation in the first replacement content, replace the image content located in the target area of ​​the first image with the first replacement content to obtain the target image.

[0098] For example, the foreground content to be avoided could include people, plants, and animals.

[0099] S340: If the first replacement content contains foreground content that is to be avoided from being generated, then based on the first image, the target region and the second processing model, the third replacement content is obtained, and the image content in the target region of the first image is replaced with the third replacement content to obtain the target image.

[0100] In the embodiments of this application, there can be various ways to avoid generating foreground content. The foreground content to be avoided can be determined during the training process of the first model, or it can be determined during the model application phase.

[0101] This embodiment provides an image processing method in which, in the process of generating image content by the first processing model, in addition to performing a background generation task, a foreground generation task is also performed. This allows the foreground content in the first generated content to be eliminated by the second generated content obtained from the foreground generation task. This helps to avoid the first replacement content from including new foreground content, thereby making the generated image content (first replacement content) used to replace the target area more controllable and improving the image elimination effect.

[0102] Furthermore, in this embodiment, after obtaining the first replacement content through the first processing model, it is also detected whether there is any foreground content to be avoided in the first replacement content. If so, the second processing model can be used to process the target area, thereby improving the flexibility and effectiveness of the solution, and also helping to avoid regenerating the foreground content that the user originally wanted to eliminate.

[0103] Next, we will proceed... Figure 12 The flow of an image processing method according to an embodiment of this application will be described.

[0104] exist Figure 12 In the illustrated process, after the image processing method begins, text detection can be performed on the first image. The purpose of text detection is to determine whether the processing target is to remove text content from the first image. If the processing target is detected to be removing text content from the first image, a pre-completion operation can be performed. During pre-completion, a second replacement content can be obtained based on the first image, the target region, and the second processing model. Then, the image content in the first image located in the target region is replaced with the second replacement content to obtain the target image. Alternatively, the second replacement content can be super-resolution processed before being applied to the first image to obtain the target image. If the processing target is detected to be removing foreground content from the target region, a large model completion operation can be performed. The large model completion process can include: obtaining a first replacement content based on the first image, the target region, and the first processing model; and replacing the image content in the first image located in the target region with the first replacement content to obtain the target image.

[0105] Furthermore, a security check can be performed on the target content obtained through large model completion. During the security check, it is determined whether the first replacement content obtained through large model completion includes foreground content that should be avoided (e.g., characters). If so, the security check fails, and a pre-completion operation is then performed to process the target area. Additionally, the first replacement content obtained through large model completion can be discarded.

[0106] Correspondingly, the first replacement content can be processed sequentially by color difference correction (e.g., the aforementioned color optimization processing) and super-resolution processing (e.g., the aforementioned resolution optimization processing) before being replaced into the first image.

[0107] Please see Figure 13 This application provides an image processing apparatus 400, which includes:

[0108] The target region determination unit 410 is used to determine a target region from the first image, wherein the target region represents the region to be processed.

[0109] The content generation unit 420 is used to obtain first replacement content based on the first image, the target region, and the first processing model. The first replacement content is obtained by the first processing model through the elimination of foreground content in the first generated content by the second generated content. The first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task.

[0110] The image processing unit 430 is used to replace the image content located in the target area of ​​the first image with first replacement content to obtain the target image.

[0111] In one approach, the second generated content is obtained by the first processing model performing a foreground generation task based on the reverse prompt words, where the reverse prompt words represent the foreground content to be avoided.

[0112] Optionally, the reverse prompts include foreground generation flags, and the correspondence between the foreground generation flags and various foreground contents is established during the training of the first processing model. The foreground generation flags are used as inputs into the first processing model to trigger the first processing model to perform the foreground generation task.

[0113] In one approach, the content generation unit 420 is specifically used to obtain first replacement content based on the first image, the target mask image, and the first processing model, wherein the target mask image is used to characterize the position of the target region in the first image.

[0114] In one approach, the content generation unit 420 is specifically configured to, if it is detected that the processing target is not for text elimination processing, obtain first replacement content based on the first image, the target region, and the first processing model. The content generation unit 420 is also specifically configured to, if it is detected that the processing target is for text elimination processing, obtain second replacement content based on the first image, the target region, and the second processing model, wherein the method by which the second processing model generates image content differs from the method by which the first processing model generates image content; the image content located in the target region of the first image is replaced with the second replacement content to obtain the target image.

[0115] Optionally, the content generation unit 420 is further configured to determine that the processing target is not to perform text elimination processing if it is recognized that the first image includes a text region and the proportion of the intersection region of the text region and the target region in the target region is less than or equal to a preset proportion threshold.

[0116] In one manner, the image processing unit 430 is specifically used to replace the image content located in the target area of ​​the first image with the first replacement content if there is no foreground content to be avoided in the first replacement content, so as to obtain the target image; if there is foreground content to be avoided in the first replacement content, then based on the first image, the target area and the second processing model, a third replacement content is obtained, and the image content located in the target area of ​​the first image is replaced with the third replacement content, so as to obtain the target image.

[0117] In one approach, the image processing unit 430 is specifically used to replace the image content located in the target area of ​​the first image with a first replacement content to obtain a second image; and to perform optimization processing on the first replacement content in the second image so as to use the optimized second image as the target image; wherein the optimization processing includes one or more of color optimization processing and resolution optimization processing.

[0118] This embodiment provides an image processing apparatus that, during the image content generation process of the first processing model, performs both a background generation task and a foreground generation task. This allows the foreground content in the first generated content to be eliminated by the second generated content obtained from the foreground generation task. This helps to prevent the generated first replacement content from including new foreground content, thereby making the generated image content (first replacement content) used to replace the target area more controllable and improving the image elimination effect.

[0119] It should be noted that the device embodiments in this application correspond to the aforementioned method embodiments. The specific principles in the device embodiments can be found in the content of the aforementioned method embodiments, and will not be repeated here.

[0120] The following will combine Figure 14 This application describes an electronic device.

[0121] Please see Figure 14 Based on the aforementioned image processing method and apparatus, this application embodiment also provides another electronic device 100 capable of executing the aforementioned image processing method. The electronic device 100 includes one or more (only one shown in the figure) processors 102, a memory 104, and a network module 106 coupled to each other. The memory 104 stores programs capable of executing the contents of the aforementioned embodiments, and the processors 102 can execute the programs stored in the memory 104.

[0122] The processor 102 may include one or more processing cores. The processor 102 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 104, and by calling data stored in the memory 104. Optionally, the processor 102 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 102 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the displayed content; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 102 and may be implemented separately using a communication chip.

[0123] The memory 104 may include random access memory (RAM) or read-only memory (ROM). The memory 104 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 104 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the terminal 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0124] The network module 106 is used to receive and transmit electromagnetic waves, realizing the mutual conversion between electromagnetic waves and electrical signals, thereby communicating with communication networks or other devices, such as audio playback devices. The network module 106 may include various existing circuit elements for performing these functions, such as antennas, radio frequency transceivers, digital signal processors, encryption / decryption chips, user identity modules (SIM cards), memory, etc. The network module 106 can communicate with various networks such as the Internet, corporate intranets, and wireless networks, or communicate with other devices through wireless networks. The aforementioned wireless networks may include cellular telephone networks, wireless local area networks (WLANs), or metropolitan area networks (MANs). For example, the network module 106 can interact with base stations.

[0125] Sensor module 108 may include at least one sensor. Specifically, sensor module 108 may include, but is not limited to, pressure sensors, motion sensors, acceleration sensors, and other sensors.

[0126] The pressure sensor detects pressure generated by pressing on the electronic device 100. Specifically, it detects pressure generated by contact or pressing between the user and the electronic device 100, such as pressure generated by contact or pressing between the user's ear and the electronic device 100. Therefore, the pressure sensor can be used to determine whether contact or pressing has occurred between the user and the electronic device 100, and the magnitude of the pressure. The accelerometer detects the magnitude of acceleration in various directions (generally three axes), and when stationary, it can detect the magnitude and direction of gravity. This can be used for applications that identify the posture of the electronic device 100 (such as screen orientation switching, related games, magnetometer posture calibration), and vibration recognition functions (such as pedometers, tapping). Additionally, the electronic device 100 may also be equipped with other sensors such as a gyroscope, barometer, hygrometer, and thermometer, which will not be elaborated upon here.

[0127] Please refer to Figure 15This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 800 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0128] The computer-readable storage medium 800 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 800 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 800 has storage space for program code 810 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 810 may be compressed, for example, in a suitable form.

[0129] In summary, the image processing method, apparatus, and electronic device provided in this application, after determining a target region from a first image, can obtain first replacement content based on the first image, the target region, and a first processing model. Then, the image content located in the target region of the first image is replaced with the first replacement content to obtain the target image. The first replacement content is generated by the first processing model through a background generation task and a foreground generation task. During the generation of the first replacement content, the second generation content is used to cancel out the foreground content in the first generation content. Thus, in the process of the first processing model generating image content, in addition to performing a background generation task, a foreground generation task is also performed. This allows the second generation content obtained from the foreground generation task to eliminate the foreground content in the first generation content, thereby helping to avoid the generation of new foreground content in the first replacement content. This makes the generated image content (first replacement content) used to replace the target region more controllable, improving the image removal effect.

[0130] In one scenario, the image processing method provided in this application embodiment can also be understood as an image element elimination method, allowing users to eliminate any target in an image and perform background completion at the eliminated target location without regenerating new objects. In this application embodiment, targeted optimizations have been made for the completion of text and subjects obscured by passersby, ensuring that text can be eliminated and limbs can be perfectly completed. The algorithm has high controllability, good security, and the completed content is clear, natural, and semantically reasonable.

[0131] This application embodiment designs an image element elimination process that ensures effective text elimination, controls generated content to improve security, optimizes color cast issues, and improves the clarity of generated content. Furthermore, this application embodiment divides elimination into two tasks: background completion and foreground generation. During inference, the foreground generation flag prompt is sent to the network as a negative prompt, effectively suppressing the regeneration of foreground targets. This application embodiment makes targeted optimizations at the data level for uncontrollable generation, uncontrollable generation caused by shadows, and completion of main body limbs, effectively improving the performance of the corresponding optimization items.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. An image processing method, characterized in that, The method includes: A target region is determined from the first image, the target region representing the area to be processed. Based on the first image, the target region, and the first processing model, a first replacement content is obtained. The first replacement content is obtained by the first processing model through the second generated content, which eliminates the foreground content in the first generated content. The first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task. The image content located in the target region of the first image is replaced with the first replacement content to obtain the target image.

2. The method according to claim 1, characterized in that, The second generated content is obtained by the first processing model performing a foreground generation task based on reverse prompts, where the reverse prompts represent foreground content to be avoided.

3. The method according to claim 2, characterized in that, The reverse prompt word includes a foreground generation flag. The correspondence between the foreground generation flag and the various foreground contents is established during the training process of the first processing model. The foreground generation flag is used to input into the first processing model to trigger the first processing model to execute the foreground generation task.

4. The method according to claim 1, characterized in that, The first replacement content is obtained based on the first image, the target region, and the first processing model, including: Based on the first image, the target mask image, and the first processing model, the first replacement content is obtained, wherein the target mask image is used to characterize the position of the target region in the first image.

5. The method according to claim 1, characterized in that, The first replacement content obtained based on the first image, the target region, and the first processing model includes: If it is detected that the processing target is not for text elimination processing, then the first replacement content is obtained based on the first image, the target region, and the first processing model.

6. The method according to claim 5, characterized in that, The method further includes: If the processing target is detected to be text elimination processing, then based on the first image, the target region and the second processing model, the second replacement content is obtained. The way the second processing model generates image content is different from the way the first processing model generates image content. The image content located in the target area of ​​the first image is replaced with the second replacement content to obtain the target image.

7. The method according to claim 5, characterized in that, The method further includes: If a text region is detected in the first image, and the intersection of the text region and the target region accounts for a proportion less than or equal to a preset proportion threshold in the target region, then it is determined that the processing target is not to perform text elimination processing.

8. The method according to claim 1, characterized in that, The step of replacing the image content located in the target region of the first image with the first replacement content to obtain the target image includes: If the first replacement content does not contain any foreground content to avoid generation, the image content located in the target area of ​​the first image is replaced with the first replacement content to obtain the target image; If the first replacement content contains foreground content that should be avoided from being generated, then based on the first image, the target region, and the second processing model, a third replacement content is obtained. The image content in the first image located in the target region is replaced with the third replacement content to obtain the target image.

9. The method according to claim 1, characterized in that, Replacing the image content located in the target region of the first image with the first replacement content to obtain the target image includes: The image content located in the target area of ​​the first image is replaced with the first replacement content to obtain the second image; The first replacement content in the second image is optimized so that the optimized second image can be used as the target image. The optimization process includes one or more of the following: color optimization and resolution optimization.

10. The method according to claim 9, characterized in that, The color optimization process includes: Based on the color information of the image content in the second image excluding the first replacement content, and the color information of the image content outside the target area in the first image, color optimization processing is performed on the first replacement content in the second image.

11. An image processing apparatus, characterized in that, The device includes: A target region determination unit is used to determine a target region from a first image, wherein the target region represents the region to be processed. The content generation unit is used to obtain first replacement content based on the first image, the target region, and the first processing model. The first replacement content is obtained by the first processing model through the elimination of foreground content in the first generated content using the second generated content. The first generated content is obtained by the first processing model performing a background generation task, and the second generated content is obtained by the first processing model performing a foreground generation task. An image processing unit is configured to replace the image content located in the target region of the first image with the first replacement content to obtain a target image.

12. An electronic device, characterized in that, It includes a processor and a memory; one or more programs are stored in the memory and configured to be executed by the processor to implement the method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program code, wherein the program code, when executed by a processor, performs the method according to any one of claims 1-10.

Images