Image processing method and device, readable storage medium and program product

Abstract

The invention relates to an image processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining a to-be-processed image; the image to be processed comprises an object to be eliminated; detecting an area of the to-be-eliminated object included in the to-be-processed image to obtain a detection result image; generating an elimination result image based on the detection result image; and post-processing the elimination result image to obtain a target image not including the object to be eliminated. The method can effectively improve the visual effect of image processing while effectively improving the image processing efficiency, achieves the automatic positioning and elimination of the to-be-eliminated target in the image, improves the user experience, and brings convenience to a user.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an image processing method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] With the development of computer technology and Internet technology, image processing is widely used in civil and commercial fields, and the interference target elimination technology in images plays an increasingly important role in beautifying images and other fields, and faces higher requirements.

[0003] However, in the current image processing method, the mainstream method is usually based on manual smearing or manual cropping to eliminate the target to be eliminated (such as pedestrians) in the original image. If the method of eliminating pedestrians based on manual smearing is used, the dynamic blur tool usually needs the user to manually adjust the blur radius, direction and intensity, which requires high operation skills. For example, the blur direction needs to be consistent with the direction of the moving subject, otherwise it is easy to show distortion. In addition, excessive blur may cause the picture to be out of focus or lose the sense of hierarchy, and needs to be combined with layer opacity (such as 70%-80%) and mask local adjustment to optimize the effect. This processing method is cumbersome to operate, and the target image effect obtained is easy to be unnatural. If the method of eliminating pedestrians based on cropping is used, that is, the pedestrians on the edge are cropped through secondary composition. This method destroys the original composition, and the picture is easy to be incomplete, although it can achieve the purpose of eliminating pedestrians, but it also has the problems of cumbersome operation, unnatural effect and limited use scene. Therefore, how to effectively improve the image processing efficiency while effectively improving the visual effect of image processing has become a problem to be solved. SUMMARY

[0004] Therefore, the present application provides an image processing method and device, computer equipment, computer readable storage medium and computer program product, which can effectively improve the visual effect of image processing while effectively improving the image processing efficiency, realize automatic positioning and elimination of the target to be eliminated in the image, and improve the user experience and bring convenience to the user.

[0005] In one aspect, the present application provides an image processing method, comprising: acquiring a to-be-processed image; the to-be-processed image includes a to-be-eliminated object; detecting a region of the to-be-eliminated object included in the to-be-processed image to obtain a detection result image; generating an elimination result image based on the detection result image; and post-processing the elimination result image to obtain a target image not including the to-be-eliminated object.

[0006] In an aspect, the present application also provides an image processing apparatus, comprising: an acquisition module configured to acquire a to-be-processed image; the to-be-processed image comprising a to-be-eliminated object; a detection module configured to detect a region of the to-be-eliminated object comprised in the to-be-processed image to obtain a detection result image; a generation module configured to generate an elimination result image based on the detection result image; and a processing module configured to perform post-processing on the elimination result image to obtain a target image not comprising the to-be-eliminated object.

[0007] In an aspect, the present application also provides a computer device, comprising a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program: acquiring a to-be-processed image; the to-be-processed image comprising a to-be-eliminated object; detecting a region of the to-be-eliminated object comprised in the to-be-processed image to obtain a detection result image; generating an elimination result image based on the detection result image; and performing post-processing on the elimination result image to obtain a target image not comprising the to-be-eliminated object.

[0008] In an aspect, the present application also provides a computer-readable storage medium, storing a computer program, the computer program being executed by a processor to implement the following steps: acquiring a to-be-processed image; the to-be-processed image comprising a to-be-eliminated object; detecting a region of the to-be-eliminated object comprised in the to-be-processed image to obtain a detection result image; generating an elimination result image based on the detection result image; and performing post-processing on the elimination result image to obtain a target image not comprising the to-be-eliminated object.

[0009] In an aspect, the present application also provides a computer program product, comprising a computer program, the computer program being executed by a processor to implement the following steps: acquiring a to-be-processed image; the to-be-processed image comprising a to-be-eliminated object; detecting a region of the to-be-eliminated object comprised in the to-be-processed image to obtain a detection result image; generating an elimination result image based on the detection result image; and performing post-processing on the elimination result image to obtain a target image not comprising the to-be-eliminated object.

[0010] The image processing method, device, computer device, computer readable storage medium and computer program product obtain a to-be-processed image, the to-be-processed image includes a to-be-eliminated object, and a region of the to-be-eliminated object included in the to-be-processed image is detected to obtain a detection result image. Further, based on the detection result image, an elimination result image is generated, and the elimination result image is post-processed to obtain a target image that does not include the to-be-eliminated object. Since the target image in the present application is obtained by post-processing the preliminary elimination result image, the target image obtained in the present application is the elimination result image after post-processing optimization, so that the elimination result image after post-processing optimization is better, avoiding the edge artifacts and discontinuity problems that are prone to occur in traditional methods. While effectively improving the image processing efficiency, the visual effect of image processing is also effectively improved, automatic positioning and elimination of the to-be-eliminated target in the image are realized, and the user experience is improved, bringing convenience to users. BRIEF DESCRIPTION OF DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0012] Figure 1 An application environment diagram of an image processing method in an embodiment;

[0013] Figure 2 A flowchart of an image processing method in an embodiment;

[0014] Figure 3 A general framework diagram of an image processing method provided in an embodiment;

[0015] Figure 4 A flowchart of a pedestrian detection algorithm provided in an embodiment;

[0016] Figure 5 A flowchart of determining a high-confidence pedestrian region after fusion in an embodiment;

[0017] Figure 6 A network structure diagram of a target diffusion model in an embodiment;

[0018] Figure 7 A flowchart of post-processing an elimination result image in an embodiment;

[0019] Figure 8A general framework diagram of a pedestrian elimination method and system based on a diffusion model provided in an embodiment;

[0020] Figure 9 A structural block diagram of an image processing device in an embodiment;

[0021] Figure 10 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0022] In order to make the purposes, technical solutions and beneficial effects of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.

[0023] The image processing method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The server 104 can be a background server of an image application. After the terminal 102 obtains the to-be-processed image, the terminal can send the obtained to-be-processed image to the background server of the image application, that is, the server 104, so that the server 104 obtains the to-be-processed image, detects the region of the to-be-eliminated object included in the to-be-processed image, and obtains a detection result image. Further, the server 104 can generate an elimination result image based on the detection result image, and post-process the elimination result image, so as to obtain a target image not including the to-be-eliminated object. The server 104 can return the obtained target image not including the to-be-eliminated object to the terminal 102, so that the terminal 102 displays the target image not including the to-be-eliminated object.

[0024] The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, Internet of Things devices, and portable wearable devices. The Internet of Things device can be a smart speaker, a smart television, a smart air conditioner, a smart vehicle-mounted device, a projection device, etc. The portable wearable device can be a smart watch, a smart bracelet, a head-mounted device, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0025] In an exemplary embodiment, asFigure 2 As shown, an image processing method is provided, and the method is applied to Figure 1 The terminal in the method is taken as an example for illustration, and the method includes the following steps 202 to 208. Wherein:

[0026] Step 202, obtaining a to-be-processed image; the to-be-processed image includes a to-be-eliminated object.

[0027] Wherein, the to-be-processed image refers to an image to be beautified, for example, the to-be-processed image in the present application can refer to an original image including a to-be-eliminated object, that is, some unnecessary objects (such as pedestrian interference objects) in the original image need to be eliminated.

[0028] The to-be-eliminated object refers to an interference object contained in the to-be-processed image, for example, the to-be-eliminated object in the present application includes but is not limited to: pedestrians (passers-by), shielding objects and the like.

[0029] Step 204, detecting the region of the to-be-eliminated object included in the to-be-processed image to obtain a detection result image.

[0030] Wherein, the region of the to-be-eliminated object refers to the region where the to-be-eliminated object exists, for example, the region of the to-be-eliminated object in the present application can refer to a portrait region, that is, the to-be-eliminated object is a pedestrian, and the portrait region of one or more pedestrians included in the to-be-processed image is detected, that is, a detection result image with a mask region (mask image or mask image) is obtained.

[0031] The detection result image refers to an image including a detection result obtained by positioning and detecting the to-be-eliminated object in the to-be-processed image, for example, the detection result image in the present application can be an original image with a mask region (mask image or mask image), that is, the region (such as coordinates, mask) where the to-be-eliminated object exists in the detected original image (to-be-processed image) is marked and displayed, for example, the region where the to-be-eliminated object exists in the detected original image (to-be-processed image) is displayed as a black mask, and other regions display original pixel content, that is, an original image with a mask is obtained, and the original image with a mask is taken as a detection result image.

[0032] Optionally / illustratively, the devices used by different users (operation objects) can all interact with the image application (or image generation system), when a user (operation object) wants to beautify a personalized image (such as a personalized face image) (for example, eliminate a disturbing pedestrian in the original image), the user can open the image application (Application, APP) on the terminal by triggering an operation, and enter the image generation page of the image application by selecting an operation, that is, the user can log in to the image application (such as an image beautification application) by triggering an operation, further, the user can input information or select information in the image generation page of the image application displayed on the terminal, so that the terminal acquires the input image to be processed in response to the information input operation or information selection operation triggered by the user, further, the terminal automatically detects the region of the object to be eliminated included in the image to be processed in response to the image processing request (such as a request for eliminating a pedestrian included in the image) triggered by the operation object in the image generation page of the image application, obtains a detection result image, and generates an elimination result image based on the detection result image; further, the terminal can post-process the elimination result image, that is, obtain a target image not including the object to be eliminated, and visually display the target image not including the object to be eliminated.

[0033] It can be understood that the method provided by the present application can be realized by interaction between the terminal and the background server of the image application, or realized by interaction between the front end and the back end of the terminal, that is, the front end of the terminal is used to display the image to be processed and the target image not including the object to be eliminated obtained, and the back end of the terminal is equivalent to the background server, which is used for detecting the region of the object to be eliminated included in the image to be processed, obtaining a detection result image, and the like logical processing.

[0034] Specifically, the way in which the terminal acquires the image to be processed includes but is not limited to: the image to be processed captured by the camera in real time, the image to be processed intercepted on the network, the historical image acquired from the local database or the database of the background server of the image application, and the like. That is, after the terminal acquires the image to be processed including the object to be eliminated, the terminal can detect the region of the object to be eliminated included in the image to be processed, that is, obtain a detection result image with a mask region (mask image). For example, the terminal can use a portrait instance segmentation algorithm and a pedestrian detection algorithm to detect the region of the object to be eliminated in the image to be processed, obtain a detection result set, and select the detection result satisfying the confidence threshold from the detection result set, and finally determine the detection result image based on the selected detection result satisfying the confidence threshold.

[0035] For example, taking pedestrians as an example. Assuming that the terminal obtains a to-be-processed image A including one or more pedestrians (irrelevant interference objects), the terminal can detect the portrait area of the pedestrian included in the to-be-processed image A to obtain a detection result image including the position information of the portrait area (labeled and displayed, such as labeled in the form of a mask). The portrait area in the detection result image is a binary image (that is, a mask image). For example, the terminal can perform portrait instance segmentation on the pedestrian included in the to-be-processed image through a pre-trained segmentation model to obtain a portrait area instance set of the pedestrian. Each portrait area instance in the portrait area instance set includes pixel-level mask data and a class label. Meanwhile, the terminal can also detect the portrait area of the pedestrian included in the to-be-processed image through a pre-trained detection model to obtain a portrait area set of the pedestrian, and determine the position information of a target portrait area that meets a confidence threshold based on the above portrait area instance set (that is, a large-scale portrait mask) and the portrait area set (that is, a small-scale portrait mask). Finally, the terminal generates a detection result image including the position information of the portrait area (one or more mask regions) based on the position information of the target portrait area.

[0036] In step 206, a removal result image is generated based on the detection result image.

[0037] The removal result image refers to an image obtained by removing (removing) the to-be-removed object included in the to-be-processed image, and can also be referred to as a beautified image. For example, the removal result image in the present application can be a beautified image obtained by removing each pedestrian in the to-be-processed image.

[0038] In step 208, the removal result image is post-processed to obtain a target image that does not include the to-be-removed object.

[0039] The post-processing refers to re-optimizing the removal result image. For example, the post-processing in the present application can be a series of post-processing such as multi-scale fusion processing, detail enhancement processing, global consistency adjustment processing, and the like. For another example, the post-processing in the present application can also be smoothing, edge transition, detail adjustment, and the like on the repair region (that is, the pedestrian mask region) in the removal result image.

[0040] The target image refers to an image in which the to-be-removed object is removed (removed). For example, the target image in the present application can be an image in which one or more pedestrians (irrelevant interference objects) included in the original image are removed, and the repair region (that is, the mask region in which the pedestrian is removed) in the target image in the present application has a smooth, natural boundary transition visual effect.

[0041] Specifically, the terminal acquires the image to be processed and detects the regions containing the objects to be eliminated within the image. After obtaining the detection result image, the terminal can generate an elimination result image based on the detection result image. For example, the terminal can call a pre-trained target diffusion model and use the target diffusion model to fill in the content of the mask region (i.e., the region of the object to be eliminated) in the detection result image to obtain the elimination result image. Further, the terminal can perform post-processing on the elimination result image to obtain a target image that does not include the objects to be eliminated. For example, the terminal can upsample the elimination result image to obtain a target elimination result image with the same resolution as the image to be processed, and determine a composite image based on the mask region in the detection result image, the target elimination result image, and the image to be processed. Further, the terminal can perform fusion processing on the composite image to obtain a fused composite image, and use the fused composite image as the final target image that does not include the objects to be eliminated.

[0042] For example, let's take pedestrians as an example to illustrate this. Figure 3 The diagram shown is an overall framework diagram of the image processing method provided in this application. Assuming that the image A to be processed acquired by the terminal includes one or more pedestrians (irrelevant interference objects), the terminal can, based on... Figure 3 The pedestrian detection algorithm shown detects the human image regions in the image to be processed, A. After obtaining a detection result image including the location information of the human image regions (one or more mask regions), the terminal can use a diffusion model-based elimination algorithm. That is, it uses a pre-trained target diffusion model to fill in the content of one or more mask regions (i.e., the regions of the objects to be eliminated) in the detection result image to obtain the eliminated result image A (one or more mask regions have been repaired, i.e., filled with pixel content). Further, the terminal can perform fusion processing on the eliminated result image A to obtain the result image. For example, the terminal can upsample the eliminated result image A to the same resolution as the image to be processed (e.g., an upsampling resolution of 512×512) to obtain a target eliminated result image. Based on one or more mask regions in the detection result image, the target eliminated result image, and the image to be processed, a composite image is determined. Further, the terminal can perform fusion processing on the composite image (i.e., while preserving the details inside the regions, making the transitions at the boundaries more natural and smooth) to obtain a fused composite image, and use the fused composite image as the final target image that does not include the objects to be eliminated.

[0043] In this embodiment, by acquiring a to-be-processed image, the to-be-processed image includes a to-be-eliminated object, and the region of the to-be-eliminated object included in the to-be-processed image is detected to obtain a detection result image; further, based on the detection result image, an elimination result image is generated, and the elimination result image is post-processed to obtain a target image which does not include the to-be-eliminated object. Since the target image in this application is obtained by post-processing the preliminary elimination result image, the final target image obtained in this application is the elimination result image after post-processing optimization, so that the elimination result image after post-processing optimization is better, avoiding the edge artifacts and discontinuity problems that are prone to occur in traditional methods, effectively improving the image processing efficiency and visual effect of image processing, realizing automatic positioning and elimination of the to-be-eliminated target in the image, and improving the user experience and bringing convenience to the user.

[0044] In one exemplary embodiment, the to-be-eliminated object includes a pedestrian; the step of detecting the region of the to-be-eliminated object included in the to-be-processed image to obtain a detection result image includes:

[0045] The portrait region of the pedestrian included in the to-be-processed image is detected to obtain a detection result image including position information of the portrait region; wherein the portrait region is a binary image.

[0046] The portrait region of the pedestrian refers to the region (positioning region) corresponding to each pedestrian included in the to-be-processed image. For example, assuming that the terminal detects a portrait region A of a pedestrian A at the lower right corner of the to-be-processed image A, the terminal can display the portrait region A as a black mask image (i.e. mark the detected portrait region A of the pedestrian A), and display other regions without detecting pedestrians as pixel content of the original image.

[0047] The binary image refers to a mask image, which can also be called a mask image or a mask region.

[0048] Specifically, taking the to-be-eliminated object as a pedestrian as an example. As shown in Figure 4 , it is a flowchart of the pedestrian detection algorithm provided by this application. Assuming that the terminal acquires the to-be-processed image A input by the user in response to the image input operation of the user, which includes 2 pedestrians (irrelevant interference objects), the terminal can detect the portrait region of the pedestrian included in the to-be-processed image A based on different detection algorithms to obtain a detection result image including position information of the portrait region (i.e. containing two portrait masks). For example, as shown in Figure 4As shown in the specification, the terminal can perform portrait instance segmentation on pedestrians included in the to-be-processed image based on an instance segmentation algorithm to obtain a portrait region instance set of the pedestrians; meanwhile, the terminal can also detect the portrait region of the pedestrians included in the to-be-processed image based on a pedestrian detection algorithm to obtain a portrait region set of the pedestrians; further, the terminal can judge the portrait region in the portrait region instance set (i.e., a large-scale portrait mask) and the portrait region set (i.e., a small-scale portrait mask) based on a passerby judgment processing logic, that is, determine the position information of a target portrait region that meets a confidence threshold, and generate a detection result image including a mask region (i.e., one or more mask regions) to be filled based on the position information of the target portrait region. Thus, even for a to-be-processed image of a complex scene with multiple interferences (such as a partially occluded passerby, a dynamically blurred pedestrian, and a person in an extreme pose), the embodiment of the present application can achieve the technical effect of accurately positioning the portrait region in the image by simultaneously using the portrait instance segmentation algorithm and the passerby detection algorithm, thereby improving the accuracy of passerby positioning, providing reliable data for subsequent elimination of the passerby region, achieving automatic positioning and elimination of the to-be-eliminated target in the image, improving the user experience, and bringing convenience to the user.

[0049] In an exemplary embodiment, the step of detecting the portrait region of the pedestrian included in the to-be-processed image to obtain the detection result image including the position information of the portrait region includes:

[0050] performing portrait instance segmentation on the to-be-processed image by a segmentation model to obtain a portrait region instance set of the pedestrian; wherein each portrait region instance in the portrait region instance set includes pixel-level mask data and a class label;

[0051] detecting the portrait region of the pedestrian included in the to-be-processed image by a detection model to obtain a portrait region set of the pedestrian;

[0052] determining the position information of the portrait region that meets the confidence threshold based on the portrait region instance set and the portrait region set;

[0053] generating a detection result image including the position information of the portrait region based on the position information of the portrait region.

[0054] The segmentation model refers to a model for segmenting the portrait area of a pedestrian in an image. For example, the segmentation model in the present application can be a trained deep learning model (Mask R-CNN), that is, the deep learning model (Mask R-CNN) is used for pixel-level segmentation of the image to be processed to obtain the portrait area mask of the pedestrian. The specific implementation includes: the Mask R-CNN segmentation model is trained in advance using the portrait instance segmentation data, and the backbone of the Mask R-CNN segmentation model in the present application uses Swin-Transformer as the Image Encoder, and other settings can remain unchanged.

[0055] For another example, the segmentation model in the present application can also be a two-branch training model composed of two encoders, commonly known as a twin network. The two-branch model in the present application is mainly used to improve the discrimination ability of the model, that is, it can adaptively adjust the information contribution of the first image feature and the second image feature extracted from the same target sample image to strengthen important features and suppress redundant information, thereby effectively improving the expression ability of the fused feature, that is, the target embedding feature, so that the fused target embedding feature considers global information and retains local details, which helps to improve the discrimination ability of the model.

[0056] The portrait area instance set refers to a set of portrait area masks, that is, the set includes multiple portrait area masks. The portrait area mask in the set is obtained by processing the segmentation model.

[0057] The detection model refers to a model for detecting the portrait area of a pedestrian in an image. For example, the detection model in the present application can be a network model based on yolo 11. The detection model in the present application is used to detect the portrait area of a small-scale pedestrian (small-scale portrait mask) in an image, that is, to detect a small-scale pedestrian (small-scale portrait mask). Figure 3 As shown in the flowchart in the present application, in order to improve the segmentation accuracy, a pedestrian detection algorithm (detection model) is used as a supplement to the aforementioned instance segmentation algorithm.

[0058] The portrait area set refers to a set of portrait area masks, that is, the set includes multiple portrait area masks. The portrait area mask in the set is obtained by processing the detection model (such as yolo 11).

[0059] The confidence threshold refers to a threshold for measuring the reliability (confidence) of the portrait area mask. For example, the confidence threshold in the present application can be set to 0.8.

[0060] Specifically, let's take pedestrians as an example to illustrate this. Figure 5 The diagram shown illustrates the process for determining the fused high-confidence pedestrian region according to this application. Assuming the terminal responds to the user's image input and obtains that the user-input image A contains two pedestrians (irrelevant interference objects), the terminal can detect the human image regions of the pedestrians in image A based on different detection algorithms to obtain a detection result image containing the location information of the human image regions (i.e., containing two human masks). For example, the terminal can use a pre-trained segmentation model to segment the human image in the image to obtain a set of pedestrian human image region instances (i.e., a large-scale human mask set); where each human image region instance in the set contains pixel-level mask data and category labels; simultaneously, the terminal can also use a detection model to detect the human image regions of the pedestrians in the image to obtain a set of pedestrian human image regions (i.e., a small-scale human mask set); further, as... Figure 5 As shown, the terminal can determine the location information of portrait regions that meet the confidence threshold (i.e., IoU greater than 0.8) based on the set of portrait region instances (i.e., the large-scale portrait mask set) and the set of portrait regions (i.e., the small-scale portrait mask set). Based on the location information of the portrait regions, it generates a detection result image that includes the location information of the portrait regions; that is, the final generated detection result image includes two black portrait mask regions. Therefore, even for images with multiple interferences in complex scenes (such as partially occluded pedestrians, dynamically blurred pedestrians, and people in extreme poses), this embodiment of the application, by simultaneously employing a portrait instance segmentation algorithm and a pedestrian detection algorithm, can achieve the technical effect of accurately locating portrait regions in the image. This improves the accuracy of pedestrian location, provides reliable data for subsequent pedestrian region removal, and realizes automatic location and removal of targets to be eliminated in the image, improving the user experience and bringing convenience to the user.

[0061] In one exemplary embodiment, the step of determining the location information of a portrait region that meets a confidence threshold based on a set of portrait region instances and a set of portrait regions includes:

[0062] Get the first bounding rectangle of each portrait region instance in the portrait region instance collection;

[0063] Obtain the second bounding rectangle of each portrait region in the portrait region set;

[0064] Based on the first and second bounding rectangles, determine the position information of the target bounding rectangle that meets the confidence threshold;

[0065] The position information of the target outer-rectangular frame is taken as the position information of the portrait region satisfying the confidence threshold.

[0066] Specifically, taking pedestrians as an example of the object to be eliminated, it is assumed that the terminal obtains a user input image A to be processed including 2 pedestrians (irrelevant interference objects), and the terminal can detect the portrait region of the pedestrian included in the image A to be processed based on different detection algorithms to obtain a detection result image including the position information of the portrait region (i.e., a mask containing two portraits). For example, the terminal can perform portrait instance segmentation on the image to be processed through a pre-trained segmentation model to obtain a set of portrait region instances of pedestrians (i.e., a set of large-scale portrait masks); at the same time, the terminal can also detect the portrait region of the pedestrian included in the image to be processed through a detection model to obtain a set of portrait regions of pedestrians (i.e., a set of small-scale portrait masks); further, as shown in Figure 5 the terminal can obtain a first outer-rectangular frame of each portrait region instance in the set of portrait region instances (i.e., the set of large-scale portrait masks) and a second outer-rectangular frame of each portrait region in the set of portrait regions (i.e., the set of small-scale portrait masks), and determine the position information of a target outer-rectangular frame satisfying a confidence threshold (i.e., IoU greater than 0.8) based on the first outer-rectangular frame and the second outer-rectangular frame, and take the position information of the target outer-rectangular frame as the position information of the portrait region satisfying the confidence threshold. Thus, the scheme provided by the present application performs much better than the traditional solution even in a high-texture region or a complex lighting condition, and has better processing capability in a complex scene, especially in processing a photo taken in a tourist attraction or other crowded area, that is, the technical effect of accurately positioning the portrait region in the image can be achieved by simultaneously using a portrait instance segmentation algorithm and a pedestrian detection algorithm, and thus the accuracy of pedestrian positioning can be improved, reliable data for subsequent elimination of the pedestrian region is provided, automatic positioning and elimination of the target to be eliminated in the image are achieved, user experience is improved, and convenience is brought to the user.

[0067] In one of the example embodiments, the step of determining the position information of the target outer-rectangular frame satisfying the confidence threshold based on the first outer-rectangular frame and the second outer-rectangular frame includes:

[0068] calculating an intersection region of the first outer-rectangular frame and the second outer-rectangular frame;

[0069] determining an intersection area of the intersection region;

[0070] calculating a union area of the first outer-rectangular frame and the second outer-rectangular frame;

[0071] determining an intersection-union ratio of the first outer-rectangular frame and the second outer-rectangular frame based on the intersection area and the union area;

[0072] Determine the position information of the target bounding rectangle that meets the confidence threshold based on the intersection over union.

[0073] Specifically, as shown in Figure 5 After the terminal obtains the first bounding rectangle of each portrait region instance in the portrait region instance set (i.e., the large-scale portrait mask set) and the second bounding rectangle of each portrait region in the portrait region set (i.e., the small-scale portrait mask set), the terminal can calculate the intersection area of the first bounding rectangle and the second bounding rectangle, and determine the intersection area of the intersection area; further, the terminal can calculate the union area of the first bounding rectangle and the second bounding rectangle, and determine the intersection over union (IoU) of the first bounding rectangle and the second bounding rectangle based on the intersection area and the union area, and determine the position information of the target bounding rectangle that meets the confidence threshold based on the intersection over union (IoU). The formula for calculating the IoU (intersection over union) of two bounding rectangles is as follows:

[0074] IoU = intersection area / union area

[0075] Specific steps:

[0076] 1. Calculate the intersection area of two bounding rectangles:

[0077] Intersection top-left x coordinate = max (bbox1 top-left x, bbox2 top-left x)

[0078] Intersection top-left y coordinate = max (bbox1 top-left y, bbox2 top-left y)

[0079] Intersection bottom-right x coordinate = min (bbox1 bottom-right x, bbox2 bottom-right x)

[0080] Intersection bottom-right y coordinate = min (bbox1 bottom-right y, bbox2 bottom-right y)

[0081] 2. Calculate the intersection area:

[0082] Intersection width = max (0, Intersection bottom-right x - Intersection top-left x)

[0083] Intersection height = max (0, Intersection bottom-right y - Intersection top-left y)

[0084] Intersection area = Intersection width * Intersection height

[0085] If there is no intersection, the width or height may be negative, so using max (0,...) can ensure that the result is not negative.

[0086] 3. Calculate the area of each bounding rectangle:

[0087] bbox1_area = (bbox1_right_bottom_x - bbox1_top_left_x) * (bbox1_right_bottom_y - bbox1_top_left_y)

[0088] bbox2_area = (bbox2_right_bottom_x - bbox2_top_left_x) * (bbox2_right_bottom_y - bbox2_top_left_y)

[0089] 4. Calculate the union area:

[0090] Union Area = bbox1_area + bbox2_area - Intersection Area

[0091] 5. Calculate the IoU:

[0092] IoU = Intersection Area / Union Area

[0093] That is, the terminal can draw the mask determined as a pedestrian in instance segmentation and the mask determined as a pedestrian in pedestrian detection on the same image (i.e., the detection result image), forming a final detection result image including a mask region to be filled. Thus, while effectively improving image processing efficiency, the visual effect of image processing can also be effectively improved, automatic positioning and elimination of the target to be eliminated in the image are achieved, and user experience is improved, bringing convenience to users.

[0094] In one of the exemplary embodiments, the detection result image includes a mask region; the mask region is determined by detecting a region of an object to be eliminated included in the image to be processed; and based on the detection result image, the step of generating an elimination result image includes:

[0095] calling a target diffusion model;

[0096] performing content filling processing on the mask region in the detection result image by the target diffusion model to obtain an elimination result image;

[0097] The target diffusion model includes a conditional control signal, and the conditional control signal is used to enhance the guiding effect of the representation of the mask region.

[0098] Specifically, as Figure 6As shown, it is a network structure diagram of the target diffusion model in the present application. Taking pedestrians as examples of objects to be eliminated, it is assumed that the terminal obtains user input of a to-be-processed image A including 2 pedestrians (irrelevant interference objects), the terminal detects the portrait area of the pedestrians included in the to-be-processed image A based on different detection algorithms, obtains a detection result image including the position information of the portrait area (i.e., a mask containing two portraits), and then the terminal can call the target diffusion model as shown in Figure 6 , and perform content filling processing on the two mask regions in the detection result image through the target diffusion model, so as to obtain an elimination result image (i.e., the two mask regions have been filled with pixel content). As shown in Figure 6 , the input of the diffusion model is a masked image, that is, the image obtained by multiplying the original image by the mask is the masked image, wherein the to-be-filled region is blackened and the non-filled region retains the original image information. In the generation stage, the diffusion model as shown in Figure 6 adopted in the present application will encode the corresponding missing region representation in the latent space according to the input masked image, and by adding the latent of the original encoder, the repair signal of the missing region is amplified, guiding the model to focus on filling the region. The latent representation of the non-mask region will mainly come from the original encoder block, thereby weakening the influence of the ControlNet, so that the model basically maintains the original content of the region. The difference between the representations inside and outside the mask region enables the model to distinguish between the repair region and the known region and give different attention.

[0099] In the present embodiment, by the above method, that is, the ControlNet module in the diffusion model explicitly provides prior information of the repair region for the Stable Diffusion, enhances the guiding role of the missing region representation, and basically maintains the encoding of the known region. The addition of ControlNet enables the model to accurately locate the region that needs to be repaired and focus on filling, rather than blindly changing the entire image, thereby greatly improving the effect of image repair. At the same time, the training of ControlNet is also relatively efficient, which only needs to train the newly added parameters without retraining the entire model, thereby also improving the processing efficiency of the model and the training efficiency.

[0100] In one of the exemplary embodiments, the detection result image includes a mask region, and the resolution of the to-be-processed image is a first resolution; the step of post-processing the elimination result image to obtain a target image not including the object to be eliminated includes:

[0101] Upsample the resulting image to obtain a target removal image with the first resolution.

[0102] Based on the masked region, the target elimination result image, and the image to be processed, determine the composite image;

[0103] The composite images are fused to obtain the fused composite image;

[0104] The fused composite image is used as the target image, excluding the objects to be eliminated.

[0105] Here, the first resolution refers to the resolution of the image to be processed, for example, the resolution of the image to be processed is (512×512). When using a diffusion model for image inpainting, due to model complexity and memory limitations, it is usually necessary to downsample the high-resolution input image (i.e., the detection result image) to a lower resolution before inputting it into the model for inpainting. After processing by the diffusion model, an inpainted result with the same resolution can be obtained. However, this process inevitably leads to the loss of some high-frequency details in the original image, making the texture and sharpness of the non-masked areas after inpainting not completely consistent with the original image (i.e., the image to be processed).

[0106] Therefore, to compensate for the image details lost during downsampling while maintaining the realism of the areas restored by the diffusion model, a post-processing step based on Poisson fusion is introduced after the diffusion model. Poisson fusion is an image editing technique that achieves seamless fusion of image regions in the gradient domain by solving a Poisson equation, preserving the details within the regions while making the transitions at the boundaries more natural and smooth.

[0107] Specifically, such as Figure 7 The diagram shows a flowchart of post-processing the eliminated image. Assuming the resolution of the image to be processed I_h acquired by the terminal is a first resolution S, the terminal can upsample the eliminated image I_d output by the diffusion model, that is, upsample the eliminated image I_d to the same resolution as the original image to be processed A, thus obtaining the target eliminated image I_d^ at the first resolution S; furthermore, as... Figure 7As shown in the figure, the terminal can determine a composite image I c based on one or more mask regions in the detection result image, the target elimination result image I d and the image to be processed I h; further, the terminal can perform Poisson fusion processing on the composite image I c (i.e., make the transition at the boundary more natural and smooth while maintaining the internal details of the region), obtain a Poisson fused composite image, and take the Poisson fused composite image as the final target image I f obtained without including the object to be eliminated. Thus, through the Poisson fusion post-processing, the repair region generated by the diffusion model can realize a natural transition with the high-frequency details of the original image on the premise of maintaining reasonableness, making up for the loss caused by downsampling, i.e., the non-mask region of the final target image completely maintains the texture and sharpness of the original image. That is, the method provided in the embodiment of the application improves the repair quality while introducing only a small computational overhead, and does not need to modify the diffusion model itself, and has good versatility and practicality. Moreover, the combination of the content-aware diffusion model and the image fusion technology can achieve a more realistic and natural effect in image repair and other tasks, effectively improving the visual effect of the elimination result image.

[0108] In one of the exemplary embodiments, based on the mask region, the target elimination result image and the image to be processed, the step of determining the composite image comprises:

[0109] Based on the mask region, the target elimination result image is divided into a first mask region and a non-first mask region;

[0110] Based on the mask region, the image to be processed is divided into a second mask region and a non-second mask region;

[0111] The first mask region and the non-second mask region are combined into a composite image.

[0112] Specifically, assuming that the resolution of the image to be processed I h obtained by the terminal is a first resolution S, the terminal performs upsampling on the elimination result image I d output by the diffusion model, i.e., upsamples the elimination result image I d to the same resolution as the original image to be processed A, to obtain a target elimination result image I d of the first resolution S, and then, as shown in the figure, the terminal combines the target elimination result image I d and the image to be processed I h to obtain a composite image I c. Figure 7As shown in the middle, the terminal can determine the composite image I_c based on one or more mask regions in the detection result image, the target elimination result image I_d^ and the to-be-processed image I_h, that is, the terminal divides the target elimination result image I_d^ into a first mask region (one or more mask regions) and a non-first mask region based on one or more mask regions in the detection result image; at the same time, the terminal divides the to-be-processed image I_h into a second mask region (one or more mask regions) and a non-second mask region based on one or more mask regions in the detection result image, and combines the first mask region (one or more mask regions) in the target elimination result image I_d^ and the non-second mask region in the to-be-processed image I_h into a composite image. That is, the pixels in the mask region in the composite image I_c are taken from the target elimination result image I_d^, and the pixels outside the mask in the composite image I_c are taken from the to-be-processed image I_h. Thus, through Poisson fusion post-processing, the repair region generated by the diffusion model can realize a natural transition with the high-frequency details of the original image under the premise of maintaining reasonableness, making up for the loss caused by downsampling, that is, the non-mask region of the final target image completely retains the texture and sharpness of the original image. That is, the method provided in the embodiment of the application improves the repair quality while introducing only a small amount of calculation overhead, and does not need to modify the diffusion model itself, and has good universality and practicality. Moreover, the combination of the content-aware diffusion model and the image fusion technology can achieve a more realistic and natural effect in image repair and other tasks, effectively improving the visual effect of the target image.

[0113] The application also provides an application scenario applying the above image processing method. The method provided in the embodiment of the application can be applied to various personalized video or image scenes in which passers-by are eliminated or removed. The image processing method provided in the embodiment of the application is described below by taking the scene in which a user interacts with an image beautification system as an example.

[0114] The passer-by elimination technology aims to detect and remove passers-by in an image, mainly involving two major links of pedestrian detection and target removal. This technology is particularly useful in outdoor shooting, which can remove passers-by in a photo to avoid affecting the overall composition and aesthetic sense. Although traditional methods such as manual smearing or cropping can achieve the purpose, they have problems such as complicated operation, unnatural effect and limited use scenarios. The traditional scheme and its drawbacks are as follows:

[0115] 1. Based on the way of manual application to eliminate pedestrians. Dynamic blur tools (such as PS dynamic blur) need users to manually adjust the blur radius, direction and intensity, which requires higher operation skills. For example, the blur direction needs to be consistent with the direction of the moving object, otherwise it is easy to show distortion. In addition, excessive blur may cause the picture to be out of focus or lose the sense of hierarchy, which needs to be combined with layer opacity (such as 70%-80%) and mask local adjustment to optimize the effect, and the operation is tedious and the effect is easy to be unnatural.

[0116] 2. Based on the way of cutting to eliminate pedestrians. Through secondary composition, the edge pedestrians are cut off. This way destroys the original composition, and the picture is easy to be incomplete; moreover, it is only effective for pedestrians on the edge, and has limited scope of application.

[0117] 3. Based on the segmentation and GAN-based elimination scheme. There will be the following limitations:

[0118] (1) One, insufficient training stability. The adversarial training mechanism of GAN has very high requirements for parameter balance, and the difference in ability between the generator and the discriminator easily leads to training shock or even mode collapse. For example, in the elimination task, if the discriminator is too strong, the generator may not be able to effectively learn the distribution characteristics of the interference elements, and if the generator is too strong, it will lead to false elimination traces.

[0119] (2) Two, mode collapse risk. GAN is prone to "false elimination" in the elimination task, that is, the generator maps multiple interference elements to the same mode, causing some real elements to be incorrectly eliminated. For example, in a complex background, GAN may map the elimination of "pedestrians" and "leaves" to similar texture features, causing over-processing.

[0120] (3) Three, high sensitivity to input. The generation result of GAN is easily affected by input noise, and slight parameter adjustment or input disturbance may cause elimination effect distortion. For example, in the edge detection task, GAN may cause the elimination boundary to appear jagged due to slight rotation of the input image.

[0121] (4) Four, limited generation quality and detail restoration. In scenarios requiring high-precision elimination, GAN-generated elimination areas often appear blurred or have artifacts.

[0122] (5) Five, insufficient generalization ability. The elimination result generated by GAN is easily limited by training data, and has poor adaptability to unseen scenarios (such as extreme lighting and rare objects).

[0123] In order to improve the user experience, in view of the problems existing in the above-mentioned traditional ways, the application proposes a pedestrian elimination method and system based on diffusion model, that is, the application proposes a new scheme for pedestrian elimination based on diffusion model. For example Figure 8As shown in the overall framework diagram of a pedestrian elimination method and system based on a diffusion model provided by the present application, as shown in Figure 8 It is composed of pedestrian detection algorithm, elimination algorithm based on diffusion model and post-processing fusion algorithm module, which can realize automatic positioning and elimination of pedestrians in images. Compared with the previous scheme, it has the following advantages:

[0124] 1. The training stability is significantly improved. The diffusion model adopts a denoising training paradigm, avoiding the instability of GAN adversarial training. In the pedestrian elimination task, the model can more stably learn the distinguishing features of background texture and foreground interference, reducing the training shock phenomenon.

[0125] 2. Higher generation quality and detail fidelity. The diffusion model can more accurately reconstruct the background details through the iterative denoising process, especially in texture-rich areas (such as grass, water surface, building surface, etc.), preserving the authenticity and coherence of the original scene.

[0126] 3. Enhanced context perception ability. The scheme provided by the present application can understand the global scene semantics and consider the structural relationship of the surrounding environment when eliminating pedestrians, making the reconstructed area and the surrounding environment transition naturally, avoiding the common "patch feeling" in traditional methods.

[0127] 4. The generalization ability is significantly improved. Compared with GAN models, diffusion models show stronger adaptability to unseen scenes (such as different lighting conditions, complex weather environments, and diverse pedestrian density), with a wider range of applications.

[0128] 5. High degree of automation. Combined with advanced pedestrian detection algorithms, the system can automatically identify pedestrians in various poses and occlusion states without the need for user manual annotation, greatly improving work efficiency.

[0129] 6. Natural edge processing. In the boundary area between pedestrians and the background, the post-processing fusion algorithm provided by the present application can intelligently mix original pixels and generated content, avoiding the common edge artifacts and discontinuity problems in traditional methods.

[0130] 7. Preserve scene semantic consistency. While eliminating pedestrians, the model can understand and maintain the structural features of the scene (such as ground perspective, building lines, and light and shadow relationships), making the processed image visually logical and reasonable.

[0131] 8. Strong anti-interference ability. For multiple interferences in complex scenes (such as partially occluded pedestrians, dynamically blurred pedestrians, and people in extreme poses), the scheme provided by the present application shows stronger robustness.

[0132] As shown in the overall process in Figure 3

[0133] ​Module 1: Pedestrian Detection Algorithm

[0134] Input: User's original image (contains initial pictures of pedestrians).

[0135] Output: Detection result image with annotated pedestrian positions (image with masks).

[0136] Function:

[0137] 1. Accurately locate the portrait area in the image through portrait instance segmentation and pedestrian detection algorithm.

[0138] 2. According to the defined pedestrian judgment strategy, distinguish between the owner's mask and the pedestrian mask.

[0139] 3. Output structured data containing pedestrian position information (such as coordinates, masks), providing basis for subsequent elimination.

[0140] Module 2: Pedestrian Elimination Algorithm

[0141] Input: Detection result image output by module 1 (contains pedestrian position annotations).

[0142] Output: Preliminary pedestrian-eliminated image (background area is filled, but may have edge artifacts or inconsistent texture).

[0143] Function:

[0144] 1. Based on the detection results, use generative models (Diffusion model) to fill in the background obscured by pedestrians.

[0145] 2. Combine context information (such as surrounding texture, lighting) to generate reasonable filling content, ensuring visual reasonableness.

[0146] Module 3: Original Image Fusion Post-processing

[0147] Input: Preliminary elimination result image output by module 2.

[0148] Output: Final optimized pedestrian-free image (high naturalness, no obvious processing traces).

[0149] Function:

[0150] 1. Multi-scale fusion: multi-level fusion (such as pyramid fusion) of the elimination result and the original image to reduce edge jaggedness and color difference.

[0151] 2. Detail enhancement: optimize the texture details of the filling area through sharpening, noise reduction, and other post-processing techniques (such as repairing repetitive patterns of grass and brick walls).

[0152] 3. Global consistency adjustment: unify the color, lighting, and contrast of the entire image to ensure seamless integration of the elimination area and the original background.

[0153] The following is a detailed introduction to several algorithm modules:

[0154] I. Pedestrian detection algorithm

[0155] The following is a detailed description of the pedestrian recognition algorithm module based on the flowchart shown in Figure 4

[0156] 2. Instance segmentation algorithm module

[0157] Input: User-provided original image.

[0158] Output: Segmented instance set (each instance contains a pixel-level mask and a class label).

[0159] Function: Use a deep learning model (Mask R-CNN) for pixel-level segmentation of images.

[0160] Specific implementation:

[0161] a. Use portrait instance segmentation data to train the Mask R-CNN segmentation model

[0162] b. Use Swin-Transformer as the Image Encoder in the backbone of Mask R-CNN, and keep other settings unchanged.

[0163] The purpose of using portrait instance segmentation is to obtain the mask of portraits with a large scale (large interview area). However, for small-scale long-distance pedestrians (small area), instance segmentation is difficult to achieve accurate segmentation. To improve the segmentation accuracy, a pedestrian detection algorithm is used as a supplement, which is introduced as follows:

[0164] 3. Pedestrian detection algorithm module

[0165] Input: User-provided original image.

[0166] Output: Pedestrian region bounding box (Bounding Box) or region mask.

[0167] Function: Use a target detection algorithm (such as yolo 11) to locate small-scale pedestrians.

[0168] Specific implementation:

[0169] a. For small-scale long-distance pedestrians, label their circumscribed rectangular frame (portrait area is very small, the difference between segmentation mask and bbox is not large) as training data.

[0170] ​b. Train using yolo 11 as detection model.

[0171] 4. Pedestrian determination logic module

[0172] Input: fused pedestrian region data.

[0173] Output: target region determined as "pedestrian".

[0174] Function: filter non-pedestrian targets based on preset rules, retain subject portrait mask, output final determined "pedestrian" target and its spatial position.

[0175] The main function of this module is to distinguish between the subject and pedestrians, which mainly judges each portrait mask in instance segmentation according to the preset rules, including:

[0176] According to the user's shooting composition habit, the prediction rule is as follows:

[0177] 1. Find the mask of the portrait in the center of the picture, and the mask with the largest portrait ratio, set it as the subject portrait mask, and calculate the circumscribed rectangle of the subject portrait mask to get the length and width

[0178] 2. Extend the range to the left and right by 3 times the width to find other portrait masks, calculate the area ratio (pixel number ratio) of the mask and the subject portrait mask, if the area is less than 40% of the subject portrait mask, it is determined as the subject portrait, otherwise it is determined as the pedestrian.

[0179] 5. Result fusion module

[0180] Input: pedestrian instance mask set output by instance segmentation, pedestrian region bbox set output by pedestrian detection.

[0181] Output: fused high-confidence pedestrian region.

[0182] Function: merge the results of the two modules through spatial alignment and logical operations (such as intersection, union), that is, for the area that can be detected by portrait instance segmentation, use the mask of portrait instance segmentation, for the area that cannot be detected, use the bbox of pedestrian detection as the pedestrian mask. The specific implementation is shown in Figure 5 .

[0183] Wherein, the formula for calculating the IoU (intersection over union) of the two rectangular frames is shown in the following formula:

[0184] IoU = intersection area / union area

[0185] Specific steps:

[0186] 1. Compute the intersection area of two rectangular bboxes:

[0187] intersection left-top x = max(bbox1 left-top x, bbox2 left-top x)

[0188] intersection left-top y = max(bbox1 left-top y, bbox2 left-top y)

[0189] intersection right-bottom x = min(bbox1 right-bottom x, bbox2 right-bottom x)

[0190] intersection right-bottom y = min(bbox1 right-bottom y, bbox2 right-bottom y)

[0191] 2. Compute the intersection area:

[0192] intersection width = max(0, intersection right-bottom x - intersection left-top x)

[0193] intersection height = max(0, intersection right-bottom y - intersection left-top y)

[0194] intersection area = intersection width * intersection height

[0195] If there is no intersection, the width or height can be negative, so use max(0,...) to ensure the result is not negative.

[0196] 3. Compute the area of each bbox:

[0197] bbox1 area = (bbox1 right-bottom x - bbox1 left-top x) * (bbox1 right-bottom y - bbox1 left-top y)

[0198] bbox2 area = (bbox2 right-bottom x - bbox2 left-top x) * (bbox2 right-bottom y - bbox2 left-top y)

[0199] 4. Compute the union area:

[0200] union area = bbox1 area + bbox2 area - intersection area

[0201] 5. Compute the IoU:

[0202] IoU = intersection area / union area

[0203] That is, the terminal can draw the mask determined as a pedestrian in instance segmentation and the mask determined as a pedestrian in pedestrian detection on the same graph (i.e., a detection result image), forming a final detection result image including a to-be-filled mask region.

[0204] 6. To-be-filled mask region

[0205] Input: Pedestrian determination result mask.

[0206] Output: Binary mask of the to-be-repaired region.

[0207] Function: Generate a pedestrian region mask that needs to be removed from the original image, and provide an operation region for subsequent image inpainting (such as the Inpainting algorithm), which may include edge extension processing to ensure natural inpainting effect.

[0208] II. Pedestrian elimination algorithm

[0209] The core framework of the overall control is shown in Figure 6 , where the network structure includes Encoder, Decoder, and Unet (this part of the architecture is the same as stable diffusion), and ControlNet is also used to finely guide the control of the image drawing region.

[0210] As shown in Figure 6 , the terminal first encodes the image into a lower-dimensional latent space, and trains a diffusion model in the latent space. For the picture expansion task, the diffusion model can be trained by concatenating the masked image and the original complete image in the latent space. The LDM trained in this way can fill in the missing content according to the context information around the mask region.

[0211] As shown in Figure 6 , the input is: masked image, which is the image obtained by multiplying the original image by the mask, where the to-be-filled region is blacked out and the non-filled region retains the original image information.

[0212] Module:

[0213] 1. The encoder input masked image, map to latent space, output get latent space feature representation z = E(x), which is a 4-channel feature map. According to the time step t of the diffusion process, Gaussian noise is added to the latent space representation z to obtain z_t. That is, z_t = sqrt(1-β_t) * z + sqrt(β_t) *ε, where ε is a standard Gaussian distribution, and β_t is the noise variance at time t. The noisy latent space representation z_t and the time step t are input to the diffusion model (i.e. UNet) to obtain the noise estimate ε_θ.

[0214] 2. The input of the UNet is the 4-channel feature map input by the encoder, the set image mask, and the masked feature map obtained by multiplying the 4-channel feature map with the mask, a total of 9 channels. The output is a 4-channel extended feature map.

[0215] 3. The decoder is used to map the latent space back to the image space, and the input is the output feature map of module 5, and the output is the locally modified image.

[0216] 4. In order to give the model stronger guidance information, and make the model pay more attention to the filling of local areas, rather than the filling of part of the original image area, a contolnet module is also added to strengthen it.

[0217] Basic structure:

[0218] ControlNet injects additional conditional control signals into pre-trained text-to-image diffusion models such as Stable Diffusion, enabling guidance of the image generation process.

[0219] A trainable control block is added to each encoder block of the diffusion model, which is parallel to the original block.

[0220] The conditional control signal c is converted in dimension through a 1x1 convolution layer and concatenated with the input of the encoder block.

[0221] In this scheme, the input of the controlnet is the masked image, and the output is the encoded information with the same dimension as the stablediffusion encoder block, which is added to the network through concat.

[0222] In the generation phase, the method used in this application is as follows: Figure 6The diffusion model shown in the middle can encode the corresponding missing area representation in the latent space according to the input maskedimage graph. By adding the latent of the original encoder, the repair signal of the missing area is amplified, guiding the model to focus on filling the area. The latent representation of the non-mask area mainly comes from the original encoder block, thereby weakening the influence of the ControlNet, so that the model basically maintains the original content of the area. The difference between the representations inside and outside the mask area enables the model to distinguish between the repair area and the known area and give different attention.

[0223] In this embodiment, by the above method, that is, using the ControlNet module in the diffusion model to explicitly provide prior information of the repair area for Stable Diffusion, the guiding effect of the missing area representation is enhanced, and the encoding of the known area is basically maintained. The addition of ControlNet enables the model to accurately locate the area to be repaired and focus on filling, rather than blindly changing the entire image, thereby greatly improving the effect of image repair. At the same time, the training of ControlNet is also relatively efficient, which only needs to train the newly added parameters without retraining the entire model, thereby improving the processing efficiency of the model and the training efficiency.

[0224] III. Image fusion post-processingWhen using a diffusion model for image repair, due to the complexity of the model and the memory limit, it is usually necessary to downsample a high-resolution input image to a lower resolution (such as 512x512) before inputting it into the model for repair. After processing by the diffusion model, a repair result with the same resolution can be obtained. However, this process inevitably leads to the loss of some high-frequency details in the original image, making the texture and sharpness of the non-mask area after repair not completely consistent with the original image.

[0225] To make up for the loss of image details in the downsampling process while maintaining the realism of the repair area of the diffusion model, a post-processing step based on Poisson fusion is introduced after the diffusion model. Poisson fusion is an image editing technique that solves a Poisson equation to achieve seamless fusion of image regions in the gradient domain, maintaining the details inside the region while making the transition at the boundary more natural and smooth.

[0226] Specifically, the scheme provided by the present application is as shown in Figure 7

[0227] 1. Upsample the output image I_d of the diffusion model to the original image resolution, denoted as I_d^.

[0228] 2. Use the original mask to divide I_d^ and the high-resolution original image I_h into regions. Pixels within the mask region are taken from I_d^, and pixels outside the mask region are taken from I_h, to obtain a composite image I_c. At this time, I_c has obvious seams at the mask boundary.

[0229] 3. Smooth the mask boundary of I_c using Poisson fusion. Denote the mask region as Ω and the boundary as βΩ. The Poisson equation is defined as: ▽^2 f = div(v) in Ω f = f^* on βΩ Where f is the fused image, v is the gradient field of I_c, and f^* is the pixel value of the boundary point. Solving this equation can adjust the absolute value of the pixels at the boundary to achieve a smooth transition with the region outside βΩ while keeping the gradient unchanged within Ω.

[0230] 4. Solve the Poisson equation for each of the three channels of I_c to obtain the final fusion result I_f.

[0231] After Poisson fusion post-processing, the repair area generated by the diffusion model can achieve a natural transition with the high-frequency details of the original image while maintaining reasonableness, making up for the loss caused by downsampling. The non-mask region completely retains the texture and sharpness of the original image. The scheme provided in the present application improves the repair quality while introducing only a small computational overhead, does not need to modify the diffusion model itself, and has good universality and practicality. Furthermore, combining the content-aware diffusion model with image fusion technology can achieve more realistic and natural results in image repair and other tasks. The specific process is shown in Figure 7 , which includes the following steps:

[0232] 1. Downsample the high-resolution original image I_h to the resolution required by the diffusion model.

[0233] 2. Input the downsampled image into the diffusion model for repair to obtain a low-resolution repair result.

[0234] 3. Upsample the repair result of the diffusion model to the resolution of the original image to obtain I_d^.

[0235] 4. Use the original mask to divide I_h and I_d^ into regions to obtain a composite image I_c.

[0236] 5. Perform Poisson fusion processing on I_c, and solve the Poisson equation for each of the R, G, and B channels.

[0237] 6. Combine the fusion results of the three channels to obtain the final high-resolution inpainting image I_f.

[0238] The beneficial effects produced by the technical solutions of the present application include:

[0239] The pedestrian elimination scheme based on diffusion model provided by the present application has the following significant advantages:

[0240] 1. Technological leadership: Most existing technologies still use GAN or traditional beautification methods, while the present application adopts more advanced diffusion model technology, which is at the forefront of the image generation field.

[0241] 2. Complete solution: The present application provides a complete pipeline from detection to elimination and post-processing, while traditional technologies usually only provide a single-point solution, lacking systematic integration.

[0242] 3. Balance between professionalism and universality: Optimized for pedestrian elimination scenarios, the effect is better than that of general image inpainting tools, while maintaining adaptability to various scenarios.

[0243] 4. Commercial software integration potential: High degree of automation, easy to integrate into photography workflow, more practical than the need for professional editing software operation.

[0244] 5. Complex scene processing capability: Performs much better than existing solutions in high-texture areas and complex lighting conditions, especially in processing photos taken in crowded areas such as tourist attractions.

[0245] 6. Computational efficiency: Although diffusion models usually require iterative reasoning, the scheme provided by the present application may provide a better balance between speed and quality after optimization, making practical application more feasible.

[0246] 7. Batch processing capability: High automation makes it very suitable for photographers or tourism photo processing companies to process batch photos, improving work efficiency.

[0247] 8. No need for professional skills: Traditional solutions usually require users to have certain image editing skills, while the scheme of the present application lowers the threshold and expands the potential user group.

[0248] These advantages make the scheme provided by the present application have obvious competitiveness in the markets of professional photography, tourism photo processing and social media image optimization.

[0249] It should be understood that although the steps in the flowcharts involved in the embodiments described above are shown in sequence according to the arrows, the steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders. Moreover, at least some of the steps in the flowcharts involved in the embodiments described above can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the steps or stages is not necessarily sequential, but can be alternately or alternately executed with at least part of other steps or steps or stages in other steps.

[0250] Based on the same inventive concept, the embodiments of the present application also provide an image processing device for implementing the image processing method described above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more image processing device embodiments provided below can refer to the limitations of the image processing method described above, which will not be repeated here.

[0251] In one exemplary embodiment, as shown in Figure 9 An image processing device is provided, comprising: an acquisition module 902, a detection module 904, a generation module 906, and a processing module 908, wherein:

[0252] The acquisition module 902 is configured to acquire a to-be-processed image, wherein the to-be-processed image includes a to-be-eliminated object.

[0253] The detection module 904 is configured to detect a region of the to-be-eliminated object included in the to-be-processed image to obtain a detection result image.

[0254] The generation module 906 is configured to generate an elimination result image based on the detection result image.

[0255] The processing module 908 is configured to perform post-processing on the elimination result image to obtain a target image that does not include the to-be-eliminated object.

[0256] In one embodiment, the to-be-eliminated object includes a pedestrian; the detection module is further configured to detect a portrait region of the pedestrian included in the to-be-processed image to obtain a detection result image including position information of the portrait region; and the portrait region is a binary image.

[0257] In an embodiment, the detection module is further configured to perform portrait instance segmentation on the to-be-processed image by using a segmentation model to obtain a set of portrait region instances of the pedestrian; each portrait region instance in the set of portrait region instances includes pixel-level mask data and a class label; the detection module is further configured to detect a portrait region of the pedestrian included in the to-be-processed image by using a detection model to obtain a set of portrait regions of the pedestrian; the apparatus further includes a determination module configured to determine position information of a portrait region that meets a confidence threshold based on the set of portrait region instances and the set of portrait regions; and a generation module configured to generate a detection result image including the position information of the portrait region based on the position information of the portrait region.

[0258] In an embodiment, the acquisition module is further configured to acquire a first bounding rectangle of each portrait region instance in the set of portrait region instances; acquire a second bounding rectangle of each portrait region in the set of portrait regions; the determination module is further configured to determine position information of a target bounding rectangle that meets a confidence threshold based on the first bounding rectangle and the second bounding rectangle; and the position information of the target bounding rectangle is used as the position information of the portrait region that meets the confidence threshold.

[0259] In an embodiment, the detection result image includes a mask region; the mask region is determined by detecting a region of the to-be-eliminated object included in the to-be-processed image; the apparatus further includes a calling module configured to call a target diffusion model; and the generation module is further configured to perform content filling processing on the mask region in the detection result image by using the target diffusion model to obtain the elimination result image; the target diffusion model includes a condition control signal, and the condition control signal is used to enhance the guiding effect on the representation of the mask region.

[0260] In an embodiment, the detection result image includes a mask region, and the to-be-processed image has a first resolution; the processing module is further configured to perform up-sampling on the elimination result image to obtain a target elimination result image having the first resolution; the apparatus further includes a determination module configured to determine a composite image based on the mask region, the target elimination result image, and the to-be-processed image; the processing module is further configured to perform fusion processing on the composite image to obtain a fused composite image; and the fused composite image is used as a target image that does not include the to-be-eliminated object.

[0261] In one embodiment, the apparatus further comprises: a division module configured to divide the target elimination result image into a first mask region and a non-first mask region based on the mask region; divide the image to be processed into a second mask region and a non-second mask region based on the mask region; and a combination module configured to combine the first mask region and the non-second mask region into the composite image.

[0262] Each of the above-mentioned modules in the image processing apparatus can be implemented wholly or partially by software, hardware, and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so as to be called and executed by a processor to perform operations corresponding to each of the above-mentioned modules.

[0263] In one exemplary embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in Figure 10 The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to perform wired or wireless communication with external terminals, and the wireless communication can be implemented through WIFI, mobile cellular network, Near Field Communication (NFC), or other technologies. The computer program is executed by the processor to implement an image processing method. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, a trackball, or a touchpad arranged on the shell of the computer device, or can be an external keyboard, a touchpad, a mouse, or the like.

[0264] Those skilled in the art can understand that Figure 10 The structure shown in the above-mentioned figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or can combine certain components, or have a different arrangement of components.

[0265] In an example embodiment, a computer device is also provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps in the above method embodiments when executing the computer program.

[0266] In an example embodiment, a computer readable storage medium is provided, storing a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0267] In an example embodiment, a computer program product is provided, including a computer program, and the computer program implementing the steps in the above method embodiments when executed by a processor.

[0268] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0269] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. In the embodiments provided in the present application, any reference to memory, database or other medium can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0270] The technical features of the above embodiments can be combined in any way. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present application.

[0271] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. An image processing method, characterized in that, The method includes: Obtain the image to be processed; the image to be processed includes the object to be eliminated; The region containing the object to be eliminated in the image to be processed is detected to obtain a detection result image; Based on the detection result image, generate the elimination result image; The resulting image is post-processed to obtain a target image that does not include the object to be eliminated.

2. The method according to claim 1, characterized in that, The objects to be eliminated include pedestrians; The step of detecting the region of the object to be eliminated included in the image to be processed to obtain a detection result image includes: The human figure region of the pedestrian included in the image to be processed is detected to obtain a detection result image including the location information of the human figure region; wherein, the human figure region is a binarized image.

3. The method according to claim 2, characterized in that, The step of detecting the human figure region of the pedestrian included in the image to be processed, and obtaining a detection result image including the location information of the human figure region, includes: The image to be processed is segmented into human figures using a segmentation model to obtain a set of human figure regions for the pedestrian; wherein each human figure region instance in the set contains pixel-level mask data and category labels. The detection model is used to detect the human image regions of the pedestrians included in the image to be processed, thereby obtaining a set of human image regions of the pedestrians; Based on the set of human image region instances and the set of human image regions, determine the location information of human image regions that meet the confidence threshold; Based on the location information of the human image region, a detection result image including the location information of the human image region is generated.

4. The method according to claim 3, characterized in that, The step of determining the location information of the portrait region that meets the confidence threshold based on the set of portrait region instances and the set of portrait regions includes: Obtain the first bounding rectangle of each portrait region instance in the portrait region instance set; Obtain the second bounding rectangle of each portrait region in the portrait region set; Based on the first bounding rectangle and the second bounding rectangle, determine the position information of the target bounding rectangle that meets the confidence threshold; The location information of the target's bounding rectangle is used as the location information of the portrait region that meets the confidence threshold.

5. The method according to claim 1, characterized in that, The detection result image includes a mask region; the mask region is determined by detecting the region of the object to be eliminated included in the image to be processed; The step of generating an elimination result image based on the detection result image includes: Invoke the target diffusion model; The target diffusion model is used to fill in the masked region in the detection result image to obtain the removal result image; The target diffusion model includes a conditional control signal, which is used to enhance the guidance of the representation of the mask region.

6. The method according to claim 1, characterized in that, The detection result image includes a masked region, and the resolution of the image to be processed is a first resolution; The post-processing of the eliminated result image to obtain a target image that does not include the object to be eliminated includes: The elimination result image is upsampled to obtain the target elimination result image with the first resolution; Based on the masked region, the target elimination result image, and the image to be processed, a composite image is determined; The composite image is fused to obtain a fused composite image; The fused composite image is used as the target image that does not include the object to be eliminated.

7. The method according to claim 6, characterized in that, The step of determining the composite image based on the mask region, the target elimination result image, and the image to be processed includes: Based on the mask region, the target elimination result image is divided into a first mask region and a non-first mask region; Based on the mask region, the image to be processed is divided into a second mask region and a non-second mask region; The first masked region and the non-second masked region are combined to form the composite image.

8. An image processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the image to be processed; the image to be processed includes the object to be eliminated; The detection module is used to detect the region of the object to be eliminated included in the image to be processed, and obtain a detection result image; The generation module is used to generate an elimination result image based on the detection result image; The processing module is used to post-process the elimination result image to obtain a target image that does not include the object to be eliminated.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images