Image processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method, device, electronic device and storage medium, and belongs to the field of computer technology. The method comprises: obtaining an area to be erased of an original image, the area to be erased includes an object to be erased; based on the area to be erased, generating an erased area that does not include the object to be erased; filtering the erased area to obtain a target area after filtering out high-frequency information; replacing the area to be erased in the original image with the target area to obtain a target image. The present disclosure greatly improves the erasing effect of the object to be erased by filtering out the afterimage and residual noise left when erasing the object to be erased in the filtering stage.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the development of computer technology, users can perform secondary editing on images or video frames on the terminal to achieve the purpose of secondary creation and content generation. Usually, there is a processing requirement to erase an object in the image, such as erasing watermarks in the image, obstacles that block the line of sight, passers-by who accidentally enter the camera, and facilities in the background that affect the appearance.

[0003] Taking watermark erasure as an example, for watermarked images, the server deploys the STTN (Spatial-Temporal Transformations Networks) model based on the Transformer framework to remove watermarks and fill in content for watermarked images. However, after the image is processed by the STTN model, many afterimages will appear in the watermark erasure area, that is, the erasing effect of the specified object in the image is poor. Summary of the invention

[0004] The present disclosure provides an image processing method, device, electronic device and storage medium to at least improve the erasing effect of a specified object in an image. The technical solution of the present disclosure is as follows:

[0005] According to one aspect of an embodiment of the present disclosure, there is provided an image processing method, including:

[0006] Acquire a to-be-erased area of ​​the original image, wherein the to-be-erased area includes an object to be erased;

[0007] Based on the to-be-erased area, generating an erased area that does not include the to-be-erased object;

[0008] Filtering the erased area to obtain a target area after filtering out high-frequency information;

[0009] The area to be erased in the original image is replaced by the target area to obtain a target image.

[0010] In a possible implementation manner, filtering the erased area to obtain a target area after filtering out high-frequency information includes:

[0011] The erased area is input into an edge-preserving filter, and the edge-preserving filter filters out high-frequency information in the erased area while retaining edge information in the erased area, and outputs the target area.

[0012] In one possible implementation, the edge-preserving filter is a bilateral filter;

[0013] The filtering out high-frequency information in the erasing area by the edge-preserving filter while retaining edge information in the erasing area, and outputting the target area comprises:

[0014] For any pixel point in the erasing area, taking the pixel point as the center, sampling in the erasing area obtains a plurality of neighboring pixel points around the pixel point;

[0015] Determine a weighting coefficient for each neighborhood pixel, where the weighting coefficient is determined based on a Euclidean distance and a grayscale difference between the neighborhood pixel and the pixel;

[0016] Based on the weighting coefficient of each neighborhood pixel point, the pixel value of each neighborhood pixel point is weighted, and the weighted pixel values ​​are added together to obtain the pixel value of the pixel point in the target area at the same position as the pixel point.

[0017] In a possible implementation manner, determining the weighting coefficient of each neighborhood pixel includes:

[0018] Determining a distance weight component based on the Euclidean distance between the neighborhood pixel point and the pixel point;

[0019] Determining a color weight component based on a grayscale difference between the neighborhood pixel and the pixel;

[0020] The distance weight component and the color weight component are multiplied to obtain a weight coefficient of the neighborhood pixel point.

[0021] In a possible implementation manner, generating an erasure area that does not include the object to be erased based on the area to be erased includes:

[0022] Adding a mask in the area to be erased, wherein the mask is used to cover the object to be erased;

[0023] Based on the background content in the area to be erased except the mask, generating foreground content corresponding to the mask, wherein the foreground content matches the background content;

[0024] The mask in the area to be erased is replaced with the foreground content to obtain the erased area.

[0025] In a possible implementation manner, obtaining the area to be erased of the original image includes:

[0026] Determine the area to be erased based on the area location parameter input by the account number, wherein the area location parameter is used to indicate the location of the area to be erased; or,

[0027] Based on the object to be erased inputted by the account number, the area to be erased including the object to be erased is detected from the original image.

[0028] In a possible implementation manner, replacing the to-be-erased area in the original image with the target area to obtain the target image includes:

[0029] The pixel value of each pixel point in the target area is assigned to each pixel point at a corresponding position in the area to be erased to obtain the target image.

[0030] In a possible implementation manner, the object to be erased is an image watermark or a video watermark.

[0031] According to another aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:

[0032] An acquisition unit, configured to acquire a to-be-erased area of ​​the original image, wherein the to-be-erased area includes an object to be erased;

[0033] A generating unit, configured to generate an erasure area that does not include the object to be erased based on the area to be erased;

[0034] A filtering unit, configured to filter the erased area to obtain a target area after high-frequency information is filtered out;

[0035] The replacement unit is configured to replace the to-be-erased area in the original image with the target area to obtain a target image.

[0036] In a possible implementation manner, the filtering unit includes:

[0037] An input subunit, configured to input the erased area into an edge-preserving filter;

[0038] a filtering subunit configured to filter out high-frequency information in the erasure area while retaining edge information in the erasure area through the edge-preserving filter;

[0039] The output subunit is configured to output the target area.

[0040] In one possible implementation, the edge-preserving filter is a bilateral filter;

[0041] The filtering subunit comprises:

[0042] The sampling sub-sub-unit is configured to perform sampling on any pixel point in the erasing area, taking the pixel point as the center, to obtain a plurality of neighboring pixel points around the pixel point in the erasing area;

[0043] A determination sub-subunit is configured to determine a weighting coefficient of each neighborhood pixel point, wherein the weighting coefficient is determined based on a Euclidean distance and a grayscale difference between the neighborhood pixel point and the pixel point;

[0044] The adding sub-subunit is configured to perform weighting on the pixel value of each neighborhood pixel based on the weighting coefficient of each neighborhood pixel, and add the weighted pixel values ​​to obtain the pixel value of the pixel in the target area with the same position as the pixel.

[0045] In one possible implementation, the determining sub-subunit is configured to execute:

[0046] Determining a distance weight component based on the Euclidean distance between the neighborhood pixel point and the pixel point;

[0047] Determining a color weight component based on a grayscale difference between the neighborhood pixel and the pixel;

[0048] The distance weight component and the color weight component are multiplied to obtain a weight coefficient of the neighborhood pixel point.

[0049] In one possible implementation, the generating unit is configured to execute:

[0050] Adding a mask in the area to be erased, wherein the mask is used to cover the object to be erased;

[0051] Based on the background content in the area to be erased except the mask, generating foreground content corresponding to the mask, wherein the foreground content matches the background content;

[0052] The mask in the area to be erased is replaced with the foreground content to obtain the erased area.

[0053] In a possible implementation manner, the acquiring unit is configured to execute:

[0054] Determine the area to be erased based on the area location parameter input by the account number, wherein the area location parameter is used to indicate the location of the area to be erased; or,

[0055] Based on the object to be erased inputted by the account number, the area to be erased including the object to be erased is detected from the original image.

[0056] In one possible implementation, the replacement unit is configured to execute:

[0057] The pixel value of each pixel point in the target area is assigned to each pixel point at a corresponding position in the area to be erased to obtain the target image.

[0058] In a possible implementation manner, the object to be erased is an image watermark or a video watermark.

[0059] According to another aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0060] one or more processors;

[0061] one or more memories for storing the one or more processor-executable instructions;

[0062] The one or more processors are configured to execute the image processing method in any possible implementation of the above aspect.

[0063] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When at least one instruction in the computer-readable storage medium is executed by one or more processors of an electronic device, the electronic device is enabled to execute the image processing method in any possible implementation of the above-mentioned aspect.

[0064] According to another aspect of an embodiment of the present disclosure, a computer program product is provided, comprising one or more instructions, wherein the one or more instructions can be executed by one or more processors of an electronic device, so that the electronic device can execute the image processing method in any possible implementation manner of the above-mentioned aspect.

[0065] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0066] The object to be erased in the area to be erased is erased to obtain the erased area, and the target area is obtained by filtering based on the erased area, and then the target area is pasted back to the original image to obtain the target image, so that the afterimage and residual noise left when erasing the object to be erased can be filtered out in the filtering stage, thereby greatly improving the erasing effect of the object to be erased.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0069] Figure 1 This is an effect diagram of erasing a watermark based on the STTN model provided by an embodiment of the present disclosure;

[0070] Figure 2This is an effect diagram of erasing a watermark based on the STTN model provided by an embodiment of the present disclosure;

[0071] Figure 3 is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment;

[0072] Figure 4 is a flowchart of an image processing method according to an exemplary embodiment;

[0073] Figure 5 is an interactive flow chart of an image processing method according to an exemplary embodiment;

[0074] Figure 6 It is a principle flow chart of a watermark erasing method provided by an embodiment of the present disclosure;

[0075] Figure 7 is a schematic diagram of deducting a ROI area provided by an embodiment of the present disclosure;

[0076] Figure 8 is a comparison diagram of watermark removal effects provided by an embodiment of the present disclosure;

[0077] Fig. 9 is a comparison diagram of watermark removal effects provided by an embodiment of the present disclosure;

[0078] Fig.10 is a comparison diagram of watermark removal effects provided by an embodiment of the present disclosure;

[0079] Fig.11 is a comparison diagram of watermark removal effects provided by an embodiment of the present disclosure;

[0080] Fig.12 is a logical structure block diagram of an image processing device according to an exemplary embodiment;

[0081] Fig.13 A structural block diagram of a terminal provided by an exemplary embodiment of the present disclosure is shown;

[0082] Fig.14 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0083] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0084] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0085] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the original images involved in this disclosure were obtained with full authorization.

[0086] In some embodiments, the meanings of A and / or B include: A and B, A, B.

[0087] The following is an explanation of the terms in the embodiments of the present disclosure:

[0088] Digital Watermark: refers to the technology of embedding specific digital signals into digital products to protect the copyright, integrity, anti-copying or whereabouts tracking of digital products. Digital watermarks can be divided into image watermarks loaded on image data and video watermarks loaded on video data according to the carrier they are loaded on. The object to be erased involved in the embodiments of the present disclosure may include an image watermark or a video watermark.

[0089] Object Detection: The task of object detection is to find all the targets of interest (objects) in the image and determine their categories and locations. It is one of the core issues in the field of computer vision. Since various objects have different appearances, shapes and postures, and are interfered by factors such as illumination and occlusion during imaging, object detection has always been the most challenging problem in the field of computer vision. The disclosed embodiment involves: the user gives an object to be erased, and the machine performs object detection from the original image to determine the area to be erased that contains the object to be erased, such as marking the area to be erased with a rectangular frame.

[0090] Wave Filtering: It is an operation to filter out specific frequency bands in a signal, and is an important measure to suppress and prevent interference. The disclosed embodiment involves filtering a noisy image to remove noise information in the image, and is therefore also considered a noise reduction or denoising process. That is, the pixel values ​​of some pixels in an image are modified to make the image smoother and more continuous, or to reduce or delete noise points (or outliers) in the image.

[0091] Edge Preserving Filter: For noisy images, noise and edges have similar local variances. General filters cannot distinguish between noise and edges, so they are treated uniformly. Therefore, in many cases, the edges are blurred while filtering. For example, linear smoothing filters such as Gaussian filters are not edge preserving filters. While removing noise, they blur all textures, i.e. edges, in the image, causing the background details in the filtered image to become blurred. Edge preserving filters refer to a special type of filter that can effectively preserve the edge information in the image during the filtering process. Edge preserving filters include: bilateral filter, guided image filter, weighted least square filter, non-uniform local filter, double exponential edge smoothing filter, selective blur, surface filter, etc.

[0092] Bilateral filter: In image processing, the bilateral filter is a nonlinear filter that smoothes the image. Unlike traditional linear smoothing filters, the bilateral filter not only considers the geometric proximity between pixels (i.e., Euclidean distance) when filtering, but also considers the brightness / color difference between pixels (i.e., grayscale difference), so that the bilateral filter can effectively remove noise from the image while preserving the edge information on the image.

[0093] ROI (Region Of Interest): In machine vision and image processing, the area to be processed is outlined in the form of a box, circle, ellipse, irregular polygon, etc. from the processed image, which is called the ROI area. In other words, the ROI area is an image area selected from the image. This area is the focus of image analysis, and the ROI area is circled for further processing. The ROI area involved in the embodiments of the present disclosure refers to the area to be erased that contains the object to be erased.

[0094] With the development of computer technology, users can perform secondary editing on images or video frames on the terminal to achieve the purpose of secondary creation and content generation. Usually, there is a processing requirement to erase an object in an image, such as erasing a digital watermark in an image, erasing obstacles that block the line of sight in an image, erasing a specified object in an image (such as a passerby who strays into the camera, or facilities that affect the beauty of the background), etc.

[0095] Taking watermark erasure as an example, watermarks refer to some specific patterns artificially superimposed in images. These patterns can be solid or translucent. When users use watermarked images for secondary creation, such as selecting multiple images to synthesize a video, the watermark will undoubtedly affect the editing effect of the video. When different watermarks are superimposed and displayed, it will also affect the aesthetics of the video. Therefore, the production side has a great demand for watermark removal algorithms. The goal of the watermark removal algorithm is to erase the watermark in the image and add new content in the erased area of ​​the watermark. However, the current watermark removal algorithm for watermarked images uses the STTN (Spatial-Temporal Transformations Networks) model based on the Transformer framework. Due to the poor controllability of the neural network, even after the STTN model is trained, due to the gradient diffusion problem, the image processed by the STTN model will have many afterimages in the erased area of ​​the watermark (such as Figure 1 and Figure 2 The STTN model has a poor effect on erasing specified objects in the image, which also leads to a poor secondary creation experience for users. In addition, due to the complex parameters of the STTN model, it can only be deployed on the server to provide good computing resources, and it is difficult to promote and deploy it on the client.

[0096] Figure 1 This is an effect diagram of erasing watermark based on the STTN model provided by the embodiment of the present disclosure, such as Figure 1 As shown, it can be seen that there are a lot of residual images 110 in the erased area of ​​the watermark. In addition, Figure 2 This is an effect diagram of erasing watermark based on the STTN model provided by the embodiment of the present disclosure, such as Figure 2 As shown, it can be seen that there are a lot of residual images 210 in the erased area of ​​the watermark.

[0097] In view of this, the embodiments of the present disclosure provide a post-processing algorithm for erasing designated objects based on edge-preserving filtering, which can not only quickly erase watermarks, but also greatly eliminate the afterimages left after the watermarks are erased. Moreover, in addition to watermarks, the algorithm can be extended to any designated object to be erased, such as obstacles that block the line of sight, passers-by who mistakenly enter the camera, facilities in the background that affect the appearance, etc. After successfully erasing the designated object, the erased area can be filled with correct and natural content. The algorithm is easy to deploy on the client, has a fast erasing speed, and can achieve erasing at a speed close to real-time.

[0098] The following describes the system architecture of the embodiment of the present disclosure.

[0099] Figure 3 is a schematic diagram of an implementation environment of an image processing method according to an exemplary embodiment. Figure 3 , the implementation environment includes at least one terminal 301 and a server 302.

[0100] Terminal 301 is used to provide an erasing service for specified objects in an image. An application that supports image processing is installed and running on terminal 301. Optionally, the application includes at least one of: a short video application, a live broadcast application, a photo editing application, a camera application, an audio and video application, an instant messaging application, a content sharing application, or a social application.

[0101] Illustratively, the above-mentioned application is embedded with program code for processing images, so that when the user inputs the original image and specifies the object to be erased, the terminal 301 can run the above-mentioned program code to erase the object to be erased in the original image and fill the erased area with correct and natural content, and finally obtain the processed target image, ensuring that the target image no longer contains the object to be erased and no afterimage of the erasure is left, so as to achieve a better erasing effect.

[0102] The terminal 301 and the server 302 are connected via a wired network or a wireless network.

[0103] The server 302 is an electronic device for providing background services for the above-mentioned application, and the server 302 includes: at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, the server 302 undertakes the main image processing work, and the terminal 301 undertakes the secondary image processing work; or, the server 302 undertakes the secondary image processing work, and the terminal 301 undertakes the main image processing work; or, the terminal 301 and the server 302 adopt a distributed computing architecture to collaboratively perform image processing work.

[0104] In some embodiments, the terminal 301 independently executes the image processing method, which can reduce the computing load of the server 302 and avoid occupying the processing resources of the server 302 during the image processing process. At this time, the terminal 301 will call the program code embedded in the local application to perform the image processing task.

[0105] In some embodiments, the terminal 301 cooperates with the server 302 to execute the image processing method through information interaction, that is, after acquiring the original image, the terminal 301 responds to the user's trigger operation of the erase function option, inputs a region to be erased or specifies an object to be erased, and sends an image erasing instruction carrying the region to be erased or the object to be erased to the server 302. The server 302 responds to the image erasing instruction, erases the object to be erased from the region to be erased, and fills in correct and natural content, or first identifies the region to be erased containing the object to be erased, then performs erasing and filling operations, and finally outputs a target image, which is returned to the terminal 301. At this time, part of the image processing work can be migrated to the server 302 to maintain higher system performance on the terminal.

[0106] Optionally, terminal 301 refers to one of a plurality of terminals, and the device type of terminal 301 includes, but is not limited to, at least one of: a vehicle-mounted terminal, a television, a smart phone, a smart speaker, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Group Audio Layer 3) player, an MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Group Audio Layer 4) player, a laptop computer, or a desktop computer. In the following embodiments, the terminal includes a smart phone for example.

[0107] Those skilled in the art will appreciate that the number of the above terminals 301 can be more or less. For example, the above terminal 301 is only one, or the above terminals 301 are dozens or hundreds, or more. The embodiment of the present disclosure does not limit the number and device type of the terminals 301.

[0108] Figure 4 is a flowchart of an image processing method according to an exemplary embodiment. Figure 4 The image processing method is applied to electronic devices, and is described below by taking the electronic device as a terminal as an example.

[0109] In step 401, the terminal obtains a to-be-erased area of ​​the original image, where the to-be-erased area includes an object to be erased.

[0110] In step 402, the terminal generates an erased area that does not include the object to be erased based on the area to be erased.

[0111] In step 403, the terminal filters the erased area to obtain a target area after high-frequency information is filtered out.

[0112] In step 404, the terminal replaces the to-be-erased area in the original image with the target area to obtain a target image.

[0113] The method provided by the embodiment of the present disclosure obtains an erased area by erasing the object to be erased in the erased area, and obtains a target area by filtering on the basis of the erased area, and then pastes the target area back to the original image to obtain a target image, so that the afterimage and residual noise left when erasing the object to be erased can be filtered out in the filtering stage, thereby greatly improving the erasing effect of the object to be erased.

[0114] In one possible implementation, filtering the erased area to obtain a target area after filtering out high-frequency information includes: inputting the erased area into an edge-preserving filter, filtering out the high-frequency information in the erased area while retaining edge information in the erased area through the edge-preserving filter, and outputting the target area.

[0115] In a possible implementation, the edge-preserving filter is a bilateral filter; the edge-preserving filter is used to filter out high-frequency information in the erased area while retaining edge information in the erased area, and outputting the target area includes: for any pixel point in the erased area, taking the pixel point as the center, sampling in the erased area to obtain a plurality of neighborhood pixel points around the pixel point; determining a weighting coefficient for each neighborhood pixel point, wherein the weighting coefficient is determined based on the Euclidean distance and grayscale difference between the neighborhood pixel point and the pixel point; weighting the pixel value of each neighborhood pixel point based on the weighting coefficient of each neighborhood pixel point, and adding the weighted pixel values ​​to obtain the pixel value of the pixel point in the target area at the same position as the pixel point.

[0116] In one possible implementation, determining the weighting coefficient of each neighborhood pixel point includes: determining a distance weight component based on the Euclidean distance between the neighborhood pixel point and the pixel point; determining a color weight component based on the grayscale difference between the neighborhood pixel point and the pixel point; and multiplying the distance weight component and the color weight component to obtain the weighting coefficient of the neighborhood pixel point.

[0117] In one possible implementation, based on the area to be erased, generating an erased area that does not contain the object to be erased includes: adding a mask in the area to be erased, the mask being used to cover the object to be erased; based on the background content in the area to be erased except the mask, generating foreground content corresponding to the mask, the foreground content matching the background content; replacing the mask in the area to be erased with the foreground content to obtain the erased area.

[0118] In one possible implementation, obtaining the area to be erased of the original image includes: determining the area to be erased based on an area position parameter input by an account, wherein the area position parameter is used to indicate the position of the area to be erased; or, based on the object to be erased input by an account, detecting the area to be erased containing the object to be erased from the original image.

[0119] In a possible implementation, replacing the area to be erased in the original image with the target area to obtain the target image includes: assigning pixel values ​​of each pixel in the target area to each pixel at a corresponding position in the area to be erased to obtain the target image.

[0120] In a possible implementation manner, the object to be erased is an image watermark or a video watermark.

[0121] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0122] Figure 5 is an interactive flow chart of an image processing method according to an exemplary embodiment. Figure 5 As shown, the image processing method is executed by an electronic device, and is described by taking the electronic device as a terminal as an example. The embodiment includes the following steps.

[0123] In step 501, the terminal obtains a to-be-erased area of ​​an original image, where the to-be-erased area includes an object to be erased.

[0124] The terminal is any electronic device that supports image processing services, and an application for processing images is installed on the terminal. Optionally, the application includes at least one of a short video application, a live broadcast application, a photo editing application, a photo taking application, an audio and video application, an instant messaging application, a content sharing application, or a social application. The embodiments of the present disclosure do not specifically limit the type of application.

[0125] The original image is an image input by the user, or a video frame (such as a key frame or a non-key frame) selected from a video input by the user. The above image or video frame is stored locally in the terminal or downloaded from the cloud. For example, the original image is an image captured by the terminal calling a camera component, or an image selected by the user from a local album, or an image downloaded by the user from the cloud. The embodiment of the present disclosure does not specifically limit the source of the original image.

[0126] The object to be erased refers to the object in the original image that the user needs to erase. For example, the types of objects include: image watermarks (the original image is a single-frame image), video watermarks (the original image is a video frame), animals (such as cats, dogs, bears, etc.), objects (such as obstacles, buildings, tables, chairs, vehicles, etc.), cartoon characters, virtual characters, people, etc. The embodiment of the present disclosure takes the object to be erased as an image watermark as an example for explanation, but it should not constitute a limitation on the object type to be erased.

[0127] The area to be erased refers to an area in the original image that contains the object to be erased. For example, the area to be erased is a rectangular area, or a circular area, an elliptical area, an irregular shape, etc. The embodiment of the present disclosure is described using the area to be erased as a rectangular area as an example, but it should not constitute a limitation on the shape of the area to be erased.

[0128] In some embodiments, after the user selects or inputs the original image on the terminal, the original image is displayed in the application, and a series of editing functions for the original image are provided, such as: changing the size, adding a border, blurring the background, erasing the object, adding mosaics, etc. In response to the user's triggering operation on the object erasing option, the user can choose to erase the specified area or the specified object. For example, the user selects the area to be erased in the original image, and the terminal automatically erases the object to be erased contained in the area to be erased. Alternatively, the user inputs an object to be erased, such as "cat", and the terminal automatically detects the area to be erased containing the object to be erased in the original image, such as a rectangular detection box containing "cat", and then the terminal automatically erases the object to be erased contained in the area to be erased. The embodiments of the present disclosure do not specifically limit this.

[0129] In some embodiments, after the user logs in to the account in the application, in response to the user's triggering operation of the object erasing option, when the user selects the erasing mode of the specified area, the user can enter the area location parameters in the original image. After detecting that the user performs a confirmation operation, the terminal obtains the area location parameters entered by the account, and the area location parameters are used to indicate the location of the area to be erased, so that the area to be erased can be determined based on the area location parameters.

[0130] In some embodiments, when the area to be erased is a rectangular area, the area position parameter includes a pair of vertex coordinates of the area to be erased located on the diagonal line, such as the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) of the rectangular area, or the upper right corner coordinates (x3, y3) and the lower left corner coordinates (x4, y4) of the rectangular area. The position of the area to be erased in the original image can be determined by a pair of vertex coordinates. It should be noted that in addition to a pair of vertex coordinates located on the diagonal line, a center coordinate and a vertex coordinate can also determine the area to be erased, or a center coordinate and the length and width of the rectangular area can also determine the area to be erased. The disclosed embodiments do not specifically limit the area position parameters.

[0131] In some embodiments, when the area to be erased is a circular area, the area position parameter includes the center and radius of the area to be erased, for example, the center coordinates (x0, y0) and radius r (r>0) of the center area, and the position of the area to be erased in the original image can be determined by the center and radius. It should be noted that in addition to the center and radius, the center and diameter can also determine the area to be erased, and the embodiments of the present disclosure do not specifically limit the area position parameters.

[0132] In some embodiments, the user can use his fingers to zoom and slide on the original image displayed in the application to control the erase box to follow the zooming and sliding of the fingers, and perform corresponding scaling and displacement in the original image. For example, the user's two fingers slide in a direction away from each other to control the erase box to enlarge, and the user's two fingers slide in a direction close to each other to control the erase box to shrink, and the user's fingers slide in a specified direction to control the erase box to translate in the specified direction. Finally, after the user performs a confirmation operation, the erase box can be used as the position parameter of the area. The erase box is equivalent to the outer edge of the area to be erased, so that the area composed of each pixel within the erase box is determined as the area to be erased.

[0133] In some embodiments, after the user logs in to the account in the application, in response to the user's triggering operation of the object erasing option, when the user selects the erasing mode of the specified object, the user can enter the object to be erased in the application. After detecting that the user performs a confirmation operation, the terminal obtains the object to be erased input by the account, and then detects the area to be erased containing the object to be erased from the original image.

[0134] Illustratively, the user inputs the object to be erased as "cat" in the application. After detecting that the user performs a confirmation operation, the terminal obtains the object to be erased "cat" input by the account, and executes a target detection algorithm on the original image to determine the area to be erased containing the object to be erased "cat". For example, the rectangular detection frame of each target in the original image is determined by the target detection algorithm, and then the target in each rectangular detection frame is subjected to a classification algorithm, such as using a binary classification model to determine whether the category to which the target belongs is "cat", or using a multi-classification model to determine the category to which the target belongs, and finally finds a rectangular detection frame with the category of "cat" as the area to be erased.

[0135] In step 502, the terminal adds a mask in the area to be erased, and the mask is used to cover the object to be erased.

[0136] In some embodiments, the terminal determines the minimum bounding rectangle containing the object to be erased in the area to be erased, and adds a mask to the sub-area corresponding to the minimum bounding rectangle, thereby ensuring that the mask can cover the object to be erased.

[0137] In some embodiments, the terminal inputs the area to be erased into an object erasing model, and adds a mask through the object erasing model. The object erasing model is used to erase the object to be erased in the input content. The object erasing model can be a traditional erasing algorithm or an erasing model based on deep learning. For example, traditional erasing algorithms include: PatchMatch (patch matching), Simultaneous Structure and Texture Image Inpainting (image restoration based on texture and structure), and erasing models based on deep learning include: DeepFill (depth filling) V1, DeepFill V2 and other models. The disclosed embodiments do not specifically limit the object erasing model.

[0138] In some embodiments, the object erasure model is embedded in the SDK (Software Development Kit) of the application after training with sample images on the server side. After the application obtains the area to be erased, it inputs the area to be erased into the object erasure model stored in the SDK to add a mask. Optionally, if the trained object erasure model has many model parameters, in order to save the storage overhead of the terminal, the object erasure model can be pruned and compressed, and the pruned and compressed object erasure model can be embedded in the SDK.

[0139] In step 503, the terminal generates foreground content corresponding to the mask based on the background content in the area to be erased except the mask, and the foreground content matches the background content.

[0140] In some embodiments, the area to be erased is divided into a mask and background content, and the background content refers to the part of the area to be erased except the mask. After adding the mask, it is equivalent to completing the task of erasing the object to be erased. Now it is necessary to predict a corresponding pixel value for each pixel in the mask, so that the pixel value predicted for each pixel in the mask can match the background content, which is equivalent to filling the missing content in the mask (that is, covered by the mask), so that although the filled content erases the object to be erased, it can still be naturally connected with the background content. Matching here means: the texture is continuous and smooth and the residual image is eliminated as much as possible.

[0141] In some embodiments, the terminal uses an object erasure model to predict the pixel value of each pixel in the mask based on the pixel value of each pixel in the background content. After the pixel value of each pixel in the mask is predicted, the foreground content corresponding to the shape and size of the mask can be obtained. Since the foreground content is predicted based on the background content, the foreground content and the background content can also be kept matching.

[0142] Schematically, taking the object erasing model as the PatchMatch algorithm as an example, the area to be erased including the mask is divided into multiple tiles, so as to use the pixel values ​​of each pixel in the tile of the background content to predict the pixel values ​​of each pixel in the tile of the foreground content. For example, the tile size is a 3×3 or 5×5 square area, and the tile size is not specifically limited in the embodiment of the present disclosure. Then, for any tile in the mask, find the background tile with the highest matching degree with the tile from the background content, and assign the pixel values ​​of each pixel of the background tile to the pixel values ​​of each pixel of the tile in the mask, for example, assign the pixel value of the pixel in the upper left corner of the background tile to the pixel value of the pixel in the upper left corner of the tile in the mask. Optionally, when searching for the background tile with the highest match to the current tile in the mask, a random search or a nearest neighbor search can be used. The meaning of the nearest neighbor search is that if the adjacent tile B of the current tile A has found the background tile X with the highest match (that is, the background tile X is assigned to the adjacent tile B), then there is a high probability that the background tile Y with the highest match to the current tile A is also adjacent to the background tile X. Therefore, when searching, prioritizing the adjacent tiles of the background tile X can usually find the background tile Y faster, and then assign the background tile Y to the current tile A.

[0143] Schematically, taking the DeepFill V2 model as an example of an object erasing model, the DeepFill V2 model includes a coarse repair network and a fine repair network. The coarse repair network is used to roughly predict the foreground filling result of the mask, that is, the foreground filling result output by the coarse repair network is a fuzzy preliminary result, and the fine repair network is used to finely predict the foreground content of the mask, that is, the foreground content output by the fine repair network is a fine repair image filled with details. Among them, both the coarse repair network and the fine repair network are obtained through a discriminant network, namely a discriminator, using the GAN (Generative Adversarial Networks, Generative Adversarial Network) architecture adversarial learning.

[0144] First, the area to be erased with the mask added is input into the coarse repair network, or the area to be erased and the mask binary Figure 1 The mask binary map is input into the coarse restoration network, where the mask binary map is a Mask map. The pixels in the Mask map can only take the value of 0 or 1. When the value is 1, it represents that the pixel belongs to the background content. When the value is 0, it represents that the pixel belongs to the foreground content to be predicted. In other words, the area composed of all pixels with a value of 0 in the Mask map can indicate the mask position in the area to be erased. By multiplying the area to be erased and the Mask map element by element, the area to be erased with the mask added can be obtained, that is, by multiplying the elements, the pixel values ​​of the pixels in the same position in the area to be erased and the Mask map can be multiplied, so that the pixel values ​​of each pixel in the background content can be kept unchanged (these pixel values ​​remain unchanged after being multiplied by 1), and the pixel values ​​of each pixel in the mask are returned to 0 (these pixel values ​​become 0 after being multiplied by 0).

[0145] Next, the area to be erased with the mask added is downsampled once or multiple times to obtain a downsampled image, which is then input into one or more gated convolution layers in series. The downsampled image is gated convolved through the gated convolution layers, and the last gated convolution layer outputs the extracted feature map. The feature map is upsampled once or multiple times and restored to the same size as the input image (i.e., the area to be erased) to obtain a coarse restoration image. The above coarse restoration network forms an encoding-then-decoding architecture because it needs to downsample first and then upsample.

[0146] Among them, the gated convolution layer involves two types of convolution kernels: threshold convolution kernel (Soft Mask) and void convolution kernel. Each weight in the threshold convolution kernel is equivalent to the filter coefficient of the input feature map. Each weight in the threshold convolution kernel is a value between 0 and 1. The role of the threshold convolution kernel is to provide a selection mechanism for each pixel in the input feature map like a soft sieve, while the void convolution kernel is used to expand the receptive field of the convolution process, so that when predicting the pixel value of each pixel in the foreground content, it can be guaranteed to be based on a sufficiently large receptive field to cover the pixel value of each pixel in the surrounding background content. For any gated convolution layer, the input feature map (the output feature map of the previous gated convolution layer, or the downsampled image if there is no previous gated convolution layer) can be convolved with a hole convolution kernel to obtain a first feature map, and the first feature map is activated by an activation function to obtain an activated first feature map. In addition, the input feature map is convolved with the gated convolution kernel to obtain a second feature map, and the second feature map is normalized by a Sigmoid function so that each eigenvalue in the normalized second feature map is between 0 and 1. The activated first feature map and the normalized second feature map are element-wise multiplied to obtain a third feature map. Optionally, the third feature map is directly input into the next gated convolution layer, or the third feature map is convolved and fused with the gated convolution kernel and the hole convolution kernel once more, and then the obtained feature map after two gates is input into the next gated convolution layer.

[0147] Next, the coarse inpainted image output by the coarse inpainting network is input into the fine inpainting network. The fine inpainting network includes two encoding branches, one is an encoding branch containing contextual attention, and the other is a traditional encoding branch. After the encoding images obtained by the two encoding branches are concatenated (Concat), they are input into one or more decoding layers for decoding, and finally the fine inpainted image is obtained, which is the foreground content corresponding to the mask mentioned above.

[0148] In the encoding branch containing semantic attention, the coarse repair image is downsampled once or multiple times to obtain the target downsampled image, which is input into the semantic attention layer to extract the semantic attention map, which is then input into the exponential normalization (Softmax) layer for normalization to obtain a normalized map, which is input into a transposed convolution layer for transposed convolution to obtain a reconstructed feature map that reconstructs the foreground content. Since it is difficult for a traditional convolution layer to connect two regions that are far apart in spatial position, a semantic attention layer is constructed. The role of the semantic attention layer is to borrow similar feature information from known regions without spatial restrictions to generate information missing in the mask (i.e., predict the foreground content in the mask).

[0149] In the semantic attention layer, the background content of the target downsampled image is divided into multiple 3×3 patches, which are used as convolution kernels to perform convolution operations on the foreground content (i.e., the area covered by the mask, which has been filled with roughly predicted pixel values ​​in the coarse restoration network) to obtain the semantic attention map. The semantic attention map is input into the subsequent exponential normalization layer for normalization of the channel dimension to obtain a normalized map, which is finally input into the transposed convolution layer. The transposed convolution layer still uses the multiple 3×3 patches of the above background content as convolution kernels to perform transposed convolution on the input normalized map, thereby realizing pixel reconstruction of the foreground content and obtaining the reconstructed feature map. Among them, since the blocks of the background content are used as convolution kernels in the semantic attention layer to perform convolution operations with the foreground content, the meaning of the above convolution operation is equivalent to calculating the cosine similarity between each block in the background content and each block in the foreground content. The physical meaning of this semantic attention map is the similarity map (or cross-correlation map) between the foreground content and the background content. Each eigenvalue in the semantic attention map represents the similarity between a pixel point of the foreground content and a pixel point of the background content. With the semantic attention map to indicate the similarity between any pixel point of the foreground content and any pixel point of the background content, similar feature information can be borrowed from the known area (i.e., the background content) without spatial restrictions to fill in the mask for pixel reconstruction and image restoration.

[0150] In the traditional encoding branch, the coarse repaired image is also downsampled once or multiple times to obtain a target downsampled image, which is input into one or more traditional convolutional layers and one or more dilated convolutional layers. The traditional convolutional layer performs a convolution operation on the input feature map, and the dilated convolutional layer performs a dilated convolution operation on the input feature map. The last dilated convolutional layer outputs a target feature map.

[0151] In the decoding part, the reconstructed feature map obtained by the encoding branch containing semantic attention is spliced ​​with the target feature map obtained by the traditional encoding branch, and then input into one or more decoding layers for decoding, and finally the refined repaired image, that is, the foreground content corresponding to the mask, is obtained.

[0152] In step 504, the terminal replaces the mask in the area to be erased with the foreground content to obtain an erased area that does not include the object to be erased.

[0153] In some embodiments, only the foreground content synthesized in step 503 that matches the mask is used to replace the mask in the area to be erased, without changing the background content in the area to be erased except for the mask. Optionally, the pixel value of each pixel in the foreground content is assigned to the pixel at the corresponding position in the mask of the area to be erased, so that the pixel value of the pixel changes from 0 to the pixel value predicted in step 503, and the above operation is repeated for each pixel until all the pixels in the mask have been assigned, and finally a picture of the erased area with the object to be erased erased and the foreground content filled can be obtained.

[0154] In the above steps 502-504, a possible implementation method is provided for generating an erased area that does not contain the object to be erased based on the area to be erased, that is, first adding a mask, then predicting the foreground content matched by the mask, and then replacing the mask with the predicted foreground content. Optionally, in the process of generating the erased area, there is no need to add a mask to the area to be erased, and it is only necessary to synthesize the Mask map involved in the above step 502, and reconstruct a finely repaired image based on the area to be erased and the Mask map using the object erasure model. The finely repaired image can be directly used as the above-mentioned erased area, or based on the indication of the Mask map, the pixel values ​​of each pixel point in the finely repaired image located in the corresponding area of ​​the mask are assigned to the pixel points at the corresponding position in the area to be erased to obtain the above-mentioned erased area. The embodiment of the present disclosure does not specifically limit the method for obtaining the erased area.

[0155] In the embodiment of the present disclosure, since the foreground content is predicted only based on the area to be erased instead of predicting the foreground content for the entire original image, unnecessary computational overhead for areas other than the area to be erased in the original image is avoided, which not only saves the processing resources of the terminal but also improves the image processing speed.

[0156] In step 505, the terminal inputs the erased area into an edge-preserving filter, and the edge-preserving filter filters out high-frequency information in the erased area while retaining edge information in the erased area, and outputs a target area.

[0157] In some embodiments, by sending the erased area to an edge-preserving filter, it is hoped that the residual noise (usually high-frequency information) in the erased area can be removed by the filter, and the background information in the erased area (i.e., low-frequency information such as edges and textures) can be retained as much as possible. For the case where the object to be erased is an image watermark or a video watermark, adding an edge-preserving filter can greatly eliminate the afterimage after erasing the watermark.

[0158] In the above process, the residual noise in the erasing area is filtered out by an edge-preserving filter, which can avoid the situation where linear smoothing filters such as Gaussian filters blur all textures in the erasing area while removing the residual noise, causing the background details in the erasing area to become blurred, thereby improving the erasing effect of object erasing.

[0159] In some embodiments, the edge-preserving filter includes but is not limited to: bilateral filter, guided filter, weighted least squares filter, non-uniform local filter, double exponential edge smoothing filter, selective blur, surface filter, etc.

[0160] The following takes the edge-preserving filter as a bilateral filter as an example to illustrate the filtering process: for any pixel point in the erasing area, with the pixel point as the center, multiple neighborhood pixel points around the pixel point are sampled in the erasing area. Optionally, with the pixel point as the center, multiple neighborhood pixel points in the neighborhood with the same size of the filter kernel are determined in the erasing area. For example, when the filter kernel size is 3×3, each neighborhood pixel point contained in the neighborhood with a size of 3×3 with the pixel point as the center is determined; then, for each neighborhood pixel point, based on the Euclidean distance and grayscale difference between the neighborhood pixel point and the pixel point, the neighborhood pixel point is determined. The weighting coefficient, in other words, the weighting coefficient in the filter kernel of the bilateral filter, is determined based on the distance factor and the chromaticity factor. On the one hand, the weighting coefficient of the neighborhood pixel point that is closer in spatial distance, i.e., the Euclidean distance, is larger; on the other hand, the weighting coefficient of the neighborhood pixel point that is smaller in grayscale difference is larger; then, based on the weighting coefficient of each neighborhood pixel point, the pixel value of each neighborhood pixel point is weighted, and the weighted pixel values ​​are added together to obtain the pixel value of the pixel point in the target area that has the same position as the pixel point; in other words, the pixel value of each pixel point in the target area is obtained based on the weighted pixel value of each neighborhood pixel point in the erased area.

[0161] In some embodiments, the weighting coefficient of each neighborhood pixel point in the bilateral filter is obtained in the following manner: based on the Euclidean distance between the neighborhood pixel point and the pixel point, a distance weight component is determined; based on the grayscale difference between the neighborhood pixel point and the pixel point, a color weight component is determined; and the distance weight component and the color weight component are multiplied to obtain the weighting coefficient of the neighborhood pixel point. In other words, the Euclidean distance and grayscale difference between the pixel point located at the center of the neighborhood and the neighborhood pixel point are obtained respectively, and the distance weight component and the color weight component are determined respectively, and the two are multiplied to obtain the final weighting coefficient. By fully considering the spatial distance and grayscale difference when obtaining the weighting coefficient, when filtering is performed based on the weighting coefficient in the subsequent process, the edge texture, that is, the background details are more retained, and the residual noise can be filtered out.

[0162] Schematically, when the erased area is input to the bilateralFilter() function, the bilateralFilter() function is used to provide bilateral filtering function. The bilateralFilter() function involves the following parameters: A) InputArraysrc, that is, the input image, which refers to the erased area here; B) OutputArray dst, that is, the output image, which refers to the target area here; C) d, which refers to the neighborhood diameter d (i.e., the filter kernel size) of each pixel point in the filtering process; D) sigmaColor, which refers to the sigma value of the color space filter. The larger the value of the sigmaColor parameter, the wider the colors in the neighborhood will be mixed together; E) sigmaSpace, which refers to the sigma value of the coordinate space filter. The larger the value of the sigmaSpace parameter, the farther the pixels will affect each other, so that the colors of a larger area that are similar enough will obtain the same color. For example, d = 13, sigmaColor = sigmaSpace = 75 are taken, but d, sigmaColor, and sigmaSpace can also take other values, which will not be repeated here.

[0163] In the above step 505, a possible implementation method is provided for filtering the erased area to obtain the target area after filtering out the high-frequency information, that is, filtering out the high-frequency information while retaining the low-frequency information through an edge-preserving filter. The above filtering operation can be implemented by using a bilateral filter, a guided filter, a weighted least squares filter, a non-uniform local filter, a double exponential edge smoothing filter, selective blurring, a surface filter, etc., and the embodiments of the present disclosure do not specifically limit this.

[0164] In step 506, the terminal replaces the to-be-erased area in the original image with the target area to obtain a target image.

[0165] In some embodiments, when the area to be erased is replaced by the target area, the pixel value of each pixel in the target area is directly assigned to each pixel at a corresponding position in the area to be erased to obtain the target image.

[0166] In some embodiments, when replacing the area to be erased with the target area, the pixel value of each pixel in the area to be erased is first assigned to 0, and then the pixel value of each pixel in the target area is assigned to each pixel at the corresponding position in the area to be erased to obtain the target image. The embodiments of the present disclosure do not specifically limit this.

[0167] In some embodiments, the target area is pasted back to the area to be erased in the original image to cover it directly based on the area position parameters to obtain a target image. For example, taking the area to be erased as a rectangular area, assuming that the area position parameters are the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) of the rectangular area, then according to the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2), the target area can be pasted back to the original position of the area to be erased, so that the target area covers the area to be erased, thereby obtaining a target image that does not contain the object to be erased.

[0168] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0169] The method provided by the embodiment of the present disclosure obtains an erased area by erasing the object to be erased in the erased area, and obtains a target area by filtering on the basis of the erased area, and then pastes the target area back to the original image to obtain a target image, so that the afterimage and residual noise left when erasing the object to be erased can be filtered out in the filtering stage, thereby greatly improving the erasing effect of the object to be erased.

[0170] Furthermore, since an edge-preserving filter is used to eliminate residual noise, the original background details, such as texture and edge details, can be retained to the greatest extent, and no additional unnatural textures (such as afterimages) will be introduced, while maintaining a relatively fast processing speed.

[0171] Furthermore, since only the object to be erased and the foreground content are repaired in the area to be erased, there is no need to generate unnecessary computing overhead for parts other than the area to be erased, which saves computing resources and processing resources, improves processing speed, and makes it easy to deploy on the client side.

[0172] Figure 6 is a principle flow chart of a watermark erasing method provided by an embodiment of the present disclosure, such as Figure 6 As shown, taking the object to be erased as an image watermark as an example, a watermark erasure post-processing algorithm based on edge-preserving filtering is introduced. The watermark erasure method is executed by an electronic device, which can be a terminal or a server. Taking the electronic device as a terminal as an example, the watermark erasure method includes the following steps:

[0173] In step 601, an original image is acquired.

[0174] In step 602, the user selects a frame to be erased to deduct the ROI area.

[0175] In other words, based on the erasing frame selected by the user, the area to be erased containing the watermark is deducted from the original image, and this area to be erased is the ROI area of ​​this erasure.

[0176] For example, after the user selects the erasing frame, the area to be erased is deducted from the original image according to the upper left corner coordinates (x1, y1) and the lower right corner coordinates (x2, y2) of the erasing frame.

[0177] Figure 7 is a schematic diagram of deducting a ROI area provided by an embodiment of the present disclosure, such as Figure 7 As shown, the area to be erased, i.e., the ROI area 711, is deducted from the original image 701, and the area to be erased, i.e., the ROI area 712, is deducted from the original image 702. The subsequent watermark removal algorithm and edge-preserving filter will only process the ROI areas 711 and 712 without consuming additional computing overhead in the non-ROI area.

[0178] In step 603, a watermark removal algorithm is executed on the ROI area.

[0179] In other words, the area to be erased is input into a watermark removal algorithm to remove the watermark contained in the area to be erased. The watermark removal algorithm is an exemplary description of an object erasure algorithm or model.

[0180] In step 604, the watermark removal result output by the watermark removal algorithm is input into an edge-preserving filter for filtering.

[0181] In other words, the watermark removal result refers to the erased area with the watermark removed and the foreground content filled in, and the erased area is sent to the edge-preserving filter to remove the residual high-frequency noise again.

[0182] In step 605, the watermark removal result of the ROI area is obtained.

[0183] In other words, the watermark removal result after edge-preserving filtering is the final watermark removal result of the ROI region, that is, the target region after high-frequency information is filtered out in the above embodiment.

[0184] In step 606, the dewatering result of the ROI area is pasted back to the original image.

[0185] In other words, the dewatering result of the ROI area is pasted back to the original image.

[0186] In step 607, the final result, ie, the target image, is obtained.

[0187] The target image refers to the original image with the watermark removed and the residual image eliminated.

[0188] In the disclosed embodiment, a watermark removal post-processing algorithm based on edge-preserving filtering is provided. Since the watermark removal result output by the watermark removal algorithm is filtered by the edge-preserving filter, the residual noise can be filtered out while retaining the high-frequency information, so that the residual image in the watermark removal result is eliminated, which greatly improves the watermark erasure effect. In addition, since the watermark removal algorithm and the edge-preserving filter only need to focus on the ROI area, without consuming additional computing overhead in the non-ROI area, it can not only reduce the computing overhead, but also be easy to deploy on the client, and can bring better processing speed. For example, for the original image with a size of 40×110, it only takes 12 milliseconds (ms) to process the image, which is close to real-time speed.

[0189] Next, still taking the image watermark as an example, the traditional watermark removal method based on the STTN model and the watermark removal post-processing algorithm based on edge-preserving filtering in the embodiment of the present disclosure are compared to see the watermark removal effects of the two methods.

[0190] Figure 8 is a comparison diagram of a watermark removal effect provided by an embodiment of the present disclosure, such as Figure 8 As shown, 801 shows the watermark removal result of the traditional watermark removal method based on the STTN model. It can be seen that there are many afterimages in the area 811 containing the watermark, and the watermark removal effect is poor; 802 shows the watermark removal result based on the method of the embodiment of the present disclosure. It can be seen that the afterimage has been eliminated in the area 812 originally containing the watermark, achieving a very natural watermark removal effect.

[0191] Fig. 9 is a comparison diagram of a watermark removal effect provided by an embodiment of the present disclosure, such as Fig. 9 As shown, 901 shows the watermark removal result of the traditional watermark removal method based on the STTN model. It can be seen that there are many afterimages in the area 911 containing the watermark, and the watermark removal effect is poor; 902 shows the watermark removal result based on the method of the embodiment of the present disclosure. It can be seen that the afterimage has been eliminated in the area 912 originally containing the watermark, achieving a very natural watermark removal effect.

[0192] Comprehensive reference to the above Figure 8 and Fig. 9 It can be seen that in some scenes with relatively simple backgrounds, although the traditional method can remove the watermark contained in the upper left corner, obvious afterimages appear, and the watermark removal effect is very poor. The method of the embodiment of the present disclosure can remove the afterimage while removing the watermark, and will not cause the additional problem of background blur.

[0193] Next, the traditional method and the method of the embodiment of the present disclosure will be used again for some scenes with complex backgrounds to test and illustrate them respectively.

[0194] Fig.10is a comparison diagram of a watermark removal effect provided by an embodiment of the present disclosure, such as Fig.10 As shown, 1001 shows the watermark removal result of the traditional watermark removal method based on the STTN model. It can be seen that there are many afterimages in the area 1011 containing the watermark, especially the letter "P" in the upper left corner of the background. A very unnatural afterimage can be seen, and the watermark removal effect is very poor; 1002 shows the watermark removal result based on the method of the embodiment of the present disclosure. It can be seen that the afterimage has been eliminated in the area 1012 originally containing the watermark, especially the letter "P" in the upper left corner of the background. Not only has the afterimage been eliminated, but the texture and edge of the letter "P" are also well preserved without being blurred or destroyed, achieving a very natural watermark removal effect.

[0195] Fig.11 is a comparison diagram of a watermark removal effect provided by an embodiment of the present disclosure, such as Fig.11 As shown, 1101 shows the watermark removal result of the traditional watermark removal method based on the STTN model. It can be seen that there are many afterimages in the area 1111 containing the watermark, especially there is a part of unnatural afterimage in the upper right corner of the "XTRA" box in the background, it looks like the background is not smooth enough, and the watermark removal effect is very poor; 1102 shows the watermark removal result based on the method of the embodiment of the present disclosure. It can be seen that the afterimage has been eliminated in the area 1112 originally containing the watermark, especially the upper right corner of the "XTRA" box in the background, and the texture and edge of the background are well preserved without being blurred or destroyed, achieving a very natural watermark removal effect.

[0196] Fig.12 FIG. 1 is a logical structure block diagram of an image processing device according to an exemplary embodiment. Fig.12 , the device includes an acquisition unit 1201, a generation unit 1202, a filtering unit 1203 and a replacement unit 1204, which are described below:

[0197] An acquisition unit 1201 is configured to acquire a to-be-erased area of ​​an original image, where the to-be-erased area includes an object to be erased;

[0198] A generating unit 1202 is configured to generate an erasure area that does not include the object to be erased based on the area to be erased;

[0199] The filtering unit 1203 is configured to perform filtering on the erased area to obtain a target area after filtering out high-frequency information;

[0200] The replacement unit 1204 is configured to replace the to-be-erased area in the original image with the target area to obtain a target image.

[0201] The device provided by the embodiment of the present disclosure obtains an erased area by erasing the object to be erased in the erased area, obtains a target area by filtering on the basis of the erased area, and then obtains a target image by pasting the target area back to the original image, so that the afterimage and residual noise left when erasing the object to be erased can be filtered out in the filtering stage, thereby greatly improving the erasing effect of the object to be erased.

[0202] In one possible implementation, based on Fig.12 The filtering unit 1203 includes: an input subunit, configured to input the erased area into an edge-preserving filter; a filtering subunit, configured to filter out high-frequency information in the erased area while retaining edge information in the erased area through the edge-preserving filter; and an output subunit, configured to output the target area.

[0203] In one possible implementation, the edge-preserving filter is a bilateral filter; based on Fig.12 The device is composed of: the filtering sub-unit includes: a sampling sub-sub-unit, which is configured to perform sampling on any pixel point in the erasing area, with the pixel point as the center, to obtain multiple neighborhood pixel points around the pixel point in the erasing area; a determining sub-sub-unit, which is configured to perform determination of a weighting coefficient of each neighborhood pixel point, and the weighting coefficient is determined based on the Euclidean distance and grayscale difference between the neighborhood pixel point and the pixel point; an adding sub-sub-unit, which is configured to perform weighting on the pixel value of each neighborhood pixel point based on the weighting coefficient of each neighborhood pixel point, and add the weighted pixel values ​​to obtain the pixel value of the pixel point in the target area with the same position as the pixel point.

[0204] In one possible implementation, the determination sub-subunit is configured to perform: determining a distance weight component based on the Euclidean distance between the neighborhood pixel point and the pixel point; determining a color weight component based on the grayscale difference between the neighborhood pixel point and the pixel point; and multiplying the distance weight component and the color weight component to obtain a weighting coefficient for the neighborhood pixel point.

[0205] In one possible implementation, the generation unit 1202 is configured to perform: adding a mask in the area to be erased, the mask being used to cover the object to be erased; generating foreground content corresponding to the mask based on the background content in the area to be erased except the mask, the foreground content matching the background content; replacing the mask in the area to be erased with the foreground content to obtain the erased area.

[0206] In one possible implementation, the acquisition unit 1201 is configured to perform: determining the area to be erased based on an area position parameter input by an account, wherein the area position parameter is used to indicate the position of the area to be erased; or, based on the object to be erased input by an account, detecting from the original image the area to be erased containing the object to be erased.

[0207] In a possible implementation, the replacement unit 1204 is configured to execute: assigning pixel values ​​of each pixel point in the target area to each pixel point at a corresponding position in the area to be erased, so as to obtain the target image.

[0208] In a possible implementation manner, the object to be erased is an image watermark or a video watermark.

[0209] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0210] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the image processing method, and will not be elaborated here.

[0211] Fig.13 The following is a block diagram of a terminal provided by an exemplary embodiment of the present disclosure, wherein the terminal 1300 is an exemplary description of an electronic device. The terminal 1300 may be a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. The terminal 1300 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0212] Typically, the terminal 1300 includes a processor 1301 and a memory 1302 .

[0213] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0214] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one instruction, which is used to be executed by the processor 1301 to implement the image processing method provided in various embodiments of the present disclosure.

[0215] In some embodiments, the terminal 1300 may also optionally include: a peripheral device interface 1303 and at least one peripheral device. The processor 1301, the memory 1302 and the peripheral device interface 1303 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1304, a touch display screen 1305, a camera assembly 1306, an audio circuit 1307, a positioning assembly 1308 and a power supply 1309.

[0216] The peripheral device interface 1303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0217] The radio frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1304 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1304 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0218] The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1305 is a touch display screen, the display screen 1305 also has the ability to collect touch signals on the surface or above the surface of the display screen 1305. The touch signal can be input to the processor 1301 as a control signal for processing. At this time, the display screen 1305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1305 can be one, and the front panel of the terminal 1300 is set; in other embodiments, the display screen 1305 can be at least two, which are respectively set on different surfaces of the terminal 1300 or are folded; in some other embodiments, the display screen 1305 can be a flexible display screen, which is set on the curved surface or folded surface of the terminal 1300. Even, the display screen 1305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0219] The camera assembly 1306 is used to capture images or videos. Optionally, the camera assembly 1306 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1306 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0220] The audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1301 for processing, or input them into the radio frequency circuit 1304 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1300. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1307 may also include a headphone jack.

[0221] The positioning component 1308 is used to locate the current geographical location of the terminal 1300 to implement navigation or LBS (Location Based Service).

[0222] The power supply 1309 is used to power various components in the terminal 1300. The power supply 1309 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1309 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0223] In some embodiments, the terminal 1300 further includes one or more sensors 1310 . The one or more sensors 1310 include, but are not limited to, an acceleration sensor 1311 , a gyroscope sensor 1312 , a pressure sensor 1313 , an optical sensor 1314 , and a proximity sensor 1315 .

[0224] The acceleration sensor 1311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1300. For example, the acceleration sensor 1311 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1301 can control the touch display screen 1305 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1311. The acceleration sensor 1311 can also be used for collecting game or user motion data.

[0225] The gyro sensor 1312 can detect the body direction and rotation angle of the terminal 1300, and the gyro sensor 1312 can cooperate with the acceleration sensor 1311 to collect the user's 3D actions on the terminal 1300. The processor 1301 can implement the following functions based on the data collected by the gyro sensor 1312: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0226] The pressure sensor 1313 can be set on the side frame of the terminal 1300 and / or the lower layer of the touch display screen 1305. When the pressure sensor 1313 is set on the side frame of the terminal 1300, it can detect the user's holding signal of the terminal 1300, and the processor 1301 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1313. When the pressure sensor 1313 is set on the lower layer of the touch display screen 1305, the processor 1301 controls the operability controls on the UI interface according to the user's pressure operation on the touch display screen 1305. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0227] The optical sensor 1314 is used to collect the ambient light intensity. In one embodiment, the processor 1301 can control the display brightness of the touch display screen 1305 according to the ambient light intensity collected by the optical sensor 1314. Specifically, when the ambient light intensity is high, the display brightness of the touch display screen 1305 is increased; when the ambient light intensity is low, the display brightness of the touch display screen 1305 is reduced. In another embodiment, the processor 1301 can also dynamically adjust the shooting parameters of the camera assembly 1306 according to the ambient light intensity collected by the optical sensor 1314.

[0228] The proximity sensor 1315, also called a distance sensor, is usually arranged on the front panel of the terminal 1300. The proximity sensor 1315 is used to collect the distance between the user and the front of the terminal 1300. In one embodiment, when the proximity sensor 1315 detects that the distance between the user and the front of the terminal 1300 is gradually decreasing, the processor 1301 controls the touch display screen 1305 to switch from the screen-on state to the screen-off state; when the proximity sensor 1315 detects that the distance between the user and the front of the terminal 1300 is gradually increasing, the processor 1301 controls the touch display screen 1305 to switch from the screen-off state to the screen-on state.

[0229] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the terminal 1300, and the terminal 1300 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0230] Fig.14 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. The electronic device 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1401 and one or more memories 1402, wherein the memory 1402 stores at least one program code, and the at least one program code is loaded and executed by the processor 1401 to implement the image processing methods provided by the above-mentioned embodiments. Of course, the electronic device 1400 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 1400 may also include other components for realizing device functions, which will not be repeated here.

[0231] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction, and the at least one instruction can be executed by a processor in an electronic device to complete the image processing method in the above embodiment. Optionally, the computer-readable storage medium can be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium can include ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device, etc.

[0232] In an exemplary embodiment, a computer program product is also provided, including one or more instructions, which can be executed by a processor of an electronic device to complete the image processing method provided by each of the above embodiments.

[0233] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0234] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An image processing method, characterized in that: include: Acquire a to-be-erased area of ​​the original image, wherein the to-be-erased area includes an object to be erased; Adding a mask in the area to be erased, wherein the mask is used to cover the object to be erased; generating foreground content corresponding to the mask based on background content in the area to be erased excluding the mask, wherein the foreground content matches the background content; replacing the mask in the area to be erased with the foreground content to obtain an erased area that does not contain the object to be erased; Filtering the erased area to obtain a target area after filtering out high-frequency information; The area to be erased in the original image is replaced with the target area to obtain a target image.

2. The method according to claim 1, characterized in that The filtering of the erased area to obtain a target area after filtering out high-frequency information includes: The erased area is input into an edge-preserving filter, and the edge-preserving filter filters out high-frequency information in the erased area while retaining edge information in the erased area, and outputs the target area.

3. The method according to claim 2, characterized in that The edge-preserving filter is a bilateral filter; The filtering out high-frequency information in the erased area while retaining edge information in the erased area through the edge-preserving filter, and outputting the target area includes: For any pixel point in the erasing area, taking the pixel point as the center, sampling in the erasing area to obtain a plurality of neighboring pixel points around the pixel point; Determine a weighting coefficient for each neighborhood pixel point, where the weighting coefficient is determined based on a Euclidean distance and a grayscale difference between the neighborhood pixel point and the pixel point; Based on the weighting coefficient of each neighborhood pixel point, the pixel value of each neighborhood pixel point is weighted, and the weighted pixel values ​​are added together to obtain the pixel value of the pixel point at the same position as the pixel point in the target area.

4. The method according to claim 3, characterized in that Determining the weighting coefficient of each neighborhood pixel includes: Determining a distance weight component based on the Euclidean distance between the neighborhood pixel point and the pixel point; Determining a color weight component based on a grayscale difference between the neighborhood pixel and the pixel; The distance weight component and the color weight component are multiplied to obtain a weighting coefficient of the neighborhood pixel point.

5. The method according to claim 1, wherein The obtaining of the area to be erased of the original image includes: Determine the area to be erased based on an area location parameter input by the account number, wherein the area location parameter is used to indicate the location of the area to be erased; or Based on the object to be erased inputted by the account number, the area to be erased containing the object to be erased is detected from the original image.

6. The method according to claim 1, wherein Replacing the to-be-erased area in the original image with the target area to obtain the target image includes: The pixel value of each pixel point in the target area is assigned to each pixel point at a corresponding position in the area to be erased to obtain the target image.

7. The method according to claim 1, characterized in that The object to be erased is an image watermark or a video watermark.

8. An image processing device, characterized in that: include: an acquisition unit configured to acquire a to-be-erased area of ​​the original image, wherein the to-be-erased area includes an object to be erased; A generating unit configured to add a mask in the area to be erased, wherein the mask is used to cover the object to be erased; generating foreground content corresponding to the mask based on background content in the area to be erased excluding the mask, wherein the foreground content matches the background content; replacing the mask in the area to be erased with the foreground content to obtain an erased area that does not contain the object to be erased; a filtering unit configured to filter the erased area to obtain a target area after high-frequency information is filtered out; The replacing unit is configured to replace the to-be-erased area in the original image with the target area to obtain a target image.

9. The device according to claim 8, characterized in that The filtering unit comprises: an input subunit, configured to input the erased area into an edge-preserving filter; a filtering subunit configured to filter out high-frequency information in the erased area while retaining edge information in the erased area through the edge-preserving filter; The output subunit is configured to output the target area.

10. The device according to claim 9, characterized in that The edge-preserving filter is a bilateral filter; The filtering subunit includes: The sampling sub-sub-unit is configured to perform sampling on any pixel point in the erasing area, with the pixel point as the center, to obtain a plurality of neighboring pixel points around the pixel point in the erasing area; a determination sub-subunit, configured to determine a weighting coefficient for each neighborhood pixel point, wherein the weighting coefficient is determined based on a Euclidean distance and a grayscale difference between the neighborhood pixel point and the pixel point; The adding sub-subunit is configured to perform weighting on the pixel value of each neighborhood pixel point based on the weighting coefficient of each neighborhood pixel point, and add the weighted pixel values ​​to obtain the pixel value of the pixel point with the same position as the pixel point in the target area.

11. The device according to claim 10, characterized in that The determining sub-subunit is configured to execute: Determining a distance weight component based on the Euclidean distance between the neighborhood pixel point and the pixel point; Determining a color weight component based on a grayscale difference between the neighborhood pixel and the pixel; The distance weight component and the color weight component are multiplied to obtain a weighting coefficient of the neighborhood pixel point.

12. The device according to claim 8, characterized in that The acquisition unit is configured to perform: Determine the area to be erased based on an area location parameter input by the account number, wherein the area location parameter is used to indicate the location of the area to be erased; or Based on the object to be erased inputted by the account number, the area to be erased containing the object to be erased is detected from the original image.

13. The device according to claim 8, characterized in that The replacement unit is configured to perform: The pixel value of each pixel point in the target area is assigned to each pixel point at a corresponding position in the area to be erased to obtain the target image.

14. The device according to claim 8, characterized in that The object to be erased is an image watermark or a video watermark.

15. An electronic device, characterized in that: include: one or more processors; one or more memories for storing the one or more processor-executable instructions; The one or more processors are configured to execute the instructions to implement the image processing method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that When at least one instruction in the computer-readable storage medium is executed by one or more processors of an electronic device, the electronic device is enabled to perform the image processing method according to any one of claims 1 to 7.

17. A computer program product, characterized in that The image processing method comprises one or more instructions, wherein the one or more instructions are executed by one or more processors of an electronic device, so that the electronic device can execute the image processing method according to any one of claims 1 to 7.

Images