Blending for Surrounding Area Detection and Image Filtering

Abstract

A method and system for detecting and blending peripheral areas to avoid filtering artifacts due to such areas in image filtering is disclosed. The disclosed method includes a compensation step and can be applied to any image with padded areas of any shape, such as letterbox, pillarbox, ellipse, or other shapes, including logos and closed captions. Such a method detects peripheral areas in an image that may have compression artifacts and noise, and then performs blending, thereby minimizing the influence of the peripheral areas on any image filtering process.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to U.S. Provisional Application No. 63 / 209,602, filed June 11, 2021, and European Patent Application No. 21178935.9, filed June 11, 2021, each of which is incorporated by reference in its entirety.

[0002] (Technical field) FIELD OF THE DISCLOSURE The present disclosure relates generally to image processing, and more particularly to peripheral area detection and blending for image filtering. [Background technology]

[0003] In general, an image may contain inactive areas around the main subject of the image. Such inactive areas may be of any shape (e.g., rectangular, circular, elliptical or other geometric shape) or any irregular shape. Furthermore, the inactive areas in an image may contain text, labels, or captions. In image filtering, it is highly desirable to detect and blend the peripheral areas to avoid filtering artifacts due to the peripheral areas. In existing filtering processes, the peripheral areas are assumed to be noise-free rectangular and are excluded during image filtering. This may work well for in-house studio content that is noise-free. However, for other video content with logos, text, or noise in the peripheral areas or non-rectangular peripheral areas, existing basic methods may cause banding / halo artifacts near the peripheral areas. Such filtering artifacts may be amplified in further processing, such as local reshaping or other filtering output-based processing, degrading the visual quality. Summary of the Invention

[0004] The term "surround area" as used herein refers to the inactive (static) region around an image or video frame (typically referred to as the "active area"). Examples of surround areas include the black bands known as letterboxes or pillarboxes to accommodate various video / film aspect ratios within a typical 16:9 television frame. Surround areas can have any shape, such as rectangles, circles, ellipses, or other irregular shapes. Surround areas are typically distinguished by a "drab" color such as black or gray, although text (e.g., subtitles) and graphics (e.g., logos) may often be overlaid on top of these areas.

[0005] The disclosed method and apparatus provide an efficient framework for detecting and blending peripheral areas to avoid filtering artifacts due to peripheral areas in image filtering.

[0006] Compared to existing methods that can only handle rectangular areas such as letterboxes and pillarboxes, the method of the present disclosure can be applied to any image that has padded dark, solid, color, or white areas of any shape, such as letterboxes, pillarboxes, ellipses, and other shapes. Furthermore, the method of the present disclosure can also be applied to peripheral areas that contain text, logos, and closed captions. As described in more detail below, the method of the present disclosure excludes such text, logos, and closed captions from the peripheral areas.

[0007] The described method minimizes the influence of the surrounding areas on the image filtering process by detecting surrounding areas in the image that are likely to have compression artifacts and noise, and then performing the blending.

[0008] One embodiment of the present invention is a method for detecting a peripheral area in an image, comprising: calculating a histogram of a boundary area of ​​the image; determining a peak and a width of the histogram; classifying the presence of a peripheral area in the image based on the peak and the width of the histogram to generate a peak detection score; classifying the presence of the peripheral area in the image based on the proportion of pixels that belong to a peak on the smallest possible peripheral area in the boundary area of ​​the image to generate a boundary detection score; and based on a combination of the peak and the boundary detection score: total generating a score; total and detecting a presence of the surrounding area based on the score.

[0009] In some embodiments, the method may be computer-implemented, for example, the method may be implemented at least in part via a control system including one or more processors and one or more non-transitory storage media.

[0010] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, and the like. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented in a non-transitory medium having software stored thereon. The software may be executable by one or more components of a control system, such as those described herein. For example, the software may include instructions for performing one or more of the methods described herein.

[0011] At least some aspects of the present disclosure may be implemented by an apparatus or apparatuses. For example, one or more devices may be configured to at least partially execute the methods disclosed herein. In some aspects, the apparatus may include an interface system and a control system. The interface system may include one or more network interfaces, one or more interfaces between the control system and a memory system, one or more interfaces between the control system and another device, and / or one or more external device interfaces. The control system may include at least one of one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components. Thus, in some aspects, the control system may include one or more processors and one or more non-transitory storage media operably coupled to the one or more processors.

[0012] Details of one or more aspects of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the following description, drawings, and claims. Please note that the relative dimensions of the following figures may not be drawn to scale. Like reference numbers and names in the various drawings generally indicate like elements, but different reference numbers do not necessarily designate different elements between different drawings. [Brief description of the drawings]

[0013] FIG. 1 illustrates an exemplary flow chart of surrounding area detection and blending according to one embodiment of the present disclosure.

[0014] FIG. 2 illustrates an exemplary flow chart of a surrounding area detection process according to one embodiment of the present disclosure.

[0015] FIG. 3 illustrates an example image with a boundary region used to calculate a surrounding area histogram according to one embodiment of the present disclosure.

[0016] FIG. 4 illustrates an exemplary histogram of a border area of ​​an image calculated according to one embodiment of the present disclosure.

[0017] FIG. 5 is an exemplary graph illustrating the linear variation of score versus observed noise in a peak classifier according to one embodiment of the present disclosure.

[0018] FIG. 6 is an exemplary graph illustrating score as a function of pixel proportion in a boundary classifier according to one embodiment of the present disclosure.

[0019] FIG. 7 illustrates an example classifier margin according to one embodiment of the present disclosure.

[0020] 8A-8B, 9A-9B, and 10A-10B illustrate exemplary performance of a surrounding area detection process according to one embodiment of the present disclosure.

[0021] FIG. 10C shows an example image that includes an irregularly shaped peripheral area.

[0022] FIG. 11 illustrates an exemplary flow chart of a surrounding area blending process according to one embodiment of the present disclosure.

[0023] FIG. 12 illustrates an exemplary flow chart of a weighted Gaussian filtering process applied according to one embodiment of the present disclosure.

[0024] FIG. 13 illustrates an exemplary flow chart of mirroring with distance transformation processing applied according to one embodiment of the present disclosure.

[0025] FIG. 14 illustrates an exemplary flow chart of a compensation / post-processing procedure according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] As shown in the exemplary embodiment of FIG. 1, the disclosed method includes two components or steps: surrounding area detection (10) and surrounding area blending (10'). Given an input image (11), the surrounding area histogram (12) characteristics are first analyzed. Then, a surrounding area classifier (13) with soft decision boundaries is used to detect the possible presence of a surrounding area in the input image (11). If a surrounding area is detected, the pixel values ​​of the surrounding area are modeled (14), for example using a Gaussian kernel, and a surrounding area probability map (15) is generated as an output.

[0027] In step (10'), the surrounding area is blended as shown in step (16) based on a combination of the input image (11) and the generated surrounding area probability map (15) to avoid filtering artifacts near the surrounding area. According to an embodiment of the present disclosure, the filtering process (17) may be any kind of image filtering (e.g., edge-preserving filtering as described in U.S. Provisional Application No. 63 / 086,699, filed by the applicant of the present disclosure on October 2, 2020 for "Adaptive Local Reshaping For SDR-To-HDR Up-Conversion" and incorporated herein in its entirety). After filtering (17), the previously blended surrounding area in the initial filtered image is compensated to match the original input image (11), as shown in step (18). Finally, the compensated filtered image is sent to further processing (19), such as local reconstruction or other processing based on the filtering output. In the following, an exemplary embodiment of the surrounding area detection (10) and blending (10') will be described in more detail.

[0028] A. Surrounding area detection FIG. 2 is an exemplary diagram of a surrounding area detection process according to one embodiment of the present disclosure. As shown in step (20), given an input image (21), first a histogram (22) of the boundary region of the image (21) is calculated, followed by calculating the peaks of the histogram (23) and the width of the calculated peaks. As part of the next step (20'), a peak classifier (24) is used to classify the image as having or not having a surrounding area based on the characteristics of the peaks. Furthermore, as an additional part of step (20'), a border classifier (25) is also used to determine the possibility of the presence of a surrounding area in the input image (21) based on the characteristics of the image border. Step (24) 、 25) were integrated and based on those results total A score (26) is determined. If a surrounding area is detected, the pixel values ​​of the surrounding area are modelled in a subsequent surrounding area modelling step (20"), for example using a Gaussian kernel (27), which produces a surrounding area probability map (28) as output. The elements of the diagram in Figure 2 are explained in more detail below.

[0029] A.1. Surrounding Area Histogram According to one embodiment of the present disclosure, two assumptions can be made about the surrounding area: 1) the surrounding area in the image has constant pixel values ​​including noise, and 2) the surrounding area is "stuck" to the border of the image. Based on these two assumptions, if the surrounding area is present, it is expected that there will be a peak in the histogram of the border region of the image.

[0030] In the following, we denote the input image and its Y channel as S and S, respectively. YFor purposes of illustration, FIG. 3 shows an example image (30) having a boundary region (31) that is used to calculate the surrounding area histogram. The width and height of the image are denoted as W and H, respectively. For efficiency, throughout this disclosure, in the example embodiment, only the Y channel is considered in the disclosed detection framework, and the Y channel of the input image is referred to as the input image throughout this disclosure, although in other embodiments, other channels (e.g., chroma) may also be used as input.

[0031] Continuing to refer to FIG. 3, the boundary region (31) is the top and bottom H L Between lines and at the left and rightmost W L Empirically, the width W L and height H L For example, W L =round(0.05×W) and H L = round(0.05 × H). The histogram is start Bin width starting from w bin N B Since the surrounding area may have darker pixel values, it is not necessary to calculate the histogram for the complete luminance range, only for the darker parts. As an example, for a 10-bit input image with pixel values ​​ranging from 0 to 1023, the various parameters are start =0, w bin =1 and N B You can also set it to s =128. start is the pixel value where the histogram starts, and N B is the number of bins in the histogram, and w bin is the width of each bin.

[0032] The following table shows an example of how the histogram of the border region of an image may be calculated in accordance with the teachings of this disclosure. [Table 1]

[0033] Once the histogram of the boundary region is calculated, the highest peak in the histogram and its width are calculated as the potential surrounding area pixel values. The peak is defined as the leftmost contiguous bin with the maximum value, and the width is the ratio r of the histogram value to the maximum value. peak It is defined by bins on either side of a peak that falls to r peak is r peak =exp(-0.5)≈0.6065. This ensures that half the width of the peak is close to the standard deviation if the peak is close to a Gaussian distribution. Figure 4 shows an example histogram and peak. ML and b MR are the leftmost and rightmost bins that define the successive maxima of the peak. SL and b SR are the leftmost and rightmost bins that define the width of the peak. Optionally, if the image noise is too sparse, some smoothing may be performed on the histogram (e.g. convolution with a Gaussian kernel). The following table shows an example of how peaks in a histogram are calculated: [Table 2]

[0034] A.2. Surrounding Area Classifier In the following, the term "classifier" refers to a module that takes the histogram features and predicts a score indicating whether a surrounding area is present or not: a score greater than zero means that a surrounding area is present, and a score less than or equal to zero means that a surrounding area is not present.

[0035] Given the discussion in the previous section, and assuming that potential peaks in the histogram have been found, the next step may be to indicate whether such peaks are due to surrounding areas. To accomplish this step, two classifiers may be defined according to the teachings of the present disclosure: 1) a peak classifier; and 2) a boundary classifier. As will be described below, the output from these classifiers may be combined to improve accuracy.

[0036] A.2.1. Peak Classifier In one embodiment, the peak classifier predicts the surrounding area based on the width of the peak. Ideally, in a noise-free scenario, the width of the peak is 0, i.e., b SR =b SL However, in realistic conditions, the image may contain compression artifacts and other noise. The noise may be, for example, a function of standard deviation σ noise In this case, if the image has a fringe area, half the width of the peak is σ noise Thus, the observation noise (half the width of the peak) Δb peak =(b SR -b SL ) / 2, the detection score of the peak feature can be defined using a piecewise linear function, for example, as shown in the following equation (1): peak The larger the value, the more likely it is that there is a surrounding area.

number

[0037] In formula (1), θ peak is the threshold value for determining the peak characteristics. peak,pos and θ peak,neg are the judgment thresholds with margins added to the positive and negative sides of the peak characteristics. peak,pos <θ peak <θ peak,neg From Equation 1, Δb peak <θ peakWhen , the score gradually increases, and Δb peak =θ peak,pos On the other hand, Δb peak ≧θ peak When , the score gradually decreases, and Δb peak =θ peak,neg In Figure 5, Δb peak s as a function of peak 5 is an example graph (500) showing peak <∞, but later the final total The score may be clipped to [-1,1]. The default value of the decision threshold is, for example, θ peak =1.5σ noise , θ peak,pos = σ noise , and θ peak,neg =2σ noise Moreover, the default noise can be defined as, for example, a level σ for a 10-bit input image. noise = 2, but may be adjusted by the user if the expected noise is known. If the input image is known to be noise-free, σ noise may be set to a very small number close to zero.

[0038] A.2.2. Boundary classifier The function of the border classifier is to predict the surrounding area based on the border characteristics. The more pixels on the image border that belong to a peak, the more likely the peak is from the surrounding area. Therefore, the minimum width and height W of the surrounding area are L,min , H L,min Once W is predefined, we can determine the proportion of pixels that belong to the peak on the smallest possible surrounding area at the four boundaries (top, bottom, left, right) of the image. In one embodiment, the surrounding area can be at any boundary, so the maximum proportion from the four boundaries is considered. Empirically, the minimum width and height of the surrounding area can be, for example, W L,min =round(0.01×W) and H L,min ​= round(0.01 × H). The following table shows the ratio r of pixels belonging to a peak on the smallest possible surrounding area. boundary This is an example of a method for calculating [Table 3]

[0039] Referring to the table above, the ratio r boundary The higher the score, the more likely the peak is from a surrounding area. Thus, the boundary feature detection score may be defined using a piecewise linear function, for example, as shown in Equation 2 below. The higher the score, the more likely the surrounding area is present.

number

[0040] ​A.2.3 .total Detection Score total To calculate the detection score, the two weak classifiers in the previous section are combined to obtain a more accurate classifier with a soft classification margin. The surrounding area may satisfy the criteria in both classifiers above. In one embodiment, as shown in Equation 3 below: total The minimum detection score is total are taken into account for the score calculation.

number

[0041] A.3. Surrounding Area Modeling Refer to Figure 7. As a result of what has been disclosed in the previous section, if total Detection score S total >0, peak characteristic b ML , b MR , b SL , b SR It can be concluded that there exists a surrounding area with a peak having a peak-to-peak ratio of 1 / 2. The peak may be modeled, for example, with a Gaussian kernel, which allows the surrounding area to be represented as a smooth surrounding area mask. In one embodiment, two Gaussian kernels are used, one on the left and one on the right of the peak. Furthermore, the mean value of such Gaussian kernels on both sides is defined as the center between the maxima of the peak, as shown in Equation 4 below.

number

number

number

[0042] Below we present some exemplary results illustrating the performance of the disclosed method in the case of rectangular and elliptical surrounding areas, as well as in the case of surrounding areas containing text.

[0043] Figure 8A shows an example of an image containing a peripheral area (81) and a rectangular shaped main area (82). Figure 8B shows the result after applying the peripheral area detection process disclosed above. The resulting detected peripheral area (81') and main area (82') can be seen.

[0044] Figure 9A is an example of an image containing a peripheral area (91) and an oval shaped main area (92). Figure 9B shows the result after applying the peripheral area detection process disclosed above. The resulting detected peripheral area (91') and main area (92') are also shown.

[0045] Figure 10A shows an example of an image including a peripheral area (101) and a main area (102) having a rectangular shape. The peripheral area (101) also includes text (103). The text may be, for example, a closed caption, identifying information, a logo, etc. Figure 10B shows the result after applying the peripheral area detection process disclosed above. The resulting detected peripheral area (101'), main area (102') and text (103') are shown together with a probability map identifying regions of the peripheral area (101') that do not include text / logo (103').

[0046] Figure 10C is an example of an image including a peripheral area (1010) and a main area (1020). It can be seen that the peripheral area (1010) has an arbitrary irregular shape. The above-described method is equally applicable to the example image of Figure 10C.

[0047] According to an embodiment of the present disclosure, total Detection score S total If ≦0, it is concluded that there is no surrounding area, and the surrounding area probability map M L may be filled with zeros.

[0048] B. Surrounding Area Blending According to the estimation of the above-mentioned surrounding area probability map, filtering artifacts near the surrounding area can be reduced by avoiding the surrounding area in image filtering processes, such as the edge-preserving filter described in U.S. Provisional Application No. 63 / 086,699, which is incorporated herein by reference in its entirety. Filtering artifacts are mainly caused by the mismatch of pixel value trends between the image content area and the surrounding area. An efficient method of reducing filtering artifacts according to the teachings of the present disclosure is to blend the surrounding area with the neighboring image content. In this case, there is little mismatch between the image content area and the surrounding area, so filtering artifacts are reduced. From another perspective, this approach is similar to repainting the missing image content in the surrounding area. If the image filter can see the missing image content during filtering, filtering artifacts will be minimized. Common methods for processing filtering boundaries include repeating, mirroring, and periodic padding of boundary pixels, but these are for rectangular image contents and cannot process the boundaries of the surrounding area with arbitrary shapes.

[0049] FIG. 11 shows an example flow chart of a surrounding area blending process including a pre-processing / blending step (1110) and a post-processing / compensation step (1120) according to one embodiment of the present disclosure. Given an input image (1101) and a detected surrounding area probability map (1102) as disclosed above, in step (1103), the surrounding area is blended with the main area of ​​the image to generate a blended image (1104). The blending step (1103) can perform, for example, 1) weighted Gaussian filtering, or 2) mirroring with distance transform. The blended image (1104) is then sent to a filtering process (1105). The filtering process (1105) is an optional step and can consist of any filtering process, such as the edge-preserving filter of U.S. Provisional Application No. 63 / 086,699, which is incorporated herein by reference in its entirety. After filtering (1105), the original surrounding area needs to be restored. Thus, the initial filtered image (1106) is compensated so that the pixel values ​​in the surrounding area are the same as in the original input image (1101). As a result, a compensated image (1107) is generated, which can optionally undergo further processing (1108). An example of further processing (1108) is local reconstruction, as described, for example, in U.S. Provisional Application No. 63 / 086,699. In the following, the various elements of the blending process shown in FIG. 11 are described in more detail.

[0050] B.1. Weighted Gaussian Filtering As mentioned above, an exemplary method of blending is to use weighted Gaussian filtering. Figure 12 shows an exemplary flow chart of a weighted Gaussian filtering process. The goal here is to fill the surrounding area with the nearby image content. Thus, the blending weight (1208) Q is defined as shown in Equation 7 below:

number

number

[0051] With further reference to FIG. 12, after weighted Gaussian filtering, JPEG0007628194000024.jpg79 The surrounding areas within are filled in by the nearby image content. However, the image content areas are also filtered. This may be undesirable since it is only desired to remove discontinuities and not to change the image content. Therefore, to obtain the final blended image (1206) based on the input image (1201), the blending weights (1208), and the weighted filter image (1205), a weighted summation may be performed as shown in Equation 9 below:

number

[0052] B.2 Mirroring with Distance Transformation As mentioned above, another exemplary method of surrounding area blending is to use mirroring with distance transformation. A flow chart illustrating such a method is shown in FIG. 13. According to the teachings of the present disclosure, the surrounding area can be filled by mirroring the image content of the neighborhood. Based on the surrounding area probability map (1302), a thresholding operation is performed on the surrounding area mask to obtain a binary image content mask. Then, a morphological smoothing operation (1303) is applied to remove small noise, which is represented by the following equation (10):

number

[0053] Continuing with reference to the flowchart in Figure 13, the pixel in the image content area closest to each pixel in the surrounding area is known (see also reference [2] mentioned above). Then, using the closest pixel as the origin, a point reflection can be performed to blend the pixels in the surrounding area with the image content. The x- and y-index map (1305) of the closest pixel is calculated as I x and I yThen, the blended image (1306) is obtained as shown in the following equation 11.

number

[0054] (Compensation for surrounding areas) See Equation 11. In one embodiment, the blended image JPEG0007628194000032.jpg78 can be sent to an image filtering process to avoid filtering artifacts and obtain an initial filtered image, which is represented as Equation 12 below.

number

number

[0055] In the above disclosed embodiments, for illustration purposes, the histogram of the Y channel is used for the surrounding area that can be distinguished by luma. In some embodiments (e.g., when luma is not sufficient), the histograms of the U channel and the V channel can be further used for classification in the same manner as disclosed based on the histogram of the Y channel. Furthermore, the classification results of the Y, U, and V channels can also be combined to obtain improved results. As an example, the union of the surrounding areas detected by Y, U, and V may be regarded as the entire detected surrounding area.

[0056] Although several embodiments of the present disclosure have been described, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. Accordingly, other embodiments are within the scope of the following claims.

[0057] The present disclosure is directed to particular aspects for purposes of illustrating some innovative aspects described herein and examples of contexts in which these innovative aspects may be implemented. However, the teachings herein may be applied in a variety of different ways. Moreover, the described embodiments may be implemented in a variety of hardware, software, firmware, and the like. For example, aspects of the present application may be embodied at least in part in an apparatus, a system including multiple devices, a method, a computer program product, and the like. Thus, aspects of the present application may take the form of hardware embodiments, software embodiments (including firmware, resident software, microcode, and the like), and / or embodiments combining both software and hardware aspects. Such embodiments may be referred to herein as "circuits," "modules," "devices," "apparatus," or "engines." Some aspects of the present application may take the form of a computer program product embodied in one or more non-transitory mediums having computer-readable program code embodied thereon. Such non-transitory media may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. Thus, the teachings of the present disclosure are not intended to be limited to the aspects shown in the figures and / or described herein, but have broad applicability.

[0058] The examples presented above are provided as a complete disclosure and explanation to one of ordinary skill in the art of how to make and use the embodiments of the present disclosure, and are not intended to limit the scope of what the inventors regard as their disclosure.

[0059] Modifications to the above-described embodiments for carrying out the methods and systems disclosed herein that are obvious to those skilled in the art are intended to be within the scope of the following claims. All patents and publications mentioned in this specification are indicative of the level of skill of those skilled in the art to which this disclosure pertains. All documents cited in this disclosure are incorporated by reference to the same extent as if each document was individually incorporated by reference in its entirety.

[0060] It is to be understood that the present disclosure is not limited to a particular method or system, which may of course vary. It is also to be understood that the terms used herein are for the purpose of describing particular embodiments only, and are not intended to be limiting. As used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the content clearly dictates otherwise. The term "plurality" includes two or more referents unless the content clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. (References) [1] William M. Wells, "Efficient Synthesis of Gaussian Filters by Cascaded Uniform Filters," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.8, no.2, pp.234-239, 1986. [2] Maurer, Calvin, Rensheng Qi, and Vijay Raghavan, "A Linear Time Algorithm for Computing Exact Euclidean Distance Transforms of Binary Images in Arbitrary Dimensions," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.25, no.2, pp.265-270, 2003.

Claims

1. 1. A method for detecting inactive areas surrounding an active area containing a main subject in an image, comprising: calculating a histogram of pixel values ​​of a predetermined area located on the periphery of the image; determining a peak of the histogram and a width of the peak; generating a peak detection score by classifying the presence of inactive areas in the image based on peaks in the histogram and widths of the peaks; generating a boundary detection score by classifying the presence of the inactive area in the image based on a proportion of pixels in a predetermined outermost periphery of the predetermined area of ​​the image whose pixel values ​​belong to the peak; generating a total score based on a combination of the peak detection scores and the boundary detection scores; detecting the presence of the inactive area based on the total score; The method includes:

2. The method of claim 1, wherein detecting the presence of an inactive area further comprises calculating, for each of a number of bins in the histogram, a probability that it belongs to the inactive area using an inactive area mask, which is a smoothing filter that models the peaks of the histogram.

3. The method of claim 2 , wherein the smoothing filter comprises a Gaussian kernel.

4. generating an inactive area probability map based on the inactive area mask; the inactive area probability map gives, for every pixel in the image, the probability that the pixel of the image belongs to the inactive area; The method of claim 2.

5. Generating the inactive area probability map comprises: Computing the probabilities using a scaled piecewise Gaussian kernel; backprojecting the probabilities to generate the non-active area probability map; The method of claim 4 , comprising:

6. The method of claim 4 , further comprising blending the inactive areas with the active areas based on the inactive area probability map.

7. The method of claim 6 , wherein the blending is performed using weighted Gaussian filtering.

8. calculating blending weights based on the inactive area probability map; generating a weighted image based on the image and the blending weights; The method of claim 7 further comprising:

9. generating filtered weights by Gaussian filtering the blending weights; generating a filtered weighted image by Gaussian filtering the weighted image; generating a weighted filtered image based on the filtered weighted image and the filtering weights; The method of claim 8 , further comprising:

10. The method of claim 9 , further comprising generating a blended image using a weighted addition of the image and the weighting filter image.

11. The method of claim 6 , wherein the blending is performed using mirroring with a distance transform.

12. The mirroring is generating a binary image content mask by thresholding the inactive area mask; applying morphological smoothing to the binary image content mask; Computing a distance transform of the binary image content mask; using the distance transform to identify, for each pixel in the non-active area, pixels in the image content area that are closest to the each pixel; generating an index map of closest pixels; generating a blended image using the image and the index map of nearest pixels; The method of claim 11 , comprising:

13. The method of claim 12 , further comprising filtering the blended image to generate a filtered blended image.

14. The method of claim 13 , wherein the filtering comprises edge-preserving filtering.

15. The method of claim 13 , further comprising performing a weighted addition of the image and the filtered blended image.

16. The method of claim 15 , wherein the weighted summation is based on the non-active area probability map.

17. the image is a frame of a video; the detected inactive areas are areas having constant pixel values ​​and noise and located on the periphery of the image; The inactive area is located outside the active area and is a static area that changes less than the active area. The method of claim 1.

18. The method of claim 1 , wherein the predetermined area is located in the top rows (H L rows) of the image and the bottom rows (H L rows) of the image, and / or the leftmost columns (W L columns) of the image and the rightmost columns (W L columns) of the image.

19. The method of claim 1, wherein generating the boundary detection score includes calculating, for each of four partial areas located at the top, bottom, leftmost and rightmost of the predetermined area, a ratio of pixels whose pixel values ​​belong to the peak among the pixels belonging to the partial area, and determining the boundary detection score based on the maximum value of the ratios for the four partial areas.

20. The method of claim 1 , wherein the peak detection score and the boundary detection score are higher the more likely the presence of the inactive area is.

21. The method of claim 1 , wherein the total score is based on a minimum of the peak detection score and the boundary detection score.

22. The method of claim 1 , wherein the width of the peak is defined by bins on either side of the peak where the histogram values ​​fall to a percentage of the histogram's maximum value.

23. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any of claims 1 to 22.

Images