A method for generating fine-grained patch masks for film movies

Abstract

The present invention is applicable to the field of image restoration technology, and provides a method for generating a refined plaque mask of a film movie, comprising the following steps: step 1, inputting a movie screen containing plaques; step 2, image texture enhancement, specifically comprising: step 21, inputting a screen containing plaque damage, i.e., a target frame, and an adjacent frame, i.e., a reference frame; step 22, image registration; step 23, image texture enhancement based on a Laplace pyramid; step 3, plaque region segmentation, specifically comprising: step 31, image seed pixel point selection; step 32, performing regional growth; step 33, when the termination growth condition is triggered, the growth is stopped, and the final growth region is the segmentation region result of the plaque damage, which is converted into a mask form, i.e., a plaque damage region segmentation mask is obtained; step 4, the plaque damage region segmentation mask is output. The present invention enhances the texture details of a single picture, and accurately segments the plaque damage region by selecting and setting seed pixel points.

Description

Technical Field

[0001] The invention belongs to the technical field of image restoration, and in particular relates to a method for generating a refined patch mask of a film. Background Art

[0002] The gel on the film is often lost due to improper storage, physical damage and many other reasons. In the movie projection, it appears as a single-frame sudden translucent patch, which greatly affects the audience's viewing experience. Therefore, the plaque damage needs to be repaired. The repair work is generally carried out in the following two steps: first, the plaque damage area is segmented to form a plaque mask; then, the plaque mask is used to fill the plaque damage area.

[0003] The accuracy of the plaque mask directly affects the speed and effect of the subsequent plaque repair work. At present, there are four main methods for generating plaque masks: 1) Artificially draw a plaque mask, manually draw a rectangular area at the damage, and form a binary mask of the plaque in the rectangular area. However, the mask formed by this method is usually too large and contains many normal pixels. It is impossible to finely segment the plaque damage, which directly affects the speed and effect of the subsequent repair work. 2) Mask generation method based on U-net network. This method uses the plaque training set and test set to train the U-net network, train the neural network to segment the plaque in the picture, and then obtain the result of image segmentation to form a binary mask of the plaque. However, this method cannot recognize some plaques that are significantly different from the training set. 3) Mask generation method based on edge detection. This method uses the grayscale mutation and discontinuity of the plaque to generate the plaque mask. However, this method does not have obvious color changes in the plaque area, and the segmentation results of small plaques are poor. 4) Mask generation method based on the difference between previous and next frames. Since there is only a small movement between the previous and next frames of a film, the difference between previous and next frames can be used to detect the plaque area by taking advantage of the single-frame burst of the plaque. Typical algorithms include SDI (Spike Detection Index) and ROD (Rank Ordered Difference). However, due to the complex degradation existing in old movies, this type of algorithm usually has a large number of false detection areas in the generated mask image. In view of the above problems, the present invention proposes a refined plaque mask generation method for film. Summary of the invention

[0004] The purpose of the present invention is to provide a method for generating a refined patch mask for a film, aiming to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A method for generating a refined patch mask of a film includes the following steps:

[0007] Step 1, input the movie picture containing plaques;

[0008] Step 2: Image texture enhancement, including:

[0009] Step 21, inputting a picture containing plaque damage, i.e., a target frame, and an adjacent frame, i.e., a reference frame;

[0010] Step 22, image registration: perform image registration and perform optimal affine transformation from the reference frame to the target frame;

[0011] Step 23, image texture enhancement based on Laplace pyramid: perform Laplace pyramid decomposition on the reference frame and the target frame after image registration, and have features and details of different frequency bands on different decomposition layers. Then, perform weighted fusion on the Laplace pyramids of each image to finally obtain the image texture enhancement result;

[0012] Step 3: patch area segmentation, including:

[0013] Step 31, image seed pixel point selection: select seed pixel points in the patch area of ​​the texture enhanced image;

[0014] Step 32, performing region growing: performing growth according to the growth criterion, merging the same or similar pixels around the seed pixel into the seed pixel region, setting the newly added pixel as the seed pixel, and then performing growth again;

[0015] Step 33, when the termination growth condition is triggered, the growth is stopped, and the final growth area is the segmentation area result of the plaque damage, which is converted into a mask form, that is, a plaque damage area segmentation mask is obtained;

[0016] Step 4: Output the plaque damage area segmentation mask.

[0017] Furthermore, the growth criterion is as follows:

[0018] Step 321, with the seed pixel (x0, y0) as the center, search for eight surrounding pixels (x, y), if the gray value difference between the pixel (x, y) and the seed pixel is less than 25 and the newly added pixel (x, y) and the seed pixel (x0, y0) are both less than or greater than the threshold value obtained according to the growing threshold algorithm, merge (x, y) and (x0, y0) into the same area, and push (x, y) into the stack;

[0019] Step 322, take out an (x, y) from the stack and set it to (x0, y0), and repeat step 321;

[0020] Step 323: When no more pixels are added to the stack, the growth ends.

[0021] Furthermore, the growth threshold algorithm is as follows:

[0022] Initial seed pixel Draw an image region of size M×N for the geometric center, and assume that the M×N image region is at pixel point The pixel gray level is ,by The center point The average gray value of the neighborhood is ;

[0023] Set the threshold point (s, t) to divide the pixels into two categories: target and background, where s represents the pixel gray value, and t represents The average gray value of the neighborhood is t;

[0024] remember The gray value is The average gray value of the neighborhood is The frequency of the pixel points appearing in the image area, and the probability density of the pixel points is , The calculation formula is:

[0025] ;

[0026] in, L Represents the maximum grayscale value of the pixel in the image, M and N represent the length and width of the image area respectively;

[0027] The probability that a pixel appears in target area A , the probability of appearing in the background area B Respectively expressed as:

[0028] ;

[0029] ;

[0030] in, Indicates the probability that the gray value of a pixel is in the range [0, s] and the average gray value of the neighborhood is in the range [0, t]; It indicates the probability that the gray value range of the pixel is [s+1, L-1] and the average gray value of the neighborhood is [t+1, L-1];

[0031] The mean vectors corresponding to the target area A and the background area B are as follows:

[0032] ;

[0033] ;

[0034] in, is the mean vector of the target area A; is the mean vector of the background area B; Represents the average grayscale of the pixels in the target area A; Represents the average grayscale of the neighborhood of pixel A in the target area; Represents the average grayscale of pixels in the background area B, represents the average grayscale of the neighborhood of pixel B in the background area; τ represents vector transposition;

[0035] The total mean vector of the image region is expressed as:

[0036] ;

[0037] in, Represents the total mean vector of the selected area; Represents the average grayscale of the pixels in the entire image; Represents the average grayscale of the neighborhood of the pixel in the entire image;

[0038] Ignoring the edge area and the proportion of noise points, we get ;

[0039] Define the discrete measure matrix of the target class and background class :

[0040] ;

[0041] Compute the trace of the discrete measure matrix as a measure for selecting the threshold, The threshold point at which the maximum value is obtained The value of is the required threshold, that is:

[0042] ;

[0043] Among them, s' represents the optimal grayscale threshold; t' represents the optimal neighborhood average grayscale threshold.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The present invention integrates information in the time domain, performs texture enhancement processing on a single picture, improves picture quality, and reduces the negative impact of mixed degradation of film on plaque detection.

[0046] 2. By selecting and setting seed pixels, the present invention can achieve accurate segmentation of plaque damage areas, solving common problems in traditional technologies, such as areas with rapid color changes are easily misdetected as plaques, small plaques are often missed, and large-area patches are misjudged as picture scenes, thereby significantly improving the effect of plaque detection.

[0047] 3. The present invention calculates the growth threshold by using the average grayscale and the neighborhood average grayscale, which effectively reduces the influence of random noise on the threshold calculation, avoids the overgrowth or undergrowth phenomenon that may occur during the regional growth process, and improves the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Flow chart of the method of the present invention.

[0049] Figure 2 Flowchart of the steps for image texture enhancement.

[0050] Figure 3 Flowchart of the patch region segmentation steps.

[0051] Figure 4 Decompose the pyramid for Laplace.

[0052] Figure 5 is the pixel distribution of the image area.

[0053] Figure 6 : is a process diagram of the image texture enhancement step in Example 1; wherein a is the target frame, b is the reference frame, a1 is the affine transformation, b1 is the transformation result, and c is the reconstruction result.

[0054] Figure 7 : is a process diagram of the patch area segmentation step in Example 1; wherein a is the seed pixel point calibration, and b is the patch mask.

[0055] Figure 8 This is a comparison diagram of the patch mask generation effect in Example 2; wherein a is a test image, b is SDI detection, c is ROD detection, d is edge detection, and e is detection by the method of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] An embodiment of the present invention provides a method for generating a fine-grained patch mask for a film, such as Figure 1 As shown, the following steps are included:

[0058] Step 1: Input the movie picture containing plaques.

[0059] Step 2: Image texture enhancement, such as Figure 2 As shown, specifically including:

[0060] Step 21, inputting a picture containing plaque damage, i.e., a target frame, and an adjacent frame, i.e., a reference frame;

[0061] Step 22, image registration: perform image registration and perform optimal affine transformation from the reference frame to the target frame;

[0062] Step 23: Image texture enhancement based on Laplace pyramid (Laplacian decomposition pyramid): Perform Laplace pyramid decomposition on the reference frame and target frame after image registration. Different decomposition layers have features and details of different frequency bands.

[0063] Then the Laplace pyramids of each image are weighted fused to finally obtain the image texture enhancement result, that is, the target image after texture enhancement. Figure 4 As shown. First, the target frame I1 and the reference frame I2 after registration are Gaussian down-sampled three times in a row to obtain the low-frequency features of the image; the results of Gaussian down-sampling G1, G2, and G3 are Gaussian up-sampled respectively to restore to the original image size; after being restored to the original image size, they are subtracted from the corresponding original image to obtain the high-frequency residual components L1, L2, and L3 of the image; the high-frequency residual components L1, L2, and L3 of the target frame I1 and the high-frequency residual components L1, L2, and L3 of the reference frame I2 after registration are weighted fused with the target frame I1 to obtain the final output.

[0064] Step 3: segment the patch area, such as Figure 3 As shown, specifically including:

[0065] Step 31, image seed pixel point selection: for the texture enhanced image, seed pixel points are selected in the patch area;

[0066] Step 32, performing region growing: performing growth according to the growth criterion, merging the same or similar pixels around the seed pixel into the seed pixel region, setting the newly added pixel as the seed pixel, and then performing growth again;

[0067] Step 33, when the termination growth condition is triggered, the growth is stopped, and the final growth area is the segmentation area result of the plaque damage, which is converted into a mask form, that is, the plaque damage area segmentation mask is obtained.

[0068] The growth criteria are as follows:

[0069] Step 321, with the seed pixel (x0, y0) as the center, search for eight surrounding pixels (x, y), if the gray value difference between the pixel (x, y) and the seed pixel is less than 25 and the newly added pixel (x, y) and the seed pixel (x0, y0) are both less than or greater than the threshold value obtained according to the growing threshold algorithm, merge (x, y) and (x0, y0) into the same area, and push (x, y) into the stack;

[0070] Step 322, take out an (x, y) from the stack and set it to (x0, y0), and repeat step 321;

[0071] Step 323: When no more pixels are added to the stack, the growth ends.

[0072] The growth threshold algorithm is as follows:

[0073] Initial seed pixel Draw an image region of size M×N for the geometric center, and assume that the M×N image region is at pixel point The pixel gray level is ,by The center point The average gray value of the neighborhood is .

[0074] The pixel distribution of the image area can be given by Figure 5 The pixels close to the diagonal area are scene pixels, and the pixels far from the diagonal area are image edge points or noise. By setting the threshold point (s, t), the pixels are divided into two categories: target and background, where s represents the pixel gray value, and t represents The average gray value of the neighborhood is t.

[0075] remember The gray value is The average gray value of the neighborhood is The frequency of the pixel points appearing in the image area, and the probability density of the pixel points is , The calculation formula is:

[0076] ;

[0077] in, L Represents the maximum grayscale value of the pixel in the image, M and N represent the length and width of the image area respectively;

[0078] The probability that a pixel appears in target area A , the probability of appearing in the background area B Respectively expressed as:

[0079] ;

[0080] ;

[0081] in, Indicates the probability that the gray value of a pixel is in the range [0, s] and the average gray value of the neighborhood is in the range [0, t]; It indicates the probability that the gray value range of the pixel is [s+1, L-1] and the average gray value of the neighborhood is [t+1, L-1];

[0082] The mean vectors corresponding to the target area A and the background area B are as follows:

[0083] ;

[0084] ;

[0085] in, is the mean vector of the target area A; is the mean vector of the background area B; Represents the average grayscale of the pixels in the target area A; Represents the average grayscale of the neighborhood of pixel A in the target area; Represents the average grayscale of pixels in the background area B, represents the average grayscale of the neighborhood of pixel B in the background area; τ represents vector transposition;

[0086] The total mean vector of the image region is expressed as:

[0087] ;

[0088] in, Represents the total mean vector of the selected area; Represents the average grayscale of the pixels in the entire image; Represents the average grayscale of the neighborhood of the pixel in the entire image;

[0089] In practical applications, since the edge area and noise points account for a very small proportion, there are ;

[0090] Define the discrete measure matrix of the target class and background class :

[0091] ;

[0092] The trace of the discrete measure matrix is ​​calculated as the measure for selecting the threshold. This measure is essentially the sum of the variances of the target area and the overall image, the background area and the overall image, the average grayscale variance and the neighborhood average grayscale variance. The larger the sum of the variances, the greater the degree of discreteness, that is, the greater the difference between the target and the background. The threshold point at which the maximum value is obtained The value of is the required threshold, that is:

[0093] ;

[0094] Among them, s' represents the optimal grayscale threshold; t' represents the optimal neighborhood average grayscale threshold.

[0095] Step 4: Output the plaque damage area segmentation mask.

[0096] The specific implementation of the present invention is described in detail below in conjunction with specific embodiments.

[0097] Embodiment 1: Affine transformation of similar region images: transform the reference frame ( Figure 6 b) Optimal mapping to the target frame ( Figure 6 a) in the upper part, Figure 6 a1 is the affine transformation process, Figure 6 b1 is the transformation result. Figure 6 b1) and the target frame ( Figure 6 a) Perform Laplace pyramid decomposition respectively and obtain the transformation results respectively ( Figure 6 The high-frequency residual component of b1) and the target frame ( Figure 6 The high-frequency residual component of a) in the transformation result ( Figure 6 b1), target frame ( Figure 6 The high-frequency residual component of the target frame ( Figure 6 In a), weighted fusion is performed to finally obtain the output result of texture enhancement, such as Figure 6 As shown in c.

[0098] Set the target frame ( Figure 6 a) and the reconstruction result ( Figure 6 (c) is used as the input image for calculating the PSNR indicator. The calculated PSNR=41.51 shows that the image quality has been improved.

[0099] We set a five-pointed star as a mark at the seed pixel position, such as Figure 7 As shown in a, follow the above steps 32 and 33 to obtain the patch mask result as shown in Figure 7 As shown in Figure b. It can be seen that the patch area in the film is completely and accurately segmented. Due to the set seed pixel points, the influence of complex scenes in the picture on patch detection is reduced; the growth threshold calculated by the average grayscale and regional average grayscale threshold algorithm is appropriate, and there is no overgrowth or undergrowth; due to the set texture enhancement module, the texture detail quality of the patch is improved, which increases the accuracy of patch segmentation.

[0100] Example 2: Comparison of patch mask generation effects;

[0101] like Figure 8 As shown in a, it is a test image; the test image is subjected to SDI detection, ROD detection, edge detection and detection by the method of the present invention, and the results are as follows Figure 8 As shown in Figure 3, it can be seen that the three detection methods, SDI detection, ROD detection and edge detection, have different degrees of false detection in the detection results, and some noise and complex background environments are judged as patch areas. The method of the present invention can reduce the influence of complex background environments, scenes in the picture and noise, and accurately segment the patch area.

[0102] The above are only preferred embodiments of the present invention. It should be pointed out that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention. These should also be regarded as the protection scope of the present invention. These will not affect the effect of the implementation of the present invention and the practicality of the patent.

Claims

1. A method for generating a fine-grained patch mask for a film, characterized in that: The following steps are involved: Step 1, input the movie picture containing plaques; Step 2: Image texture enhancement, including: Step 21, inputting a picture containing plaque damage, i.e., a target frame, and an adjacent frame, i.e., a reference frame; Step 22, image registration: perform image registration and perform optimal affine transformation from the reference frame to the target frame; Step 23, image texture enhancement based on Laplace pyramid: perform Laplace pyramid decomposition on the reference frame and the target frame after image registration to obtain high-frequency residual components on different decomposition layers, and then perform weighted fusion on the high-frequency residual components on each decomposition layer with the target frame to finally obtain the image texture enhancement result; Step 3: patch area segmentation, including: Step 31, image seed pixel point selection: select seed pixel points in the patch area of ​​the texture enhanced image; Step 32, performing region growing: performing growth according to the growth criterion, merging the same or similar pixels around the seed pixel into the seed pixel region, setting the newly added pixel as the seed pixel, and then performing growth again; Step 33, when the termination growth condition is triggered, the growth is stopped, and the final growth area is the segmentation area result of the plaque damage, which is converted into a mask form, that is, a plaque damage area segmentation mask is obtained; Step 4: Output the plaque damage area segmentation mask; The growth criteria are as follows: Step 321, with the seed pixel (x0, y0) as the center, search for eight surrounding pixels (x, y), if the gray value difference between the pixel (x, y) and the seed pixel is less than 25 and the newly added pixel (x, y) and the seed pixel (x0, y0) are both less than or greater than the threshold value obtained according to the growing threshold algorithm, merge (x, y) and (x0, y0) into the same area, and push (x, y) into the stack; Step 322, take out an (x, y) from the stack and set it to (x0, y0), and repeat step 321; Step 323: When no more pixels are added to the stack, the growth ends; The growth threshold algorithm is as follows: Take the initial seed pixel (x0, y0) as the geometric center to draw an image region of size M×N. Let the pixel grayscale at the pixel point (x, y) of the M×N image region be f(x, y), and the average grayscale value of the k×k neighborhood with (x, y) as the center point be g(x, y); Set the threshold point (s, t) to divide the pixels into two categories: target and background, where s represents the pixel gray value s, and t represents the average gray value of the k×k neighborhood t; Remember C ij is the frequency of the pixel with gray value i and neighborhood average gray value j appearing in the image area, and the probability density of the pixel is P ij ,P ij The calculation formula is: Where L represents the maximum grayscale value of the pixel in the image, M and N represent the length and width of the image area, i represents the grayscale value, and j represents the average grayscale value of the neighborhood; The probability P of a pixel appearing in the target area A A , the probability P of appearing in the background area B B Respectively expressed as: Among them, P A It represents the probability that the gray value of the pixel is in the range of [0, s] and the average gray value of the neighborhood is in the range of [0, t]; P B It indicates the probability that the gray value range of the pixel is [s+1, L-1] and the average gray value of the neighborhood is [t+1, L-1]; The mean vectors corresponding to the target area A and the background area B are as follows: Among them, μ A is the mean vector of the target area A; μ B is the mean vector of the background area B; μ Ai Indicates the average grayscale of the pixels in the target area A; μ Aj Indicates the average grayscale of the neighborhood of pixel A in the target area; μ Bi Represents the average grayscale of pixels in background area B, μ Bj represents the average grayscale of the neighborhood of pixel B in the background area; τ represents vector transposition; The total mean vector of the image region is expressed as: Where μ represents the total mean vector of the selected area; μ i Represents the average grayscale of the pixels in the entire image; μ j Represents the average grayscale of the neighborhood of the pixel in the entire image; Ignore the edge area and the proportion of noise points, and get P A +P B ≈1; μ≈P A μ A +P B μ B ; Define the discrete measure matrix σ of the target class and background class: Calculate the trace of the discrete measure matrix as the measure for selecting the threshold. The value of the threshold point (s', t') when tr(σ) reaches the maximum value is the required threshold, that is: Among them, s' represents the optimal grayscale threshold; t' represents the optimal neighborhood average grayscale threshold.

Images