Video noise reduction method and video noise reduction system based on portrait mode
Through the video noise reduction method based on portrait mode, the images are divided into foreground, background and contour areas, combined with time domain and airspace filtering technology, the problems of poor noise suppression and high-resolution image processing in the existing technology are solved, and the problem of poor noise suppression effect and high-resolution image processing is realized. Efficient noise suppression and image quality improvement.
Patent Information
- Application Number
- CN202310642073.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-01
- Publication Date
- 2024-12-06
AI Technical Summary
Existing video noise suppression technology is difficult to distinguish between object movement and noise caused by noise in ultra-low brightness environments with excessive noise intensity, and it is difficult to realize real-time computing of high-resolution images by neural network models, and the noise suppression effect is limited by training data. .
The video noise reduction method based on portrait mode is adopted, and the image is divided into foreground, background and contour areas through the portrait mode model. Combined with time domain and airspace filtering technology, filtering is performed separately for stationary and moving areas to reduce noise and avoid afterimage.
It realizes effective noise suppression in environments with high noise intensity, reduces afterimage phenomena, and realizes real-time computing in high-resolution image processing, improving the clarity of image quality.
Smart Images

Figure CN119090754A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a noise reduction method and a noise reduction system, and in particular to a video noise reduction method and a video noise reduction system based on a portrait mode. Background Art
[0002] Video noise suppression technology can be divided into two categories: traditional image processing technology and neural network technology. Among them, the traditional image technology with better effect is mainly temporal noise suppression, which uses the previous image information to filter the current image in the temporal domain to achieve the effect of noise removal. However, compared with spatial noise suppression, which will lose details, temporal noise suppression can better preserve details.
[0003] It should be noted that temporal noise suppression will produce residual images when the object moves, so motion detection of the moving object is required to avoid the residual image phenomenon. However, in an ultra-low brightness environment with excessive noise intensity, the above motion detection method is difficult to determine whether the image change is caused by the real movement of the object or noise, and cannot achieve a good noise suppression effect.
[0004] In addition, in the existing noise suppression technology, a noise suppression method based on a neural network model is also used. In this method, a noisy image can be input into the neural network model, and a noise-suppressed image is output. However, since the size of the input image determines the scale of the model, for high-resolution images, the amount of calculation will be too large to achieve real-time calculation, and the effect of noise suppression is usually limited by the training data and has its limits. Summary of the invention
[0005] The technical problem to be solved by the present invention is to provide a video noise reduction method and a video noise reduction system based on a portrait mode in view of the deficiencies in the prior art.
[0006] In order to solve the above technical problems, one of the technical solutions adopted by the present invention is to provide a video denoising method based on portrait mode, which includes configuring a processor to perform the following steps: inputting a current image into a portrait mode model to generate a current portrait probability image, which defines the probability that each sub-region in the current input image is a foreground portrait; executing a post-processing program, including: performing a binarization process on the current portrait probability image according to a probability threshold to generate a mask image; and executing a contour extraction program to extract a foreground area, a background area and a contour area from the mask image. The video denoising method also includes executing a filtering program, including: executing a motion detection program, comparing the current image with the previous image, and finding a moving area and a static area from the current image based on the extracted foreground area, background area and contour area. Performing a temporal filtering operation on the static area. For the moving area, finding a similar part in the previous image that is similar to the current image and obtaining the degree of difference, so as to determine whether to compensate the moving area with the similar part, or to perform a spatial filtering operation. Outputting the filtered static area and moving area to generate a filtered current image.
[0007] In order to solve the above technical problems, another technical solution adopted by the present invention is to provide a video denoising system based on portrait mode, which includes a processor and a memory. The memory stores a plurality of computer-readable instructions, a current image and a previous image. The processor is configured to read these computer-readable instructions and perform the following steps: input the current image into the portrait mode model to generate a current portrait probability image, which defines the probability that each sub-region in the current input image is a foreground portrait; execute a post-processing program, including: binarizing the current portrait probability image according to the probability threshold to generate a mask image; and executing a contour extraction program to extract the foreground area, background area and contour area from the mask image. The video denoising method also includes executing a filtering program, including: executing a motion detection program, comparing the current image with the previous image, and finding the moving area and the static area from the current image according to the extracted foreground area, background area and contour area. Performing a temporal filtering operation on the static area. For the moving area, find the similar part in the previous image that is similar to the current image and obtain the degree of difference, so as to determine whether to compensate the moving area with the similar part, or to perform a spatial filtering operation. The filtered static area and the moving area are output to generate a filtered current image.
[0008] In order to further understand the features and technical contents of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and description and are not used to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] Figure 1FIG. 4 is a functional block diagram of a video noise reduction system according to an embodiment of the present invention.
[0010] Figure 2 The present invention is a flow chart of a video noise reduction method based on portrait mode.
[0011] Figure 3 for Figure 2 Detailed flowchart of step S11.
[0012] Figure 4 is a schematic diagram of a current image according to an embodiment of the present invention.
[0013] Figure 5 Schematic diagram of the mask image.
[0014] Figure 6 for Figure 3 Detailed flowchart of step S114.
[0015] Figure 7 Schematic diagram of the judgment results of the foreground area, the background area and the contour area according to an embodiment of the present invention.
[0016] Figure 8 for Figure 2 Detailed flowchart of step S12. DETAILED DESCRIPTION
[0017] The following is an explanation of the implementation methods of the "video noise reduction method and video noise reduction system based on portrait mode" disclosed in the present invention through specific embodiments. Those skilled in the art can understand the advantages and effects of the present invention from the contents disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and the details in this specification can also be modified and changed in various ways based on different viewpoints and applications without departing from the concept of the present invention. In addition, it is stated in advance that the drawings of the present invention are only simple schematic illustrations and are not depicted according to actual sizes. The following embodiments will further explain the relevant technical contents of the present invention in detail, but the disclosed contents are not intended to limit the scope of protection of the present invention. In addition, the term "or" used in this article may include any one or more combinations of the associated listed items depending on the actual situation.
[0018] Figure 1 FIG. 1 is a functional block diagram of a video noise reduction system according to an embodiment of the present invention. Figure 1As shown, one embodiment of the present invention provides a video noise reduction system 1 based on portrait mode, which includes a processor 10 and a memory 12. The processor 10 and the memory 12 can be, for example, arranged on an image processing chip 2, and the image processing chip 2 is used to receive a video source VS, and after processing the image signal provided by the video source VS, display it through a display 3. The present invention mainly combines the portrait mode technology with the noise reduction technology when the processor 10 processes the image signal input by the video source VS, so as to achieve a good quality and clear display effect. The processor 10 is electrically coupled to the memory 12, and is configured to access computer-readable instructions D1 from the memory 10. The memory 12 is any storage device for storing data, such as, but not limited to, a random access memory (RAM), a read only memory (ROM), a flash memory, a hard disk or other storage device that can be used to store data. The memory 12 stores at least a plurality of computer-readable instructions D1. In one embodiment, the memory 10 is also divided into an image buffer D3, which can be used to store the current image and previous images from the video source VS, as well as the temporary image generated when the processor 12 performs calculations, and the portrait mode model D2, post-processing program D4 and filtering program D5 used in subsequent steps.
[0019] Please refer to Figure 2 , which is a flow chart of the video denoising method based on the portrait mode of the present invention. It should be noted that the video denoising method provided by the present invention can be applied to different color spaces, such as Gray, Raw, RGB, YUV and other color spaces.
[0020] like Figure 2 As shown, the video noise reduction method includes configuring the processor 10 to perform the following steps:
[0021] Step S10: Input the current image into the portrait mode model to generate the current portrait probability image. In detail, the current input image includes multiple sub-regions (e.g., pixels), and the portrait probability image defines the probability that each sub-region in the current input image is a foreground portrait. The portrait mode model D2 can be trained based on a deep learning algorithm, and the deep learning algorithm includes but is not limited to Yolact++, PortraitNet, and Deeplabv3+. When the portrait mode model D2 is running, the current image is input into the portrait mode model D2, and the model will output a segmentation result to distinguish between the background area and the foreground portrait area. More precisely, the portrait mode model D2 can determine the probability that each sub-region (each pixel) of the current image is a foreground portrait, and can, for example, represent this probability by a value ranging from 0 to 1 (i.e., 0% to 100%), and correspondingly represent it in the current portrait probability image. However, the above description of the type of the portrait mode model D2 is only an example, and various existing portrait acquisition models can also be used.
[0022] Specifically, the portrait mode model D2 can be trained to distinguish (segment) the foreground portrait from the background in the target image. In this embodiment, the portrait mode model D2 can be trained using the following training process: First, a target training image is selected from an image sample data set including multiple labeled images, and is input into the portrait mode model for processing to obtain a segmentation result of the target training image.
[0023] Next, the loss function is determined according to the segmentation result corresponding to the target training image and the annotated segmentation label. The portrait mode model D2 is trained according to the loss function, that is, the parameters of the portrait mode model D2 are updated according to the loss function. Finally, the portrait mode model D2 is repeatedly trained according to the loss function, and each time the training is performed, it is determined whether the predetermined training termination condition is met. If the training termination condition is not met, the training is continued until the training termination condition is met, and finally a trained portrait mode model D2 is generated.
[0024] Since the size of the input image determines the scale of the model, for high-resolution images, the amount of computation will be too large to achieve real-time computing. In other implementations of step S10, in order to achieve real-time computing, the current image can be reduced from the first image size to the second image size corresponding to the portrait mode model D2, and then input into the portrait mode model D2. For example, for a video source VS with a resolution of 4K, the (initial) first image size of the current image is 3840×2160, and the current image can be reduced to a second image size of 224×160 by, for example, a bilinear interpolation method. However, the above is only an example, and the present invention is not limited thereto. Since the portrait mode model D2 inputs an image of a smaller size (224x160), the amount of computation can be reduced for real-time computing.
[0025] Step S11: Execute post-processing procedure. Please refer to Figure 3 , which is Figure 2 As shown in the figure, the post-processing procedure mainly performs processing steps such as amplification, binarization, and contour extraction, for example, including the following steps:
[0026] Step S110: Enlarging the current portrait probability image from the second image size to the first image size. Specifically, if the current image is reduced in step S10, the current portrait probability image will correspond to the second image size, and this step is required to restore the size of the current portrait probability image. Similarly, the current portrait probability image can be enlarged from the second image size to the first image size by a bilinear interpolation calculation method.
[0027] Step S112: performing binarization processing on the current portrait probability image according to the probability threshold to generate a mask image.
[0028] This step is mainly used to obtain the foreground portrait and background parts from the foreground and background segmentation results of the portrait mode model D2. Figure 4 as well as Figure 5 , Figure 4 is a schematic diagram of a current image according to an embodiment of the present invention, Figure 5 is a schematic diagram of the mask image. Figure 4 , 5 As shown, after the current image IMG is input into the portrait mode model D2, a portrait probability image can be generated, and the binarization process may include setting the sub-region (e.g., pixel) whose probability is greater than or equal to the probability threshold in the current portrait probability image to a first value (e.g., Figure 5 and setting the sub-region whose probability is less than the probability threshold to a second value (such as Figure 5 The mask image MK has a plurality of mask sub-regions (eg, pixels) corresponding to the sub-regions of the current portrait probability image.
[0029] For example, step S112 can be expressed as the following formula (1):
[0030]
[0031] Wherein, Mask is the mask image MK, Mask(i,j) represents the value of the mask image at the coordinate (i,j), Probability(i,j) is the probability of the current portrait probability image at the coordinate (i,j) (in the range of 0 to 1), and TH is the probability threshold. TH can be set to 0.5, as shown in formula (1). When Probability(i,j) is 0.7, the corresponding Mask(i,j) is set to 1. Therefore, in the mask image MK, the sub-region with a higher probability of being the foreground portrait will be set to 1, and the remaining sub-regions with a higher probability of being the background will be set to 0.
[0032] Step S114: Execute the contour extraction procedure to extract the foreground area, the background area and the contour area from the mask image. Specifically, the main purpose of this step is to detect the transition area between the foreground portrait and the background, that is, the contour area of the human figure.
[0033] It should be noted that if the contour extraction step is omitted and only the foreground portrait and background regions are adjusted (for example, noise suppression processing), the noise suppression effect will depend on the accuracy of the portrait mode model D2. If the accuracy of the portrait mode model D2 is insufficient, the noise suppression effect may be poor for the boundary region between the foreground portrait and the background. In other words, afterimages may occur in the boundary region between the foreground portrait and the background.
[0034] Please refer to further Figure 6 , which is Figure 3 Detailed flowchart of step S114.
[0035] Step S1140: Divide the mask image into a plurality of blocks. Specifically, in this step, the block radius may be set first, and the blocks are divided from the mask image according to the block radius. Optionally, the blocks containing both the foreground portrait and the background may be selected first, that is, the blocks in the transition area between the foreground portrait and the background. More precisely, according to the setting method of the aforementioned steps, the blocks containing both the mask sub-areas set to the first value and the second value may be selected, which can save resources and time required for subsequent processing.
[0036] Next, for each of these (selected) blocks, perform the following steps:
[0037] Step S1141: Calculate the number of mask sub-regions corresponding to the current block that are set to a first value as the number of foreground regions of the current block.
[0038] This step can be expressed by the following formula (2):
[0039]
[0040] Among them, Number(i,j) represents the number of foreground areas in the current block centered at coordinates (i,j), r is the block radius, and Mask(a,b) is the value of the mask sub-block located at coordinates (a,b) in the current block. Therefore, a is within the range of the block radius extending outward from i as the center (i.e., ir to i+r), and b is within the range of the block radius extending outward from j as the center (i.e., jr to h+r). Since the value of the mask sub-block in the foreground area is 1, the number of foreground areas is obtained after addition according to formula (2).
[0041] Step S1142: Calculate the foreground ratio of the foreground area in the current block to determine whether the current block is a foreground area, a background area or a contour area. This step can be shown in the following formula (3):
[0042] Mask_Ratio(i,j)=Number(i,h) / (2*r+1) 2 …Formula (3);
[0043] Among them, Mask_Ratio(i,j) represents the foreground ratio of the number of foreground areas in the current block centered at coordinates (i,j), Number(i,j) is the number of foreground areas in the current block, (2*r+1) 2 Represents the total number of mask sub-regions of the current block calculated according to the block radius r.
[0044] Can also refer to Figure 7 , which is a schematic diagram of the judgment results of the foreground area, the background area and the contour area according to an embodiment of the present invention. In step S1142, the contour area AC may be judged first, for example, as shown in the following formula (3):
[0045] In addition, the contour area can be determined according to the following formula (4):
[0046]
[0047] Wherein, Contour(i,j) is a function used to indicate whether the current block at coordinate (i,j) is a contour area AC. When Contour(i,j)=1, it means that the current block at coordinate (i,j) is a contour area AC, and when Contour(i,j)=0, it is not a contour area AC. In formula (4), whether the current block is a contour area AC is determined according to the contour ratio range, Contour_TH_L is the lower limit of the contour ratio range, and Contour_TH_H is the upper limit of the contour ratio range.
[0048] Therefore, when the foreground ratio is within the contour ratio range (ie, the foreground ratio Mask_Ratio(i,j) is greater than or equal to Contour_TH_L and less than or equal to Contour_TH_H), the current block can be determined to be the contour area AC.
[0049] Please refer to Table 1:
[0050] 1 0 0 1 1 0 1 1 1
[0051] Table 1 takes the current block with a block radius r of 1 as an example, which can be obtained by (2*r+1) 2 It is calculated that there are 9 mask sub-regions (i.e., 3*3) in this current block. Assuming Contour_TH_L is 0.2 and Contour_TH_H is 0.8, the number of foregrounds calculated in this area is 6, and the ratio of the number of foregrounds is 6 / (3*3)=0.6666, which is between 0.2 and 0.8. Therefore, Contour(i,j)=1, and the current block corresponding to Table 1 is the contour area AC.
[0052] Please refer to Table 2:
[0053] 1 1 0 1 1 1 1 1 1
[0054] In the current block corresponding to Table 2, the calculated foreground number is 8, and the foreground number ratio is 8 / (3x3)=0.8889, which exceeds 0.8, so Contour(i,j)=0, and the current block is not the contour area AC.
[0055] In addition to the above judgment method, when the foreground ratio is greater than the upper limit of the contour ratio range (i.e. Contour_TH_H), the current block can be judged as the foreground area AF, and when the foreground ratio is less than the lower limit of the contour ratio range (i.e. Contour_TH_L), the front block is judged as the background area AB.
[0056] Therefore, the foreground area AF, the background area AB and the contour area AC can be obtained, as shown in FIG. Figure 7 As shown, the foreground area AF, the background area AB and the contour area AC may correspond to different weights respectively, and these weights will be used in subsequent steps. Here, the post-processing procedure comes to an end.
[0057] Please refer to Figure 2 , the video noise reduction method enters step S12: executing the filtering process.
[0058] For reference Figure 8 , which is Figure 2 Detailed flowchart of step S12.
[0059] like Figure 8As shown, the filtering procedure also includes the following steps:
[0060] Step S120: executing a motion detection procedure, comparing the current image with the previous image, and finding a moving area and a stationary area from the current image according to the extracted foreground area, background area, and contour area.
[0061] In detail, this step is mainly to distinguish between moving areas and stationary areas so that different filtering methods can be used in subsequent steps. In some embodiments, the current image can be compared with the previous image in a frame difference manner, for example, the current image is the current frame image, and the previous image is the previous frame image. Alternatively, in some embodiments, the previous frame image is a frame image processed by the aforementioned post-processing program and processed by the filtering program of this embodiment.
[0062] The step of comparing the current image with the previous image can be expressed as follows:
[0063] FrameDiff(i,j)=|Prev(i,j)-Cur(i,j)|...Equation (5);
[0064] Among them, FrameDiff(i,j) is the comparison result at the coordinate (i,j), Prev(i,j) represents the pixel value of the previous image at the coordinate (i,j), and Cur(i,j) represents the pixel value of the current image at the coordinate (i,j). The difference is the difference between the two.
[0065] As mentioned above, the foreground area AF, the background area AB and the contour area AC correspond to different weights respectively, and in the motion detection program, each area in the current image can be judged as belonging to the foreground area AF, the background area AB or the contour area AC, and the comparison result of the current image and the previous image is weighted according to the corresponding weight to find the moving area and the static area.
[0066] The following first describes a method of finding the moving area and the stationary area simply by using the frame difference method, which can be shown in the following formula (6):
[0067]
[0068] Among them, Motion(i,j) is a function used to indicate whether the area at coordinates (i,j) is a moving area, and MotionTH is a difference threshold. Therefore, when the comparison result at coordinates (i,j) is greater than or equal to the difference threshold, Motion(i,j) = 1, and the area at coordinates (i,j) is a moving area. When the comparison result at coordinates (i,j) is less than the difference threshold, Motion(i,j) = 0, and the area at coordinates (i,j) is a stationary area.
[0069] Next, based on equation (6), the foreground area AF, the background area AB or the contour area AC can be further considered, and the comparison results of the current image and the previous image can be weighted according to the corresponding weights to find the moving area, as shown in equation (7):
[0070]
[0071] Among them, FrameDiffPortrait(i,j) is a function used to indicate whether the area located at the coordinates (i,j) is a moving area. When the area located at the coordinates (i,j) is the contour area AC (i.e., Contour(i,j)=1), the comparison result (FrameDiff(i,j)) at the coordinates (i,j) is multiplied by the weight ContourRatio corresponding to the contour area AC. When the area located at the coordinates (i,j) is the foreground area AF (i.e., Mask(i,j)=1), the comparison result (FrameDiff(i,j)) at the coordinates (i,j) is multiplied by the weight ForegroundRatio corresponding to the foreground area AF. Similarly, when the area located at the coordinates (i,j) is the background area AB (i.e., Mask(i,j)=1), the comparison result (FrameDiff(i,j)) at the coordinates (i,j) is multiplied by the weight BackgroundRatio corresponding to the background area AB. After that, FrameDiff(i,j) in equation (6) is replaced by FrameDiffPortrait(i,j) to obtain the result of motion detection under consideration of the foreground area AF, the background area AB or the contour area AC. In the above steps, by dividing the image into three areas (contour area AC, foreground area AF and background area AB) and adjusting these three areas to different degrees during motion detection, the noise in the overall area can be well suppressed and the afterimage phenomenon can be reduced.
[0072] Step S122: performing a time domain filtering operation on the static area.
[0073] The temporal filtering operation is to allocate the weight of the current image and the previous image to each sub-region (pixel) in the static region according to the learning rate, as shown in the following formula (8):
[0074] TemperalDenoise(i,j)=Cur(i,j)*LearnRate+Prev(i,j)*(1-LearnRate)...Equation (8);
[0075] Among them, TemperalDenoise(i,j) represents the temporal filtering result of the area located at coordinate (i,j), Prev(i,j) represents the pixel value of the previous image at coordinate (i,j), Cur(i,j) represents the pixel value of the current image at coordinate (i,j), and LearnRate represents the learning rate, that is, the proportion of the current image and the previous image in the area located at coordinate (i,j).
[0076] Step S124: for the moving area, find out the similar part in the previous image that is similar to the current image and obtain the difference degree.
[0077] In detail, if temporal filtering is not used in the moving area, it may cause too much noise in the area. Therefore, a similar sub-area is found from the previous image for compensation. If there is no similar sub-area, spatial filtering is performed on the current sub-area.
[0078] In this step, for each sub-region corresponding to the moving region, another sub-region that is most similar to the current sub-region can be found from the previous image, which can be expressed by the following formula (9):
[0079] Compensation(i,j)=Prev(a,b),min|Prev(a,b)-Cur(i,j)|,a∈[ir,i+r],b∈[jr,j+r]…Equation (9);
[0080] Where Compensation(i,j) represents the value that can be used to compensate for the current sub-region at coordinate (i,j), Prev(a,b) is another sub-region found from the coordinate (a,b) of the previous image that has the smallest difference (i.e., is most similar) to the sub-region Cur(i,j) at coordinate (i,j) in the current image, and r is the search radius centered on coordinate (i,j).
[0081] Step S125: Determine whether to use a similar part to compensate the moving area or perform a spatial filtering operation according to the degree of difference.
[0082] This step can be expressed by the following formula (10):
[0083]
[0084] Wherein, MotionResult(i,j) is the filtering result of the current sub-region at the coordinate (i,j) (i.e., the filtering result of the moving region), DIFF_TH is the difference degree threshold, when the absolute value of the difference between the value Compensation(i,j) of the current sub-region at the coordinate (i,j) and the sub-region Cur(i,j) at the coordinate (i,j) in the current image is less than the difference degree threshold DIFF_TH, MotionResult(i,j)=Compensation(i,j), which means that the difference degree is small, and the process can proceed to step S126 to compensate the moving region with the similar part (i.e., Compensation(i,j)). When the absolute value of the difference between the value Compensation(i,j) of the current sub-region at the coordinate (i,j) and the sub-region Cur(i,j) at the coordinate (i,j) in the current image is greater than or equal to the difference degree threshold DIFF_TH, MotionResult(i,j)=SpatialDenosion(i,j), which means that the difference degree is small, and the process needs to proceed to step S127 to perform the spatial domain filtering operation. That is to say, after performing spatial filtering on the current sub-region at the coordinate (i, j), the spatial filtering result SpatialDenosie(i, j) is obtained.
[0085] In step S127, the spatial domain filtering operation may be processed by, for example, mean filtering, as shown in the following equation (11):
[0086]
[0087] Among them, SpatialDenosie(i,j) is the spatial domain filtering result of the current sub-region at the coordinate (i,j), Cur(a,b) is the value of the sub-region at the coordinate (a,b), and formula (11) adds the values of all sub-regions in the region of radius r and averages them according to the total number of sub-regions in the region of radius r.
[0088] Next, the filtering results of the moving area in steps S126 and S127 can be combined to serve as the filtering result for the moving area. Here, the filtering process for the moving area and the stationary area is completed.
[0089] Finally, the video noise reduction method enters step S13: outputting the filtered static area and the moving area to generate a filtered current image. This step can be represented by the following formula (12):
[0090]
[0091] Among them, FinalResult(i,j) represents the value of the current image after filtering at the coordinate (i,j), MotionResult(i,j) is the filtering result of the sub-area at the coordinate (i,j) for the moving area, and TemperalDenoise(i,j) is the temporal filtering result of the area located at the coordinate (i,j).
[0092] Advantageous Effects of Embodiments
[0093] One of the beneficial effects of the present invention is that the video noise reduction method and video noise reduction system based on the portrait mode provided by the present invention can combine the portrait mode technology with the noise reduction technology to achieve a display effect with good picture quality and clarity.
[0094] Going further, the portrait mode technology divides the image into three areas (contour area, foreground area and background area), and adjusts these three areas to varying degrees during motion detection. Finally, time domain filtering, spatial domain filtering and / or compensation processing are used for the static area and the moving area. This can achieve good noise suppression in the overall area and reduce the afterimage phenomenon.
[0095] The contents disclosed above are only preferred feasible embodiments of the present invention, and do not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention description and drawings are included in the scope of the patent application of the present invention.
[0096] Reference numerals
[0097] Video noise reduction system: 1
[0098] - Processor: 10
[0099] Memory: 12
[0100] Image processing chip: 2
[0101] Displays: 3
[0102] Video source: VS
[0103] Computer readable instructions: D1
[0104] Portrait mode model: D2
[0105] Image buffer: D3
[0106] Post-processing program: D4
[0107] Filter program: D5
[0108] Foreground area: AF
[0109] Background area: AB
[0110] Contour area: AC
[0111] Mask image: MK
[0112] Current image: IMG
[0113] Area: A1, A2
[0114] Steps: S10~S13, S120~S127, S1140~1142
Claims
1. A method for video noise reduction based on portrait mode, comprising configuring a processor to perform the following steps: Input the current image into the portrait mode model to generate a current portrait probability image, which defines the probability that each sub-region in the current input image is a foreground portrait; Perform post-processing procedures, including: performing binarization processing on the current portrait probability image according to a probability threshold to generate a mask image; and executing a contour extraction procedure to extract a foreground region, a background region and a contour region from the mask image; Execute filtering procedures, including: executing a motion detection program, comparing the current image with a previous image, and finding a moving area and a stationary area from the current image based on the extracted foreground area, the background area, and the contour area; Performing a time domain filtering operation on the static area; For the moving area, find a similar part in the previous image that is similar to the current image and obtain the degree of difference between them, so as to determine whether to compensate the moving area with the similar part or perform a spatial domain filtering operation; and The filtered static area and the moving area are output to generate the filtered current image.
2. The video denoising method as described in claim 1 further includes reducing the current image from a first image size to a second image size corresponding to the portrait mode model, and then inputting the portrait mode model, and the post-processing procedure also includes executing an enlargement step, enlarging the current portrait probability image from the second image size to the first image size, and then performing the binarization process.
3. The video noise reduction method according to claim 1, wherein: The current portrait probability image represents the probability that each sub-region of the current image is the foreground portrait with a numerical value ranging from 0 to 1.
4. The video noise reduction method according to claim 2, wherein: The binarization process includes setting the sub-region of the current portrait probability image whose probability is greater than or equal to the probability threshold to a first value, and setting the sub-region of the probability less than the probability threshold to a second value, so as to generate the mask image having a plurality of mask sub-regions corresponding to the sub-regions of the current portrait probability image.
5. The video noise reduction method according to claim 3, wherein: The mask image is divided into a plurality of blocks, and the contour extraction procedure includes performing the following steps for each of the blocks: Calculating the number of the mask sub-areas corresponding to the current block that are set to the first value as the number of foreground areas of the current block; as well as The foreground ratio of the foreground area quantity in the current block is calculated to determine whether the current block is the foreground area, the background area or the contour area.
6. The video noise reduction method according to claim 5, wherein: When the foreground ratio is within the contour ratio range, the current block is judged to be the contour area; when the foreground ratio is greater than the upper limit of the contour ratio range, the current block is judged to be the foreground area; when the foreground ratio is less than the lower limit of the contour ratio range, the current block is judged to be the background area.
7. The video noise reduction method according to claim 1, wherein: The foreground area, the background area, and the contour area correspond to different weights respectively, and the motion detection procedure includes performing the following steps for each area in the current image: Determining whether the current area belongs to the foreground area, the background area or the outline area; and The comparison result between the current image and the previous image is weighted according to the corresponding weights to find the moving area and the static area.
8. The video noise reduction method according to claim 1, wherein: The current image is a current frame image, and the previous image is a previous frame image processed by the post-processing program and after passing through the filtering program.
9. The video noise reduction method according to claim 1, wherein: For the moving area, the step of finding the similar part in the previous image that is similar to the current image and obtaining the degree of difference therebetween to determine whether to compensate the moving area with the similar part or to perform the spatial domain filtering operation further includes: For each sub-region corresponding to the moving region, find another sub-region similar to the current sub-region from the previous image, and compare the difference between the two; When the difference is less than a predetermined difference threshold, compensating the current sub-region of the moving region with the other sub-region; and When the difference is greater than or equal to the predetermined difference threshold, a spatial domain filtering operation is performed on the current sub-region of the moving region.
10. A video noise reduction system based on portrait mode, comprising: processor; as well as a memory storing a plurality of computer readable instructions, a current image, and a previous image, The processor is configured to read the computer-readable instructions and execute the following steps: Inputting the current image into a portrait mode model to generate a current portrait probability image, which defines the probability that each sub-region in the current input image is a foreground portrait; Perform post-processing procedures, including: performing binarization processing on the current portrait probability image according to a probability threshold to generate a mask image; and executing a contour extraction procedure to extract a foreground region, a background region and a contour region from the mask image; Execute filtering procedures, including: executing a motion detection program, comparing the current image with the previous image, and finding a moving area and a stationary area from the current image based on the extracted foreground area, the background area, and the contour area; Performing a time domain filtering operation on the static area; For the moving area, find a portion similar to the current image from the previous image, and compare the difference between the portion and the current image, when the difference is less than a predetermined difference threshold, compensate the moving area with the similar portion, and when the difference is greater than or equal to the predetermined difference threshold, perform a spatial domain filtering operation on the moving area; and The filtered static area and the moving area are output to generate the filtered current image.
